Eliminate createRequire/require from EXPORT_ES6 output#26384
Eliminate createRequire/require from EXPORT_ES6 output#26384vogel76 wants to merge 6 commits intoemscripten-core:mainfrom
createRequire/require from EXPORT_ES6 output#26384Conversation
sbc100
left a comment
There was a problem hiding this comment.
Wow, thanks for working on this! Nice work.
src/lib/libwasi.js
Outdated
| if (ENVIRONMENT_IS_NODE) { | ||
| #if EXPORT_ES6 | ||
| return (view) => nodeCrypto.randomFillSync(view); | ||
| #else |
There was a problem hiding this comment.
To keep thing simpler could we just always use the nodeCrypto global via __deps? (i.e. make the difference only in how nodeCrypto is globally defined?
src/shell_minimal.js
Outdated
| Module['wasm'] = fs.readFileSync(new URL('{{{ TARGET_BASENAME }}}.wasm', import.meta.url)); | ||
| #endif | ||
| #endif | ||
| #else |
There was a problem hiding this comment.
This seems like a lot of extra complexity/duplication. Is there some way to avoid it maybe?
createRequire/require from EXPORT_ES6 output
0ec991e to
6b74d2a
Compare
|
@sbc100 I hope your remarks have been addressed in added fixup commits. Once you accept it, I will do autosquash rebase to leave only actual commits in clean history. |
6b74d2a to
8f585b7
Compare
|
Is it possible to write a test for this? When you say "This breaks bundlers", do you know why our current bundler tests in test_browser.py are working? (see |
|
Re-commiting, we always squash all changes in the emscripten repo. If you think this change can be split into to commits then please send a two separate PRs. This is just how we do things in emscripten. It helps keep the tests passing on every commit (for the benefit of bisectors). |
src/preamble.js
Outdated
| var v8 = nodeV8; | ||
| #else | ||
| var v8 = require('node:v8'); | ||
| #endif |
There was a problem hiding this comment.
No need for this var anymore if you use makeNodeImport I thinik?
I think you for this globalVar its reasonable to call it v8?
| var fs = require('node:fs'); | ||
| var fs = {{{ makeNodeImport('node:fs') }}}; | ||
| #if WASM == 2 | ||
| if (globalThis.WebAssembly) Module['wasm'] = fs.readFileSync(__dirname + '/{{{ TARGET_BASENAME }}}.wasm'); |
There was a problem hiding this comment.
Does this code (which uses __dirname) simply not work with EXPORT_EST today?
If not, this seems like maybe a separate fix that we could land in isolation. e.g. Fix for EXPORT_ES6 + MINIMAL_RUNTIME + ???
Sure - will add such tests. The sake of such work are maintenance problems of our Hive blockchain interfacing library: wax which compiles blockchain protocol C++ code into wasm and uses it at TS/JS level. We have set of tests related to bundlers and different frameworks where we got problems while loading wasm etc. https://gitlab.syncad.com/hive/wax/-/tree/develop/examples/ts?ref_type=heads |
8f585b7 to
6c822be
Compare
Deeper analysis gave conclusions why Existing tests don't catch the require() problem. The real breakage happens when:
I extended test to cover more usecases altough adding tests covering whole frameworks seems to heavy for your project. Our wax library covers that... |
I'm curious about this case. I'd love to add a test to Are you saying that rollup doesn't actually break (since it marks node builtins as external by default)? If so, is rollup just not effected by this bug? Is vite effected? Could we add a test to test_other.py alongside |
Here are new tests targeted to main where they are expected to fail: They shall pass here ofc. Once you accept it, I will cherry-pick them to this PR |
|
Those tests look great thanks! |
2270aac to
d068717
Compare
sbc100
left a comment
There was a problem hiding this comment.
(sorry I forgot the send my pending comment)
src/lib/libwasi.js
Outdated
|
|
||
| #if ENVIRONMENT_MAY_BE_SHELL | ||
| #if ENVIRONMENT_MAY_BE_NODE && MIN_NODE_VERSION < 190000 | ||
| $nodeCrypto: "ENVIRONMENT_IS_NODE ? {{{ makeNodeImport('node:crypto') }}} : undefined", |
There was a problem hiding this comment.
I think maybe the ENVIRONMENT_IS_NODE ? xxx : undefined pattern can just be folded into makeNodeImport?
You might also want to add assert(!ENVIRONMENT_MAY_BE_NODE, "makeNodeImport called when environment can never be node") to the macro.
d068717 to
8b2fb1c
Compare
| # require() calls for node support which vite externalizes but leaves | ||
| # as require() in the bundle output. | ||
| self.run_process([EMCC, test_file('hello_world.c'), '-sEXPORT_ES6', '-sEXIT_RUNTIME', | ||
| '-sMODULARIZE', '-o', 'hello.mjs']) |
There was a problem hiding this comment.
If you pass -sEXPORT_ES6 you can remove -sMODULARIZE here and above.
Also does this test actually need -sEXIT_RUNTIME? If not maybe just delete it.
Also, I think can can even delete -sEXPORTE_ES6 since emcc will enable that by default when the output filename ends in .mjs.
8b2fb1c to
23863c1
Compare
|
Changes in this update:
Known remaining issues:
|
When EXPORT_ES6 is enabled, the generated JS used createRequire() to
polyfill require(), which breaks bundlers (webpack, Rollup, esbuild)
and Electron's renderer process. Since EXPORT_ES6 requires MODULARIZE,
the module body is wrapped in an async function where await is valid.
- shell.js: Remove createRequire block entirely. Use await import()
for worker_threads, fs, path, url, util. Replace __dirname with
import.meta.url for path resolution.
- shell_minimal.js: Same pattern for worker_threads and fs. Replace
__dirname with new URL(..., import.meta.url) for wasm file loading.
- runtime_debug.js: Skip local require() for fs/util when EXPORT_ES6,
reuse outer-scope variables from shell.js instead.
- runtime_common.js: Guard perf_hooks require() with EXPORT_ES6
alternative.
- preamble.js: Hoist await import('node:v8') above instantiateSync()
for NODE_CODE_CACHING since await can't be used inside sync functions.
Library functions run in synchronous context where await is unavailable.
Define top-level library symbols that use await import() at module init
time, then reference them via __deps from synchronous functions.
- Add libnode_imports.js with shared $nodeOs symbol, register in
modules.mjs when EXPORT_ES6 is enabled.
- libatomic.js, libwasm_worker.js: Use $nodeOs for os.cpus().length
instead of require('node:os').
- libwasi.js: Define $nodeCrypto for crypto.randomFillSync in
$initRandomFill. Combine conditional __deps to avoid override.
- libcore.js: Define $nodeChildProcess for _emscripten_system.
- libnodepath.js: Switch $nodePath initializer to await import().
- libsockfs.js: Define $nodeWs ((await import('ws')).default) for
WebSocket constructor in connect() and Server in listen().
Bundlers (webpack, rollup, vite, esbuild) and frameworks (Next.js, Nuxt) cannot resolve CommonJS require() calls inside ES modules. This test statically verifies that EXPORT_ES6 output uses `await import()` instead of `require()` for Node.js built-in modules, and that the `createRequire` polyfill pattern is not present. Parameterized for default, node-only, and pthreads configurations to cover the various code paths that import Node.js built-ins (fs, path, url, util, worker_threads).
…ire() Add two tests that verify EXPORT_ES6 output is valid ESM and works with bundlers: - test_webpack_esm_output_clean: Compiles with EXPORT_ES6 and default environment (web+node), then builds with webpack. On main, webpack hard-fails because it cannot resolve 'node:module' (used by emscripten's createRequire polyfill). This breaks any webpack/Next.js/Nuxt project. - test_vite_esm_output_clean: Compiles with EXPORT_ES6 and default environment, then builds with vite. On main, vite externalizes 'node:module' for browser compatibility, emitting a warning. The resulting bundle contains code referencing unavailable node modules. These tests are expected to fail on main and pass after eliminating require() from EXPORT_ES6 output.
23863c1 to
2b964d5
Compare
The previous fix moved await import() for node:fs and node:util outside dbg() to module scope. This broke test_dbg because dbg() can be called from --pre-js before those module-scope imports execute. Use lazy initialization instead: declare dbg_node_fs/dbg_node_utils early but leave them undefined. Initialize them in shell.js after fs and utils are loaded (reusing the same imports). dbg() checks if the modules are available and falls back to console.warn if not. This handles all cases: - dbg() from --pre-js (before init): uses console.warn - dbg() after init on Node.js with pthreads: uses fs.writeSync - dbg() in browser/non-node: uses console.warn
30c1f96 to
14fdc3b
Compare
In ESM mode (WASM_ESM_INTEGRATION), the runtime uses dynamic import()
instead of require() for node modules. Update the test_environment
assertion to check for 'import(' when in ESM mode, rather than always
expecting 'require('.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
@sbc100 could you take a look for test-bun and test-deno failures? Probably I will need some help in getting local env. to reproduce problems. Please point me some doc or scripts how to setup it. Best if you have some docker image to run them in... |
When building with
-sEXPORT_ES6, the generated JavaScript currently usescreateRequire(import.meta.url)to polyfillrequire(), then callsrequire()for Node.js built-in modules. This breaks bundlers (webpack, Rollup, esbuild, Vite) and Electron's renderer process, forcing users to post-process emscripten output withsedor custom plugins to strip out the offending calls.This PR replaces all
require()calls withawait import()whenEXPORT_ES6is enabled, eliminating the need forcreateRequireentirely.Why this is safe
EXPORT_ES6requiresMODULARIZE(enforced intools/link.py).MODULARIZEwraps the module body in anasync function, soawaitis valid at the top level of the generated code.await import(...)is transformed to anEMSCRIPTEN$AWAIT$IMPORTplaceholder during preprocessing (parseTools.mjs:83) and restored during linking (link.py:2136). This mechanism is already used elsewhere in the codebase.(await import('node:fs')).readFileSyncworks identically torequire('node:fs').readFileSync. For CJS npm packages likews,.defaultgives themodule.exportsvalue.#if EXPORT_ES6/#elsepreprocessor conditionals. The existingrequire()code path is untouched.Approach
The changes split naturally into two categories:
Shell and runtime files (top-level async context where
awaitworks directly):src/shell.js— Remove thecreateRequireblock. Useawait import()forworker_threads,fs,path,url,util.src/shell_minimal.js— Same pattern forworker_threadsandfs. Replace__dirnamewithnew URL(..., import.meta.url)for wasm file loading.src/runtime_debug.js— Skip localrequire()forfs/utilwhenEXPORT_ES6; reuse outer-scope variables fromshell.js.src/runtime_common.js— Guardperf_hooksrequire()withEXPORT_ES6alternative.src/preamble.js— Hoistawait import('node:v8')aboveinstantiateSync()forNODE_CODE_CACHING(can't useawaitinside a sync function).Library files (synchronous function bodies where
awaitis unavailable):Library functions execute in synchronous context, so
await import()cannot be called inline. Instead, we define library symbols initialized at module top level (async context) and reference them via__deps:$nodeOsnode:oslibatomic.js,libwasm_worker.js$nodeCryptonode:cryptolibwasi.js($initRandomFill)$nodeChildProcessnode:child_processlibcore.js(_emscripten_system)$nodeWsws(.default)libsockfs.js(WebSocket connect + listen)$nodePathnode:pathlibnodepath.js(NODERAWFS)Shared symbols live in a new
src/lib/libnode_imports.js, registered insrc/modules.mjswhenEXPORT_ES6is enabled.Intentionally skipped
src/lib/libembind_gen.js— Build-time code running in Node.js directly, not in emitted runtime JS.src/closure-externs/closure-externs.js—createRequireextern kept; still needed for the non-ES6 path.What changes in the output
Before (EXPORT_ES6):
After (EXPORT_ES6):
The non-ES6 path remains unchanged:
Files modified
13 files modified, 1 new file — 123 insertions, 12 deletions (source files only).
Testing
All existing ESM tests pass:
test_esm(default, node, pthreads variants)test_esm_worker(default, node variants)test_esm_worker_single_filetest_esm_closuretest_esm_implies_modularizetest_esm_requires_modularizetest_pthread_export_es6(default, trusted variants)Additionally verified:
emcc -sEXPORT_ES6 -sMODULARIZEoutput contains zerocreateRequireorrequire(occurrencesemcc -sEXPORT_ES6 -sMODULARIZE -pthreadoutput contains zerocreateRequireorrequire(occurrencesrequire()as before (no regression)Related issues
EXPORT_ES6output containsrequire()which breaks bundlersEXPORT_ES6use only ES6 semantics? #20503 —createRequirebreaks Electron rendererrequire()require()in ESM output breaks RollupRelated PRs
nodebuilt-in module imports withnode:. #18235 — Earlier attempt to address this (partial)