Conversation
tools/update-modules(.sh/.ps1) keeps the embedded modules in sync with modules/*.js. - -current updates every entry whose modules/<name>.js changed, -add and -remove work on named entries, -sync follows modules/ exactly, -dryrun only reports, -list and -export show or save what is embedded. - While ILibDuktape_Polyfills.c still carries its addCompressedModule() statements, the scripts read and write those in place in the file's own format: an inline duk_peval_string_noresult(), or the chunked ILibMemory_Allocate/memcpy_s block for entries over 16000 base64 characters. Every run re-renders the whole table, so an unchanged entry produces no diff. - -striplegacy migrates: the modules move into a generated ILibDuktape_EmbeddedModules.c as deflated const unsigned char[] arrays plus a name/timestamp/data/length table, #included from the polyfills file, and the addCompressedModule() statements are replaced by one call to ILibDuktape_Polyfills_EmbeddedModules(ctx). That drops the base64 layer and the startup duk_peval of one giant string. The migration itself is a separate commit.
And while we're at it, update -current to -update
… settings Under node the tool wrote stamps like 'WedTSepT02T2026T...' (node's Date.toString() joined with T), which the agent turns into a NaN ModuleFileDate. Stamps are now rendered as 2026-09-02T15:36:17.000+02:00 on both engines, and taken from the commit that last touched the module (one git log over modules/), falling back to mtime for files with uncommitted changes, untracked files, or when git is missing. That mtime fallback read an hour or two late under a POSIX agent: its fs.statSync() formats file times with localtime() but appends 'Z', so a January file edited at 03:04:05 CET was stamped 04:04:05+01:00 while node wrote 03:04:05+01:00. The tool now writes and stats a probe file once per run, and when the agent's time is ahead by the timezone offset it reads mtimes back as local time. A fixed agent, node, or a machine on UTC gets no correction. Node's zlib is Chromium's fork and never reproduces the agent's stock zlib stream (0 of 100 entries), so an entry's bytes depended on which engine regenerated it. pako, a port of stock zlib, matches all 100; it is used under node when installed and the launchers offer to npm install it. The generated file's hex is lower-cased, since the agent emits upper case. Workarounds for the agent: fs._readdirSync() is called directly, because the Windows readdirSync wrapper loads 'os' and crashes an agent without the alloca fix; git is located on PATH by hand and argv[0] repeated, as the agent's execFile does neither; the -Check exit code travels through a variable, since the agent's own process.exit() logs itself through the launcher's catch. Settings now travel as environment variables instead of JS source, which removes all quoting: a Windows -Export path pasted into a '...' literal had its backslashes read as escapes and landed on drive C:. The bootstrap chdir covers a Windows agent run from Git Bash, which moves to its own directory at startup. Also: - Update is passed through so -Update -Remove x updates the rest. - Sync implies update; -Check for CI exits 1 when the target would change. - Add runs count untouched entries; a readdir failure throws rather than reading as nothing new. - Export creates nested directories - A bare -BinaryPath file name resolves - Node is used when no agent binary is found. - The script and its package files move to tools/update-modules/ to keep tools/ clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I was tired of the annoying process of generating the embedded modules in polyfills.c, so made this little script to do it automatically.
Sadly the limit on string literals is still a thing in MSVC, so the memcpy way is still needed in the current version.
There is an option to split the embedded modules into a seperate ILibDuktape_EmbeddedModules.c. This has a few advantages:
The scripts (linux and windows wrappers for the core update-modules.js) can manage both the current style, embed in polyfills, and the proposed split style.
By default it uses the list of embedded modules in the source, but there is a -sync option to add/remove/update the embedded modules depending on the contents of the modules/ dir. At this moment the modules/ dir contains more than needed, but a way could be to (re)move the extra modules, as these are served from meshcentral. A point of discussion how to manage those and keep them in sync between meshcentral/meshagent repo's (github actions maybe?). There should be even less embedded modules then there are now I think.
There are the -add & -remove options to do that to specific modules.
The option -current updates the embedded modules with the modules in modules/. Why I didn't call it -update is a mystery to me now that I'm typing this. But not going to change that now. It is late.
One of the nice things I think is that this script uses the meshagent as a javascript engine! But it can also use node if it's installed.
FIxed: !! Oh and a important issue to be aware of (which I may fix tomorrow, maybe) is that if you use the meshagent for the engine, you need a meshagent with #376 applied, as the js script is larger the 4096 characters. Simple fix is to use node.
14-09-2026: Proper (annoying) CRLF handling between windows/POSIX, agent timebug & readdirsync workaround, added pako to node to keep the zipstreams byte-equal (node uses a chrome fork with different match finder), and a few usage improvements, like naming the parameter '-update' instead of '-current'.
Details
I understand that I am responsible for and able to explain every line of code I submit.
the impact on platforms and architectures I could not test.
modules/, I re-embedded them so the compiled-in copies inmicroscript/ILibDuktape_Polyfills.cmatch (the agent runs the embedded copies, not the files on disk)..mshoptions table in readme.md.Testing
Tested on linux and windows x86/x64 with the testscripts and a connectivity test to a meshcentral server (desktop/files/terminal/console work)