-
Notifications
You must be signed in to change notification settings - Fork 7
Add draft of static require RFC #39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,219 @@ | ||
| # RFC: Static require() for ServerLua | ||
|
|
||
| * Status: **DRAFT** | ||
|
|
||
| ## Summary | ||
|
|
||
| Compile-time static `require()` resolution that bundles multiple modules into a single deployable unit. Client-side bundling | ||
| avoids the kind of security issues (see MySQL's `LOAD LOCAL`) where the server has to tell the client what local resources are needed | ||
| to continue bundling, and the client can't reason about what resources are needed on its own. Preserves original filenames and line numbers | ||
| for debugging via per-Proto source info in bytecode. | ||
|
|
||
| ## Motivation | ||
|
|
||
| **Problem:** No code reuse beyond copy-paste in-viewer. Large scripts are unwieldy. LSL's `#include` loses dependency info after processing, | ||
| if you don't have the local dependencies you have no option but to edit the post-processed source code soup. | ||
|
|
||
| **Why static `require()`:** SL's creation and permission models don't play nice with dynamic runtime dependencies. Bundling together dependencies when | ||
| building makes it easy for us to support multiple development styles (external editor with custom build toolchain, in-viewer editor have different opinions about | ||
| how to include files and from where) | ||
|
|
||
| **Why it's tricky:** Server compiles, but dependencies live on client. Can't reasonably have server request files (i.e. `LOAD LOCAL` from MySQL). | ||
| Need debug info across modules. Want future tree shaking/inlining for optimization purposes. | ||
|
|
||
| ## Path Resolution | ||
|
|
||
| All paths are explicit - no relative paths (`./`, `../`). | ||
|
|
||
| | Pattern | Resolves to | | ||
| |---------|---------------------------------------------------------------------------------------| | ||
| | `require("foo")` | `package.path`-like semantics. Includes from the top of the package, or from libs dir | | ||
| | `require("@myalias/utils")` | User-configured alias → local path | | ||
| | `require("@sl/json")` | Platform libs (reserved namespace for LL-provided native Lua libs) | | ||
|
|
||
| **Client resolves aliases before bundling.** Bundle stores alias paths; runtime does simple string lookup when actually | ||
| calling `require()` | ||
|
|
||
| ## Bundle Format | ||
|
|
||
| Text-based, valid Luau syntax, lightly inspired by MIME multipart RFC: | ||
|
|
||
| ```lua | ||
| --[[!!SLUA:BUNDLE!!]] | ||
| -- NOTE: May have some metadata in the header too | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is the meta data just any comments after the bundle comment? Alternatively as the bundle comment is a multi line one, anything after the first newline inside the comment could be ignored by the bundle system e.g. --[[!!SLUA:BUNDLE!!
<meta info about bundle>
]]
--[[!!SLUA:MODULE:@libs/lib!!
<meta info about module>
]]
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep, the metadata would be part of the slua bundle directive itself.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could change the line in the rfc to this then to show it explicitly? --[[!!SLUA:BUNDLE!!
NOTE: May have some metadata in the header too
]] |
||
| -- MAIN is implicit (first section after header) | ||
| local foo = require("@myproject/lib/foo") | ||
| local json = require("@sl/json") | ||
| foo.bar() | ||
|
|
||
| --[[!!SLUA:MODULE:@myproject/lib/foo!!]] | ||
| local helpers = require("@myproject/lib/helpers") | ||
| return { bar = function() return helpers.helper() end } | ||
|
|
||
| --[[!!SLUA:MODULE:@myproject/lib/helpers!!]] | ||
| return { helper = function() return "hello" end } | ||
| ``` | ||
|
|
||
| **Rules:** | ||
| - `--[[!!SLUA:BUNDLE!!]]` header must be first line | ||
| - MAIN is implicit (content between header and first MODULE) | ||
| - `--[[!!SLUA:MODULE:path!!]]` marks each dependency | ||
| - `--[[!!SLUA:` in user source is rejected | ||
| - Platform libs (`@sl/...`) not included - provided by runtime | ||
| - Generally the user only sees and directly edits the `MAIN` bit of the bundle | ||
| - Users may define their own global aliases that refer to particular libs on their disk | ||
| - - For ex. you might `require("@textlib/v2")` to pull in v2 of your text rendering library | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Which default file semantics will be used for the luau file if a directory is specified instead of a file directly? At the moment it seems luau rfcs state either
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm open to either, may have to look at Lute to see if they have different resolution strategies there. |
||
|
|
||
| Conceptually, the bundle provides a sort of virtual filesystem for the runtime `require()` implementation. | ||
|
|
||
| Notably, since the bundle format includes the source code as-is, before any optimization and tree-shaking, file name | ||
| and line mappings for errors are automatically correct, without `--!@line` directives or similar. | ||
|
|
||
| ## Editing Workflow | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| subgraph Start | ||
| direction LR | ||
| OpenExisting[Open existing script] | ||
| CreateNew[Create new script] | ||
| end | ||
|
|
||
| DownloadArchive[Download source archive] | ||
| IsValidBundle{Valid bundle?} | ||
| MainView[MAIN-only view] | ||
| RawView[Raw view] | ||
| UserEdits[User edits] | ||
|
|
||
| CreateNew --> MainView | ||
| OpenExisting --> DownloadArchive | ||
| DownloadArchive --> IsValidBundle | ||
| IsValidBundle -->|Yes| MainView | ||
| IsValidBundle -->|No| RawView | ||
| MainView --> UserEdits | ||
| RawView --> UserEdits | ||
|
|
||
| UserEdits -->|Toggle view| CheckUnsaved | ||
| UserEdits -->|Save| StartSave | ||
|
|
||
| subgraph Toggle[Toggle View] | ||
| CheckUnsaved{Unsaved changes?} | ||
| TogglePrompt{Discard or cancel?} | ||
| CheckToggleDir{Raw to MAIN?} | ||
| ValidBundle{Valid bundle?} | ||
| DoToggle[Toggle view] | ||
| ShowParseError[Show parse error] | ||
|
|
||
| CheckUnsaved -->|No| CheckToggleDir | ||
| CheckUnsaved -->|Yes| TogglePrompt | ||
| TogglePrompt -->|Discard| CheckToggleDir | ||
| CheckToggleDir -->|No| DoToggle | ||
| CheckToggleDir -->|Yes| ValidBundle | ||
| ValidBundle -->|Yes| DoToggle | ||
| ValidBundle -->|No| ShowParseError | ||
| end | ||
|
|
||
| subgraph Save | ||
| StartSave[User saves] | ||
| IsRawView{Viewing raw?} | ||
| SendRaw[Send as-is] | ||
| HasLocalDep{Local dep exists?} | ||
| CheckOwnership{Same user last saved?} | ||
| BundleLocal[Bundle in local dep] | ||
| CheckMatch{Local matches bundle?} | ||
| ConfirmUntrusted{Confirm: pull local dep\ninto untrusted script?} | ||
| DepInBundle{Dep in existing bundle?} | ||
| BundleFromBundle[Bundle from existing] | ||
| ResolveError[Error: can't resolve] | ||
| SendBundle[Send bundle] | ||
| AbortSave[Abort] | ||
|
|
||
| StartSave --> IsRawView | ||
| IsRawView -->|Yes| SendRaw | ||
| IsRawView -->|No| HasLocalDep | ||
| HasLocalDep -->|Yes| CheckOwnership | ||
| CheckOwnership -->|Yes| BundleLocal | ||
| CheckOwnership -->|No| CheckMatch | ||
| CheckMatch -->|Yes| BundleLocal | ||
| CheckMatch -->|No| ConfirmUntrusted | ||
| ConfirmUntrusted -->|Yes| BundleLocal | ||
| ConfirmUntrusted -->|No| AbortSave | ||
| HasLocalDep -->|No| DepInBundle | ||
| DepInBundle -->|Yes| BundleFromBundle | ||
| DepInBundle -->|No| ResolveError | ||
| ResolveError --> AbortSave | ||
| BundleLocal --> SendBundle | ||
| BundleFromBundle --> SendBundle | ||
| end | ||
|
|
||
| TogglePrompt -->|Cancel| UserEdits | ||
| ShowParseError --> UserEdits | ||
| DoToggle --> UserEdits | ||
| AbortSave --> UserEdits | ||
|
|
||
| SendRaw --> Compile | ||
| SendBundle --> Compile | ||
|
|
||
| subgraph Server | ||
| Compile[Compile & store] | ||
| Done[Return result] | ||
| Compile --> Done | ||
| end | ||
| ``` | ||
|
|
||
| **Key points:** | ||
| - MAIN-only view: user sees entry script, client bundles on save | ||
| - Raw view: user sees/edits full archive, sent as-is | ||
| - Ownership check: security measure to prevent leaking local code into scripts others control | ||
| - - Open question as to how this should work, we'll need to do some server-side enrichment | ||
| - Fallback: if local dep missing, use version from downloaded bundle | ||
| - - This allows users to do one-off edits to scripts from the viewer even if they don't have all the | ||
| constituent parts on their drives. | ||
|
|
||
| ## Runtime | ||
|
|
||
| - `require()` implemented in C (hides module table/cache from scripts) | ||
| - Module results cached after first execution | ||
| - Each module runs in sandboxed environment via `dangerouslyexecuterequiredmodule()` | ||
| - - This function will be hidden from view and not directly usable. | ||
| - Simple string lookup - no alias resolution at runtime | ||
|
|
||
| ## Bytecode Extension | ||
|
|
||
| **Problem:** Standard Luau shares one `chunkname` across all functions. Bundles have multiple | ||
| source files. It's useful to be able to have proper filename and line mappings in errors. | ||
|
|
||
| **Solution:** Per-Proto source in bytecode. Each function stores its source filename. Loader reads per-function source for correct stack traces. | ||
|
|
||
| ```cpp | ||
| // In lvmload.cpp | ||
| TString* protoSource = hasPerProtoSource | ||
| ? strings[readVarInt(data, size, offset)] | ||
| : source; // Fallback for non-bundles | ||
| p->source = protoSource; | ||
| ``` | ||
|
|
||
| ## Errors | ||
|
|
||
| | Error | When | Message | | ||
| |-------|------|---------| | ||
| | Dynamic require | Compile | `require() argument must be a string literal` | | ||
| | Relative path | Compile | `relative paths not supported` | | ||
| | Unknown alias | Compile | `unknown alias '@foo' in require` | | ||
| | Module not found | Compile | `cannot find module '<path>'` | | ||
| | Circular dependency | Compile | `circular dependency: <path1> -> <path2> -> <path1>` | | ||
| | Delimiter in source | Compile | `source cannot contain '--[[!!SLUA:'` | | ||
| | Depth exceeded | Compile | `require depth exceeds maximum` | | ||
| | Module not in bundle | Runtime | `module not found: <path>` | | ||
|
|
||
| ## Limits | ||
|
|
||
| - Max dependency depth: 100 | ||
| - Max total modules: 1000 | ||
| - No circular dependencies (no different from typical Luau here!) | ||
|
|
||
| ## Future Work | ||
|
|
||
| - Tree shaking (eliminate unused exports) | ||
| - Cross-module inlining (`--!pure` modules) | ||
| - Inventory-based module resolution | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this have a namespace reserved in advance?
It may be prudent to declare For instance if inventory based requires don't receive an alias ahead of time, reserving There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We're reserving @sl-*, so
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That should probably be added to the RFC then.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm concerned about specifying where to pull things from instead of what to pull inside the actual code. What would If the answer is "
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's sort of why I was thinking Like your bit about Basically the way i picture it on save the viewers built in bundler code will try and resolve any
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is
libs dirin this instance?If its a user defined dir, I'd personally prefer keeping that functionality strictly to the user set aliases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking it could implicitly search through the
@libsalias, so that you don't necessarily have to set up a specific alias for every single package you install. In other languages terms, where@libsis equivalent to yournode_modulesordist-packagesor what have you. I imagine the same would work for in-world editing where you could eithera) have a "modules" subfolder with symbolic links to what you want to use for that particular folder of scripts, but this gets weird in script-inside-object cases since you can't have subfolders there
b) just load from some user-defined global "modules" folder in user inventory
whichever turns out to be less annoying for people's usecases, will need to do some investigation there. Option B seems the most obvious and frictionless, to me at the expense of things getting weird if you have two projects that both want different, incompatible versions of the same library and you want to manage your dependencies in inventory. Anything more than that and you end up effectively having to implement a package manager in the viewer for the developer experience to be acceptable.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Scripters currently are used to specifying a system folder to the
fs_preprocessorI think having them provide one explicitly and using@libs/..instead of magically including something would be preferable.Automagic choosing of a source feels like it will end up eventually conflicting with something.
For instance the viewer needs its own built in resolver for some aliases to do bundling into the package for users that don't user external tools. that would need to be disabled if the user wants to use the external vscode plugin, unless the plugin can provide a bundle where the dependencies are included in it, without aliases, so the viewer wont resolve those to anywhere but things already explicitly in the bundle.
So a save in vscode would produce this something like this.
The viewer would do no further bundling there, as
"thing"was provided, if it wasn't it should error imo, rather than behaviour being different if it is and isn't provided.Essentially the viewer / server bundling steps only to pull in defined code with
@aliases, not "magic" that could get something not well defined by the user.So a
require("test")would errorunknown moduleif that is not explicitly included by the user/plugin in the bundle.