Support system database sharing (application names) and cross-application enqueue - #471
Merged
Conversation
devhawk
force-pushed
the
application-name
branch
from
August 26, 2026 22:25
eff5f68 to
53b8990
Compare
devhawk
changed the title
devhawk
force-pushed
the
application-name
branch
from
August 26, 2026 22:39
53b8990 to
5c2fe1e
Compare
Ports python#809 / ts#1326 / go#444+#445: every DBOS object -- workflows, steps, queues, schedules, application versions -- records the application that owns it, so several applications can share one system database in isolation, or interoperate deliberately by naming each other's objects. NULL means unclaimed: the columns are added nullable with no default, so every pre-existing row, and every row an SDK that does not know the column writes, belongs to all of them. Reads match `application_name = ? OR application_name IS NULL` throughout. Schema: migrations 100-107, the start of a cross-SDK shared history where every SDK defines the same migration at the same index. This language's own history is padded out to 99; the runner and the generated script skip the padding without a round trip each and record it in one write. Scoped: listings (workflows, queues, schedules, versions, aggregates, metrics), the queue dequeue and its flow-control counts, the latest-version lookup, the delayed-workflow and garbage-collection sweeps, and the debounce holder lookup. Recovery follows, since it lists by executor ID -- which defaults to the literal `local` in all four SDKs, so two unnamed applications on one database would otherwise recover each other's work. Unscoped, deliberately: everything addressed by workflow ID. A fork is the forking application's own work, steps included. Registering a queue, schedule, or application version under a name another application holds raises DBOSApplicationNameConflictException: names are shared address space, so a collision is not ours to resolve. The application name is hashed into the computed application version, so two applications built from one jar do not collide on a single version row. Existing deployments therefore compute a new version on upgrade. DBOSClient takes an optional application name; without one it owns nothing and sees every application's rows.
The other four read types this change touches -- WorkflowStatus, Queue, WorkflowSchedule, VersionInfo -- expose the application that owns the row. StepInfo was the odd one out, which also cost export/import fidelity: it is what carries steps through the export, so importing collapsed every step onto its workflow's owner. The two normally agree, but they need not: a workflow one application owns can be resumed or restarted by another addressing it by ID -- deliberately unscoped -- and the steps that run then belong to the application that ran them. An import now keeps the owner each step was exported with, falling back to the workflow's for an export predating the column. Python and Go do not expose this on their step type, so this is Java reading one column further than they do, not a divergence in what is stored.
The fallback to the workflow's owner invented ownership the payload did not
carry. Python (`output.get("application_name")`) and TypeScript
(`output.application_name ?? null`) both take the exported value verbatim, so
an export predating the column imports its steps unclaimed -- which is the
representation's own rule: NULL means unclaimed, and no writer should guess an
owner for a row it did not stamp.
DBOS.enqueueWorkflow takes the same EnqueueOptions as DBOSClient and writes the same row, so a Java application can enqueue a workflow implemented in another process or another language without holding a reference to its function. The portable variant carries named arguments, for targets like a Python workflow with keyword arguments. Called from inside a workflow it behaves like startWorkflow: the enqueued workflow is recorded as a child under a derived ID, so a crash-replay collides with the original enqueue instead of enqueueing a second workflow. Called from a step it throws, as starting a workflow from a step already does. Two deliberate differences from startWorkflow: the workflow name and queue are not checked against the local registries, since neither may exist here, and the application version is left unset unless given, so the owning application's own latest version picks the work up. DBOS.listQueues and DBOS.listSchedules gained the same application filter the client has.
devhawk
force-pushed
the
application-name
branch
from
August 26, 2026 22:59
5c2fe1e to
22903d1
Compare
DBOS.enqueueWorkflow documented an EnqueueOptions.applicationName that did not exist, which broke the javadoc build. The doc was right and the code was missing: without it every enqueue stamps the enqueueing application, so a row one application writes for a peer is owned by the writer and the peer's dequeue predicate -- application_name = its own OR IS NULL -- never matches it. Naming a peer at the enqueue boundary is the whole of the cross-application contract, so it has to be expressible. EnqueueOptions gains applicationName, and WorkflowStatusInternal carries it down to the insert, as initStatus.applicationName does in TypeScript and status["application_name"] in Python. The status's own name wins over the handle's; unset, the row belongs to the enqueueing application, or to nobody when the enqueuer has no name of its own. Threaded as a parameter through enqueueWorkflow and persistWorkflow rather than as a component on ExecutionOptions: only the two enqueue entry points ever set it, and ExecutionOptions has seventeen withX copy methods that would each have to re-list it for no gain.
google-java-format wraps the insertWorkflowStatus call now that resolving the owner pushed the argument list past the line limit.
The dequeue flipped status to PENDING without stamping the application that took the row. An unclaimed row -- written before this feature, or by a nameless client -- therefore stayed unclaimed while PENDING, inside every peer's recovery and global-timeout sweeps, until the executor that took it got as far as initWorkflowStatus and claimed it there. Crash in that window and it stays unclaimed for good. Claiming belongs on the statement that takes the row, which is where Python (_sys_db.py:4465) and TypeScript (system_database.ts:3673) put it. COALESCE rather than a bare assignment, following TypeScript: a nameless dequeuer then leaves an existing owner untouched instead of erasing it, and it is already this SDK's idiom for claiming only what nobody owns. The WHERE re-checks ownership alongside status, so the claim is never wider than the candidate SELECT that scoped it. The delayed-to-enqueued transition is deliberately left alone: it is scoped but does not stamp, matching Python, since the row is claimed when it is actually dequeued.
Listing by workflow ID was narrowed to this application like every other listing, so a Java executor could not answer for a workflow it had handed to a peer -- and neither could Conductor through it, since get-workflow arrives as a list keyed by one ID and no application filter. A workflow ID is a global address, so a read keyed by one is an identity read: it honours an explicit filter but is never defaulted to the reader's own application. Python branches on workflow_ids for exactly this (_sys_db.py:2062) and TypeScript on idKeyed (system_database.ts:3644). DbContext gains requestedNames, the list form of Python's _name_filter, beside scopeNames, which is its _observability_filter. Keeping both resolutions in one place is what stops the two predicates drifting apart. Only listWorkflows is keyed by ID. The aggregate reads keep the observability default unconditionally, as they do in both other SDKs.
Enqueue could name a peer; nothing else could. A queue, a schedule or an application version written through a handle was always owned by that handle's own application, so a deploy tool acting for several applications had to run as each one in turn to set any of them up. Python and TypeScript both take an application name on all three. Queues and versions take it as a parameter, since neither API has a record at its boundary: upsertQueue, createApplicationVersion and updateApplicationVersionTimestamp grow one, and their call sites pass null for the handle's own. Schedules read it off WorkflowSchedule, which already carries the component and now has a withApplicationName to set it -- the shape TypeScript uses for both. Internal signatures change in place rather than gaining siblings, so the compiler enumerates the call sites; DBOSClient gains overloads beside registerQueue and setLatestApplicationVersion, since existing code must keep compiling and keep meaning what it meant. Naming a peer is not impersonating it. The owner still resolves through RowOwner, so a name a third application already holds is the same conflict it always was.
Three loose ends in the application-name work. getMetrics counted only what the calling handle owned, with no way to ask for anything else, so Conductor could not count a peer's workflows through a Java executor. It now takes the same nullable filter every other observability read takes, and GetMetricsRequest carries application_name on the wire, both matching Python (_sys_db.py:5425, _conductor/protocol.py:502) and TypeScript. A Conductor predating the field sends nothing and still gets this application's own count. DBOSConfig now validates the application name to Python's rule: three to thirty characters of [a-z0-9-_]. The name is durable cross-language identity now -- written onto every row this application owns and read back by peers in other SDKs -- so a Java application must not be able to claim rows under a name a Python peer could never be configured with, and rename-application needs a rule to enforce. This is a breaking config change and belongs in the same release as the application version discontinuity. Two test application names used mixed case and had to be renamed, so the design note that every name in this repository already passed was wrong. The computed-version fallback carries the application name as well, so peers that cannot read their own bytecode do not converge on one version_name, which is still globally unique.
The conflict error told operators to re-own their rows with the Python or TypeScript CLI, which is a poor answer for a Java application: the escape hatch a Java error names should be one a Java installation has. It now names this SDK's own. renameApplication gives a name ownership of the rows another name holds, of unclaimed rows, or of both -- the second being the upgrade path, since every row written before this feature is unclaimed. Adopting is not renaming, so unclaimed rows move only when asked, which is the one place the scope differs from a listing's or a dequeue's. Queues, schedules, versions and in-flight workflows move in one transaction: a half-owned application would dequeue work whose version row it can no longer see. Terminal workflows and steps follow in key ranges, since they scope only observability and garbage collection and a long history should neither move in one transaction nor rescan what it already moved. Ranges rather than LIMIT, so an interrupted run resumes instead of repaging rows it already moved. The name rule moves to Validation, so DBOSConfig and the rename enforce one rule -- an operator must not be able to strand rows under a name no application could then be configured with. That also fixes a formatted() that bound to the wrong half of a concatenation and left %s in the message.
Adding the filter to getMetrics updated its call sites mechanically, which turned three Mockito stubs into getMetrics(any(), any(), null). Mixing a raw value with argument matchers is not allowed, so all three aborted before reaching the assertion they were written for. isNull() rather than any(), since what these tests are checking is that a Conductor request carrying no application_name reaches the system database as an unset filter.
maxdml
reviewed
Aug 27, 2026
kraftp
approved these changes
Aug 27, 2026
kraftp
left a comment
kraftp
left a comment
Member
There was a problem hiding this comment.
Let's also add this to https://github.com/dbos-inc/dbos-demo-apps/tree/main/interops
Collaborator
Author
this PR has to merge first |
Nothing in Transact requires an application name to match Conductor's
^[a-z0-9-_]{3,30}$: the column is TEXT, the value is always a bound
parameter, and the version hash just digests the bytes. Go, TypeScript,
and dbosctl do not check it at all.
So the rename DAO warns instead of throwing, and Validation classifies a
name rather than gating one. Launch still throws, but only when the name
is about to be used against something that rejects it -- DBOS Cloud, or a
configured Conductor key, whose websocket URL addresses the application
by name.
The rename keeps an explicit empty-name check, which the shape check had
been providing implicitly.
DBOS.lookupDebouncer now records a DeduplicationHolder where it used to record a bare workflow id. runDbosFunctionAsStep returns Object and casts, so the recorded payload has to describe its own type -- and portable_json does not. DBOSPortableSerializer passes an unrecognised object straight to Jackson with no type tag and reads it back into Object, so the record round-trips to a LinkedHashMap and toDeduplicationHolder threw. A workflow that debounced, collided, and then replayed under that serializer failed deterministically, with no way forward. The old bare id round-tripped under every serializer. toDeduplicationHolder is already the adapter for shape drift, so it takes the map too. Nothing is lost on the way out -- both fields are in the JSON -- only the type, which the map branch reconstructs. A map that is not a holder still raises. Python, TypeScript and Go cannot hit this: a TypedDict is a dict, a TS result is a plain object, and Go decodes into a statically known *DebounceResult from the call site. None of them has to rebuild a nominal type out of an untyped blob.
The comment said the field was ignored when creating a schedule, which always recorded the creating application. SchedulesDAO.scheduleOwner prefers it over the writing handle's own name, and DBOSClient.createSchedule takes nothing else, so the field is the only way a client creates a schedule another application will run. A caller trusting the comment could hand a schedule to a peer and be left with one that never fires here. Python resolves the owner the same way, from the schedule's own application_name, so the behaviour is the parity-correct one and the comment was simply stale. Queue carries the same wording and keeps it: QueueOptions has no applicationName, queueFromOptions writes null, and ownership reaches upsertQueue as a separate argument -- so there the record field really is read-back only.
devhawk
added a commit
that referenced
this pull request
Aug 28, 2026
A sweep of every public signature #471 changed turned up one more record developers construct: DBOSClient.EnqueueOptions, which gained a trailing applicationName. Add the constructor taking the pre-application-name argument list, forwarding a null owner, which enqueues for the enqueueing application. That sweep found 30 changed public signatures in all. The rest are either read-only outputs callers receive rather than construct (WorkflowStatus, StepInfo, VersionInfo) or internal plumbing: the conductor wire DTOs, the DAO layer, DbContext, SystemDatabase, DBOSExecutor, and the internal packages. Those are left to change in place.
devhawk
added a commit
that referenced
this pull request
Aug 28, 2026
Compatibility fallout from #471, which added application names and system database sharing. Two independent kinds, in one PR because they share that cause. The first three commits were opened before as #475, which was closed unmerged; they are unchanged here. ## Reading rows another application wrote Sharing a system database means a read can land on a row this runtime did not write, and two such reads failed outright. **Read an empty error column as no error** (`e07f89b`). The Go SDK stores a workflow's error as a non-nullable string, so a workflow that *succeeded* leaves `""` where this SDK writes NULL. Java parsed that as JSON and threw `MismatchedInputException: No content to map due to end-of-input` — on a workflow that never failed. TypeScript and Python already tolerate both forms. Applied where the column is read, not inside the deserializers, which should not have to know how a column came to be written. What this SDK writes is unchanged. **Don't try to deserialize a format we don't recognize** (`11897bc`). `getWorkflowStatus` threw `IllegalArgumentException: Serialization is not available` for any row in a format this runtime has no deserializer for. A workflow ID addresses the whole database, so a status read reaches other applications' rows by design, and what is wanted from one is the metadata. This is not only cross-SDK: two Java apps on one database, one with a custom serializer, hit it identically. `SerializationUtil.canDeserialize` now asks directly, and the reads that assemble a record from a row report unreadable payloads as null. Only where the payload is *one field of a record*. Where the payload is the answer it still throws — `getEvent`, a workflow's result, a recv'd message, a stream, and a recorded step result on replay. That last one matters most: a step that "returned null" because its output could not be read would corrupt the run it is replaying, so it has a test of its own. Three paths read a record but must not accept a null payload and check for themselves: running a workflow (refusing rather than invoking it with arguments it never had), exporting one, and importing one. ## Restoring constructors #471 broke #471 added `applicationName` to a number of public records. For a Java record that changes the *canonical constructor in place*, so every caller that built one positionally stopped compiling — including the DBOS conductor test app, whose CI cannot build against the published SDK ([failing run](https://github.com/dbos-inc/dbos-conductor/actions/runs/33212895492)): ``` App.java:277: error: no suitable constructor found for WorkflowSchedule(...) ``` **Restore constructors that omit the application name** (`2a7aeeb`) — `WorkflowSchedule`, `Queue`. **Restore input constructors that omit the application name** (`68297a2`) — `ListWorkflowsInput`, `GetStepAggregatesInput`, `GetWorkflowAggregatesInput`. **Restore the EnqueueOptions constructor that omits the application name** (`3054cc6`) — `DBOSClient.EnqueueOptions`. Each takes the pre-#471 argument list and forwards a null owner, which is the same default the existing no-arg constructors already use: the creating or enqueueing application for the records that record an owner, and "this application's rows plus unclaimed ones" for the filters. `GetWorkflowAggregatesInput` needed care — its two new components landed *mid-list* rather than appended, so that delegation reorders its arguments and passes `groupByApplicationName` as false. ### Scope A sweep of every public signature #471 changed (1,614 → 1,678 across all six published modules) found 30 changed in place. They fall out as: | | Disposition | |---|---| | The 6 records above | Constructors restored | | `WorkflowStatus`, `StepInfo`, `VersionInfo` | Left alone — callers receive these, they do not construct them | | 21 internal signatures | Left to change in place — conductor wire DTOs, the DAO layer, `DbContext`, `SystemDatabase`, `DBOSExecutor`, and the `internal` packages | Internal APIs deliberately get no compatibility overloads; they are fixed at their call sites when they change. Only external app code, which cannot be fixed that way, gets them. No public interface, enum, or abstract class was modified, so there are no implementor breaks, and no record component was removed or renamed — every change was an addition. `transact-cli` only gained a new class. ### Testing The overloads delegate positionally, where a mis-ordered argument would compile cleanly and silently scramble fields, so each is compared against the canonical constructor rather than spot-checked. Both new test classes are container-free. Verified non-vacuous by mutation: swapping two same-typed arguments in the `GetWorkflowAggregatesInput` and `EnqueueOptions` delegations makes the corresponding test fail. Note that this does not unblock conductor CI on its own — the test app resolves `dev.dbos:transact:+`, so it needs a new prerelease published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #468. Port of python#809, ts#1326, and go #444/#445.
Every DBOS object — workflows, steps, queues, schedules, application versions — records the application that owns it, so several applications can share one system database in isolation, or interoperate deliberately by naming each other's objects.
NULLmeans unclaimed. The columns are added nullable with no default, so every pre-existing row, and every row written by an SDK that does not know the column, belongs to all of them. Reads matchapplication_name = ? OR application_name IS NULLthroughout. That is what makes the migration safe, and also what makes the isolation one-directional: naming your applications hides them from each other, but not from an executor that ignores the column.Schema
Migrations 100–107, the start of a cross-SDK shared history: from index 100 on, every SDK defines the same migration at the same index. This language's own history (1–47) is padded out to 99, and both the runner and
dbos migrate --printskip the padding without a round trip each, recording it in a single write.application_nameonworkflow_status,queues,workflow_schedules,application_versions,operation_outputsenqueue_workflowgains a trailingapplication_name; every parameter is defaulted, so callers that omit it still resolve to itversion_name's global uniqueness (which is not dropped — not until every SDK reaching a shared database is past 107)What is scoped
Listings (workflows, queues, schedules, versions, aggregates, metrics), the queue dequeue and its rate-limit and concurrency counts, the latest-version lookup, and the delayed-workflow and garbage-collection sweeps. Recovery follows from the workflow listing — which matters, because
executor_iddefaults to the literallocalin all four SDKs, so two unnamed applications sharing a database would otherwise each treat the other's PENDING workflows as their own to recover.Deliberately not scoped:
DBOSQueueDuplicatedExceptionrather than trying to extend it — mirroring the foreign-holder arm of Python's_classify_bounce. Scoping the read instead would hide the holder while the global unique index still rejected the insert, and the retry loop would spin.Registering a queue, schedule, or application version under a name another application holds raises
DBOSApplicationNameConflictException— names are shared address space, so a collision is not ours to resolve. Claims are written asCOALESCE(application_name, ?), so a registration landing between the ownership check and the write keeps the name it just took.Enqueueing across applications
DBOS.enqueueWorkflowtakes the sameEnqueueOptionsasDBOSClientand writes the same row, so a Java application can enqueue a workflow implemented in another process or another language without holding a reference to its function.enqueuePortableWorkflowcarries named arguments, for targets like a Python workflow with keyword arguments.Called from inside a workflow it behaves like
startWorkflow: the enqueued workflow is recorded as a child under a derived ID, so a crash-replay collides with the original enqueue instead of enqueueing a second workflow. Called from a step it throws, as starting a workflow from a step already does.Two deliberate differences from
startWorkflow: the workflow name and queue are not checked against the local registries, since neither may exist here, and the application version is left unset unless given, so the owning application's own latest version picks the work up.DBOS.listQueuesandDBOS.listSchedulesgained the same application filter the client has.Behaviour changes worth knowing
DBOS_APP_NAMErather than from the config; ownership, the version hash, and the peer-ownership checks all key on that single identity, as Python threads oneGlobalParams.app_nameinto both.DBOSClienttakes an optional application name; without one it owns nothing and sees every application's rows.StepInfoexposes the application owning each step, and export/import round-trips it, so imported steps keep exactly the owner they were exported with rather than being re-stamped.application_nameon workflow, queue, schedule, and version rows, and accepts it as a filter on the list and aggregate requests. A Conductor that predates the field sends nothing, which leaves every listing scoped to the application.Not included
rename-application(re-owning rows after a rename, or adopting unclaimed ones) is a follow-up, matching Go. Until it lands, the collision error points at the Python or TypeScript CLI, which operates on the shared system database whatever wrote the rows.Queue and schedule deletion, and pause/resume, still address rows by name without an ownership predicate — matching Python and TypeScript, which do the same. Tightening that is cross-SDK work, not Java's to do alone.
Testing
ApplicationNameTestruns two applications against one system database: stamping across all five tables, listing isolation and the explicit cross-application filter, unclaimed rows visible to both, dequeue isolation, fork ownership, the three collision errors, version scoping, a nameless client, step ownership through export and import, both debouncers refusing a peer's deduplication key, an enqueue-by-name inheriting its parent's timeout, and a contested schedule batch rolling back whole.DebouncerHolderTestcovers holder ownership and the replay of aDBOS.lookupDebouncerstep recorded before this column existed.MigrationManagerTestcovers upgrading a database recorded at 47 through the padding to 107.🤖 Generated with Claude Code