Skip to content

Read rows other applications wrote without failing on their payloads - #475

Closed
devhawk wants to merge 2 commits into
mainfrom
skip-unrecognized-serialization
Closed

Read rows other applications wrote without failing on their payloads#475
devhawk wants to merge 2 commits into
mainfrom
skip-unrecognized-serialization

Conversation

@devhawk

@devhawk devhawk commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Two bugs a Java application hits when it shares a system database with applications it did not write. Both surfaced in the interops demo, where Java, Python, TypeScript and Go apps drive each other's workflows through one dbos schema.

Read an empty error column as no error

The Go SDK stores a workflow's error as a non-nullable string, so a workflow that succeeded leaves "" in error where this SDK writes NULL. Java treated any non-null value as an error and parsed it as JSON, so reading a perfectly healthy Go workflow threw:

MismatchedInputException: No content to map due to end-of-input

The other SDKs already tolerate both spellings — TypeScript coerces on the way in (row.error ? row.error : null), Python never reads the column unless the status says ERROR. SystemDatabase.errorOrNull does the same at each site that reads the column: a workflow's, a step's, and the transactional step schema's.

Empty, not blank. No SDK writes a non-empty run of whitespace here, so a blank-but-not-empty value is content to hand back rather than quietly discard. What this SDK writes is unchanged and stays conservative: an absent error is NULL.

Don't try to deserialize a format we don't recognize

getWorkflowStatus threw IllegalArgumentException: Serialization is not available on any row written in a format this runtime has no deserializer for. Workflow IDs address the whole system database, so a status read reaches another application's rows on purpose, and what is wanted from such a row is the metadata — who owns it, what it is, whether it finished. Losing all of it over a payload the caller may not have asked for is the wrong trade.

Not only an interop problem: two Java applications on one system database, one configured with a custom serializer and one not, hit exactly this. custom_base64 is as unreadable here as py_pickle is.

SerializationUtil.canDeserialize asks the question directly rather than catching a failure — the two built-in formats, plus whatever a configured custom serializer names. The reads that assemble a record from a row skip the payloads when the answer is no and report them as null.

Only where the payload is one field of a record — a workflow's status, a workflow's steps. Everywhere the payload is the answer still throws, because there is nothing else to hand back: getEvent, a workflow's result, a recv'd message, a stream, and a recorded step result on replay. That last one matters most, since a step that "returned null" because its output could not be read would corrupt the run it is replaying, so it has a test of its own.

Three paths read a record but must not accept a null payload, and check for themselves:

  • Running a workflow. The arguments are the point, and one invoked with arguments it never had is worse than one marked ERROR. executeWorkflowById refuses and names the format. The error is recorded in this runtime's own format: one we cannot read is one we cannot write, and serializing into it would throw and leave the workflow PENDING forever — the hang this refusal exists to prevent.
  • Exporting one. An export is imported back, so a payload dropped on the way out restores a workflow that never had it. Conductor can ask an application to export a peer's workflow, which is exactly when this bites.
  • Importing one. Import re-serializes every payload, so it needs a serializer for the recorded format as much as export did, and it writes a null payload straight through as NULL — a lossy batch would import as an emptied workflow rather than an error. Checked for the whole batch before the transaction opens.

Testing

  • 632 tests pass; spotlessCheck, javadoc and compileTestJava green.
  • New InteropTest cases cover both empty-error reads, peer rows in an unreadable format (status and listing), a custom serializer this app lacks, the export and import refusals, and the step-replay refusal.
  • New PortableSerializationTest case drives an ENQUEUED py_pickle row through the executor and asserts it lands ERROR naming the format, rather than hanging PENDING.
  • Verified end to end against the interops demo: 57 tests pass with the four-language apps sharing one system database, built against this branch.

🤖 Generated with Claude Code

Be liberal in what you accept. On a system database shared by several
applications the row may have been written by another SDK, and the Go SDK
stores a workflow's error as a non-nullable string — so a workflow that
succeeded leaves "" behind where this one writes NULL. Java parsed whatever was
there as JSON and threw `MismatchedInputException: No content to map due to
end-of-input`, on a workflow that never failed.

The other SDKs already read the two the same way. TypeScript coerces on the way
in (`row.error ? row.error : null`), and Python never reads the column unless
the status says ERROR.

Empty, not blank: no SDK writes a non-empty run of whitespace here, so a value
that is blank without being empty is content to hand back rather than quietly
discard.

Applied where the column is read — a workflow's and a step's alike, including
the transactional step schema — rather than inside the deserializers, which
should not have to know how a column came to be written. What this SDK writes
is unchanged and stays conservative: an absent error is NULL.
`DBOS.getWorkflowStatus` threw `IllegalArgumentException: Serialization is not
available` on any row written in a format this runtime has no deserializer for.
Workflow IDs address the whole system database, so a status read reaches rows
another application owns on purpose — and what is wanted from such a row is the
metadata: who owns it, what it is, whether it finished. Losing all of that over
a payload the caller may not have asked for is the wrong trade.

It is not only an interop problem. Two Java applications on one system database,
one configured with a custom serializer and one not, hit exactly the same thing:
`custom_base64` is as unreadable here as `py_pickle` is.

So `SerializationUtil.canDeserialize` asks the question directly — the two
built-in formats, plus whatever a configured custom serializer names — and the
reads that assemble a record from a row skip the payloads when the answer is no,
reporting them as null. A predicate rather than a caught exception: there is no
deserializer to be had, and trying and failing is a slower way to learn it.

Only where the payload is one field of a record — a workflow's status, a
workflow's steps. Everywhere the payload is the answer still throws, because
there is nothing else to hand back: getEvent, a workflow's result, a recv'd
message, a stream, and a recorded step result on replay. That last one matters
most — a step that "returned null" because its output could not be read would
corrupt the run it is replaying — so it has a test of its own.

Three paths read a record but must not accept a null payload, and check for
themselves:

  * Running a workflow. The arguments are the point, and one invoked with
    arguments it never had is worse than one marked ERROR, so executeWorkflowById
    refuses and names the format it would have taken. The error is recorded in
    this runtime's own format: one we cannot read is one we cannot write, and
    serializing into it would throw and leave the workflow PENDING forever, which
    is the hang this refusal exists to prevent.
  * Exporting one. An export is imported back, so a payload dropped on the way
    out restores a workflow that never had it. Conductor can ask an application
    to export a peer's workflow, which is exactly when this bites.
  * Importing one. Export and import are a single-SDK affair but not a
    single-configuration one, and import re-serializes every payload, so it needs
    a serializer for the recorded format as much as export did. It also writes a
    null payload straight through as NULL, so a lossy batch would import as an
    emptied workflow rather than an error. Checked for the whole batch before the
    transaction opens.
@devhawk

devhawk commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

adding some fixes to breaking changes introduced by #471

@devhawk devhawk closed this Aug 28, 2026
devhawk added a commit that referenced this pull request Aug 28, 2026
Compatibility fallout from #471, which added application names and
system
database sharing. Two independent kinds, in one PR because they share
that
cause.

The first three commits were opened before as #475, which was closed
unmerged;
they are unchanged here.

## Reading rows another application wrote

Sharing a system database means a read can land on a row this runtime
did not
write, and two such reads failed outright.

**Read an empty error column as no error** (`e07f89b`). The Go SDK
stores a
workflow's error as a non-nullable string, so a workflow that
*succeeded*
leaves `""` where this SDK writes NULL. Java parsed that as JSON and
threw
`MismatchedInputException: No content to map due to end-of-input` — on a
workflow that never failed. TypeScript and Python already tolerate both
forms.
Applied where the column is read, not inside the deserializers, which
should
not have to know how a column came to be written. What this SDK writes
is
unchanged.

**Don't try to deserialize a format we don't recognize** (`11897bc`).
`getWorkflowStatus` threw `IllegalArgumentException: Serialization is
not
available` for any row in a format this runtime has no deserializer for.
A
workflow ID addresses the whole database, so a status read reaches other
applications' rows by design, and what is wanted from one is the
metadata.
This is not only cross-SDK: two Java apps on one database, one with a
custom
serializer, hit it identically. `SerializationUtil.canDeserialize` now
asks
directly, and the reads that assemble a record from a row report
unreadable
payloads as null.

Only where the payload is *one field of a record*. Where the payload is
the
answer it still throws — `getEvent`, a workflow's result, a recv'd
message, a
stream, and a recorded step result on replay. That last one matters
most: a
step that "returned null" because its output could not be read would
corrupt
the run it is replaying, so it has a test of its own. Three paths read a
record
but must not accept a null payload and check for themselves: running a
workflow
(refusing rather than invoking it with arguments it never had),
exporting one,
and importing one.

## Restoring constructors #471 broke

#471 added `applicationName` to a number of public records. For a Java
record
that changes the *canonical constructor in place*, so every caller that
built
one positionally stopped compiling — including the DBOS conductor test
app,
whose CI cannot build against the published SDK
([failing
run](https://github.com/dbos-inc/dbos-conductor/actions/runs/33212895492)):

```
App.java:277: error: no suitable constructor found for WorkflowSchedule(...)
```

**Restore constructors that omit the application name** (`2a7aeeb`) —
`WorkflowSchedule`, `Queue`.
**Restore input constructors that omit the application name**
(`68297a2`) —
`ListWorkflowsInput`, `GetStepAggregatesInput`,
`GetWorkflowAggregatesInput`.
**Restore the EnqueueOptions constructor that omits the application
name**
(`3054cc6`) — `DBOSClient.EnqueueOptions`.

Each takes the pre-#471 argument list and forwards a null owner, which
is the
same default the existing no-arg constructors already use: the creating
or
enqueueing application for the records that record an owner, and "this
application's rows plus unclaimed ones" for the filters.
`GetWorkflowAggregatesInput`
needed care — its two new components landed *mid-list* rather than
appended, so
that delegation reorders its arguments and passes
`groupByApplicationName` as
false.

### Scope

A sweep of every public signature #471 changed (1,614 → 1,678 across all
six
published modules) found 30 changed in place. They fall out as:

| | Disposition |
|---|---|
| The 6 records above | Constructors restored |
| `WorkflowStatus`, `StepInfo`, `VersionInfo` | Left alone — callers
receive these, they do not construct them |
| 21 internal signatures | Left to change in place — conductor wire
DTOs, the DAO layer, `DbContext`, `SystemDatabase`, `DBOSExecutor`, and
the `internal` packages |

Internal APIs deliberately get no compatibility overloads; they are
fixed at
their call sites when they change. Only external app code, which cannot
be
fixed that way, gets them.

No public interface, enum, or abstract class was modified, so there are
no
implementor breaks, and no record component was removed or renamed —
every
change was an addition. `transact-cli` only gained a new class.

### Testing

The overloads delegate positionally, where a mis-ordered argument would
compile
cleanly and silently scramble fields, so each is compared against the
canonical
constructor rather than spot-checked. Both new test classes are
container-free.
Verified non-vacuous by mutation: swapping two same-typed arguments in
the
`GetWorkflowAggregatesInput` and `EnqueueOptions` delegations makes the
corresponding test fail.

Note that this does not unblock conductor CI on its own — the test app
resolves
`dev.dbos:transact:+`, so it needs a new prerelease published.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants