Add upload and download simulation dataset feature - #1852
Open
remy-rabideau wants to merge 44 commits into
Open
Conversation
… for creating simulation datasets with profiles
…plement in PostgresPlanRepository and InMemoryPlanRepository
…nstead of individual parameters Consolidate uploadSimulationDataset parameters into a single SimulationResults object. Update Hasura action definition, parsers, and repository implementations. Add SimulationResultsParser to deserialize JSON format produced by SimulationResultsWriter, including parsing of duration strings, timestamps, profile segments, activities, and events. Update PostgresPlanRepository to extract data from SimulationResults and write
…on dataset uploads
…oadSimulationDataset JSON input Add simulationArguments Map<String, SerializedValue> field to SimulationResults with constructor overload for backward compatibility. Update SimulationResultsParser to parse optional simulationArguments field from JSON. Modify PostgresPlanRepository to use parsed simulationArguments instead of empty map when creating simulation datasets. Add unit tests verifying parsing with and without simulationArguments fiel
The parser now handles optional "topics" and "events" sections in the upload JSON. Topics are parsed as a map of name -> schema and converted to List<Triple<Integer, String, ValueSchema>> with synthesized indices. Flat events are grouped by (realTime, transactionIndex), then reconstructed into EventGraph<EventRecord> objects via EventGraphUnflattener.unflatten() for insertion into the database. Previously these fields were ignored, resulting in uploaded simulation datasets always having 0 events.
…on for retrieving simulation datasets with profiles
…rieving simulation datasets with profiles
…retrieving simulation datasets by plan and dataset ID
…trieving simulation datasets with profiles, activities, topics, events, and arguments
…tActivities methods from private to package-private in PostgresResultsCellRepository
…rs for converting SimulationResults to downloadable JSON format with profiles, activities, topics, and events
…ository and StubPlanService throwing UnsupportedOperationException
…d response wrapper
… and aerie_ui, and remove unnecessary build contexts and volumes for workers
…MerlinBindings `uploadSimulationDataset`: requires `simulate` permission `downloadSimulationDataset`: requires `resource_samples` permission
… for creating simulation datasets with profiles
…plement in PostgresPlanRepository and InMemoryPlanRepository
…nstead of individual parameters Consolidate uploadSimulationDataset parameters into a single SimulationResults object. Update Hasura action definition, parsers, and repository implementations. Add SimulationResultsParser to deserialize JSON format produced by SimulationResultsWriter, including parsing of duration strings, timestamps, profile segments, activities, and events. Update PostgresPlanRepository to extract data from SimulationResults and write
…on dataset uploads
…oadSimulationDataset JSON input Add simulationArguments Map<String, SerializedValue> field to SimulationResults with constructor overload for backward compatibility. Update SimulationResultsParser to parse optional simulationArguments field from JSON. Modify PostgresPlanRepository to use parsed simulationArguments instead of empty map when creating simulation datasets. Add unit tests verifying parsing with and without simulationArguments fiel
The parser now handles optional "topics" and "events" sections in the upload JSON. Topics are parsed as a map of name -> schema and converted to List<Triple<Integer, String, ValueSchema>> with synthesized indices. Flat events are grouped by (realTime, transactionIndex), then reconstructed into EventGraph<EventRecord> objects via EventGraphUnflattener.unflatten() for insertion into the database. Previously these fields were ignored, resulting in uploaded simulation datasets always having 0 events.
remy-rabideau
force-pushed
the
feature/upload-simulation-dataset
branch
from
July 28, 2026 18:16
d2501e8 to
9a0d32c
Compare
…ion directing users to ResponseSerializers.serializeSimulationResultsForDownload()
…es in uploadSimulationDataset to ensure all activity types in the dataset exist in the plan's mission model
… in simulation dataset
Increased maximum request size for uploads from 256 MB to 64 GB (per Clipper request)
remy-rabideau
commented
Jul 29, 2026
| final var javalin = Javalin.create(config -> { | ||
| config.showJavalinBanner = false; | ||
| if (configuration.enableJavalinDevLogging()) config.plugins.enableDevLogging(); | ||
| config.http.maxRequestSize = 64L * 1024 * 1024 * 1024; |
Author
There was a problem hiding this comment.
Per request by Clipper for large simulation result datasets
remy-rabideau
marked this pull request as ready for review
July 30, 2026 16:34
remy-rabideau
marked this pull request as draft
July 30, 2026 16:35
adrienmaillard
removed their request for review
July 30, 2026 16:51
…com/remy-rabideau/plandev into feature/upload-simulation-dataset
remy-rabideau
changed the title
remy-rabideau
marked this pull request as ready for review
July 30, 2026 17:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
REQUIRES_GATEWAY_PR="148"
NASA-AMMOS/plandev-gateway#148
Description
Adds a matched pair of Hasura actions for moving simulation datasets in and out of PlanDev:
uploadSimulationDataset(mutation) — accepts a simulation results JSON payload inSimulationResultsWriterformat and persists it to the database against an existing plan, returning the newsimulationDatasetId.downloadSimulationDataset(query) — reads an existing simulation dataset back out and returns it in that same JSON format.The two are deliberately symmetric: the format
downloadSimulationDatasetemits is the formatuploadSimulationDatasetaccepts, so a dataset can be downloaded from one plan and re-uploaded to another with round-trip fidelity. Two serializer tests (testSerializeEmptyResultsRoundTrips,testSerializeFullResultsRoundTrips) and one e2e test (validRoundTrip) pin that property.The primary use case for upload is transferring a simulation dataset generated through procedural stateless scheduling into the PlanDev database for viewing in the UI; ingesting results from external simulators is also supported. Download covers the reverse direction — exporting results for offline analysis, archival, or transfer between environments. The
SimulationResultsWriterformat is also produced by aerie-cli viaplans download-simulation-full-results.Pipeline (upload): Hasura receives the mutation → forwards the action envelope to merlin-server →
SimulationResultsParserparses the JSON →PostgresPlanRepository.uploadSimulationDatasetvalidates activity types, then writes the dataset, profiles, segments, activity spans, topics, and events in a single transaction → returns the newsimulationDatasetId.Pipeline (download): Hasura receives the query → merlin-server loads the dataset record, profiles, spans, topics, events, and simulation arguments →
ResponseSerializers.serializeSimulationResultsForDownloademits theSimulationResultsWriterJSON shape.Reviewer notes
Three things worth attention:
uploadSimulationDatasetis gated onsimulate, because uploading produces a simulation dataset — the same artifact a simulation run produces — rather than an external dataset.downloadSimulationDatasetis gated onresource_samples, the permission that already governs reading simulated resource data; theviewerrole holds it asNO_CHECK, so viewers can export results from plans they do not own. The e2e testsforbiddenandviewerMayDownloadcover both sides of this.SimulationResultsgained asimulationArgumentsfield. The existing 8-argument constructor is retained and delegates withMap.of(), so no existing call site changes.Changes:
uploadSimulationDatasetmutation anddownloadSimulationDatasetquery, theirUploadSimulationDatasetResponse/DownloadSimulationDatasetResponsetypes, and theSimulationResultsInput/SimulationResultsOutputscalarsaerie_adminanduser, download additionally permitsviewerSimulationResultsWriterJSON format: DOY timestamps, real/discrete profiles with named segments, simulated and unfinished activities, simulation arguments, topics, and events. Parse-only — the unparse direction throwsUnsupportedOperationExceptionpointing atResponseSerializers.serializeSimulationResultsForDownload, which is the real serializerserializeSimulationResultsForDownload(profiles, spans, simulation arguments, topics, and events, with event graphs flattened viaEventGraphFlattener) andserializeCreatedSimulationDatasetIdPOST /uploadSimulationDatasetandPOST /downloadSimulationDataset; parses the Hasura action envelope, applies the permission checks described above, and maps failures to status codeshasuraUploadSimulationDatasetActionPandhasuraDownloadSimulationDatasetActionPUploadSimulationDatasetInputandDownloadSimulationDatasetInputrecordsINVALID_SIMULATION_DATASETformatted erroruploadSimulationDatasetanddownloadSimulationDatasetto the service interface and implementationSimulationResultson download; the in-memory mock throwsUnsupportedOperationExceptionfor bothgetSimulationTopics,getSimulationEvents,getActivities,insertSimulationTopics,insertSimulationEvents, andpostActivitiesfromprivateto package-private soPostgresPlanRepositorycan reuse them instead of duplicating the queriessimulationArgumentsfield and a 9-argument constructor; the existing 8-argument constructor delegates with an empty mapVerification
Automated coverage added:
UploadSimulationDatasetParserTest(9 tests) — real profiles, activities, simulation arguments, topics and events, invalid timestamps, missing fields, and absence defaultsDownloadSimulationDatasetParserTest(7 tests) — valid envelopes, missingplanId/simulationDatasetId, non-numeric and beyond-int-range dataset ids, and missing session variablesDownloadSimulationDatasetSerializerTest(10 tests) — real and discrete profiles, simulated and unfinished activities, simulation arguments, topics and events, concurrent events within one transaction, plus the two round-trip testsUploadSimulationDatasetTests(e2e, 8 tests) — successful upload with empty, discrete, real, and mixed profiles; invalid plan id (404); missing required field (400); unauthorized user (403); unknown activity types rejected (400)DownloadSimulationDatasetTests(e2e, 7 tests) — invalid plan id, missing and nonexistent dataset id, forbidden role, empty profiles, viewer access, and a full upload → download round tripDocumentation
No existing documentation covers these endpoints. The
SimulationResultsWriterformat they read and write is defined in theaerie-clirepository (see NASA-AMMOS/aerie-cli#187).Future work
simulationDatasetIdon download — the lookup currently throws a bareRuntimeExceptionthat falls through to the catch-all handler (asserted as-is innonexistentSimulationDatasetId)select arguments from merlin.simulation_datasetstatement inPostgresPlanRepository.downloadSimulationDataset