Repository navigation
Higher Memory Requirements for SPDX Validation in v2.0.3 #257
Description
With the release of the new v2.0.3 version it seems the tool trades performance for a greater memory usage.
For small files the performance improvements are great.
But for larger files the increased memory usage makes the tool less usable than before (v2.0.2).
| tested file | file size | v2.0.2 | v2.0.3 |
|---|---|---|---|
| sbom-source.spdx.json | 8 MB | 2m 45s | 0m 27s |
| sbom-build.spdx.json | 150MB | 1m53s | Out of memory |
For example, I got these results when running
java -Xmx15G -Xms12G -jar "tools-java-${SPDX_TOOLS_VERSION}-jar-with-dependencies.jar" Verify "$spdx_document"
via the validate-spdx-documents.sh script in spdx-validation-example.zip locally on my machine. In each run the tool received a memory limit of up to 15GB. While in v2.0.2 this was enough to validate the document in under 2 minutes now with v2.0.3 the validation process is killed due to being out of memory.
Reacted by Arthit Suriyawongkul
Activity
[-]Higher Memory requirements for SPDX Validation in v2.0.3[/-][+]Higher Memory Requirements for SPDX Validation in v2.0.3[/+]goneall commentedon Dec 4, 2025
From a quick look, this looks like an issue with the schema validation for the JSON-LD file.
This recently merged PR may help: #248
@augelu-tng - I just published a new release yesterday, if you haven't already, please try with the latestgoneall commentedon Dec 4, 2025
I just checked and it looks like the updated schema somehow didn't make it into the release - I may need to turn a new build.Stay tuned.
goneall commentedon Dec 4, 2025
I just found the inconsistency in the schemas - one of the library dependencies has an outdated version of the schema, but it should not affect the validation, so the new version should help.
I'm running with the latest - the schema validation is taking a very long time - I'll wait and see if it runs out of memory.
goneall commentedon Dec 4, 2025
I verified it is spending all of it's time and memory in the schema validation:
tools-java/src/main/java/org/spdx/tools/Verify.java
Line 184 in de4734d
cc: @JPEWdev
It looks like it is spending time in the
"anyOf"sub-schemas. I'll try to figure out which subschemas and if we can apply more"if"s to the schema logic to speed it up.goneall commentedon Dec 4, 2025
@augelu-tng I just realized from the title and comments you are using the latest.
As noted above, the changes are in the schema file and the performance problems are in the schema validator.
We're using ae-jsonschemafriend-core for validation unless we can find a more efficient JSON schema.
goneall commentedon Dec 4, 2025
It looks like the newtonsoft schema validation also runs out of memory
JPEWdev commentedon Dec 4, 2025
I'm curious why is takes more memory for "anyOf" vs. "oneOf"
goneall commentedon Dec 5, 2025
@JPEWdev - It looks like for this particular file the new schema is significantly slower than the schema version before we added the
"if"s. Not really sure why. I wonder if there were any changes between what I tested from the schema file posted in slack vs the schema file generated from the spdx-spec repo CI. I'll do some more investigating tomorrow.goneall commentedon Dec 5, 2025
Did some more experimenting - there may be a problem other than the schema. If I use the schema from the 2.0.2 version with the 2.0.3 version, it still seems to take a long time. There's a new version of the schema validator library - I suspect this may be an issue, so @JPEWdev - don't worry about any analysis for now. I'll try going back to the older version of the library tomorrow.
Revert json-schema-validator version to 1.5.9
goneall commentedon Dec 5, 2025
Thanks @augelu-tng for reporting this so quickly after a release
I just merged in a PR with a fix - reverting back to a previous version of the library
I also submitted this issue in the upstream library: networknt/json-schema-validator#1216
I'll spin another release shortly.
Revert json-schema-validator version to 1.5.9