Skip to content

tools: add support for validation scripts in test files - #893

Open
fjl wants to merge 17 commits into
mainfrom
validation-script
Open

fjl wants to merge 17 commits into
mainfrom
validation-script

Conversation

@fjl

@fjl fjl commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

This adds a facility for defining 'validation scripts' in test files. Example file:

>> {"jsonrpc":"2.0","id":1,"method":"eth_gasPrice"}
<< {"jsonrpc":"2.0","id":1,"result":"0x1047435"}
--
if (BigInt(messages[1].response.result) <= 0) {
    throw new Error("gasprice too low");
}

The validation script runs after message exchanges are completed, and can access all data in the test,
as well as the server's responses. Scripts are executed during test filling (to verify the script is syntactically correct), and will also be run in the hive runner with the actual server responses.

To make this work, I have refactored the handling of .io files a bit and there is now a Go package
for working with them. This package will be used in hive to load the files and run the scripts.

@fjl
fjl force-pushed the validation-script branch from f0f6b0e to 03717b1 Compare September 17, 2026 22:07
@MysticRyuujin

Copy link
Copy Markdown
Contributor

A couple of things maybe worth fixing?

  1. Script timeout. RunScript runs goja with no interrupt or deadline. --sim.timelimit defaults to zero. We might want to consider a deadline via vm.Interrupt or something?

  2. Loader tests. #1613 deletes hive's two loader tests (the 24 MiB single-line case and the comment/speconly parsing case) along with the old parser, but they did not move into iofile_test.go. The large-line test exists because fixtures with big receipts once broke the default scanner buffer. Should we port those two into the iofile package so that coverage does not go away?

@fjl

fjl commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for feedback. I will check it about the loader test. I enlarged the token buffer a bit to make it work.

@fjl

fjl commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Regarding execution timeout, it's a good point. I will add this.

@lightclient lightclient left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! So ideally this will also solve ethereum/hive#1588 which it looks like you already commented on.

In that scenario, I think the way it will work is that we'll check the full schema for the tracer and if that passes we know the schema is correct. Then it should just be a matter of verifying that the type of tracer matches the expected, i.e. if we're trying to test opcode tracer we use the JS script to just check that the response was that tracer type and not another. We've already validated the schema so we know it is at least properly formed.

Does that seem correct @MysticRyuujin ?

@fjl

fjl commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

I have added a jsonschema.validate(schema, value) function to the scripting environment, and the script now has access to the OpenRPC spec through the openrpc global variable. So it is now theoretically possible to validate results against a specific object schema. Ideally, we can have a script like:

let schema = openrpc.components.schemas.OpcodeBlockTransactionTrace;
jsonschema.validate(schema, messages[1].response.result);

This doesn't work at the moment because we operate on the dereferenced spec and it doesn't contain the components.schemas anymore.

@fjl

fjl commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

Ahh, I realize that making this work also requires a different API for the jsonschema module in JS. In order to resolve references in the spec, the validation function needs access to the whole spec object. So it will look more like this:

let schema = "#/components/schemas/OpcodeBlockTransactionTrace";
jsonschema.validate(openrpc, schema, messages[1].response.result);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants