Skip to content

Handle vector64 of structs when loading binary schemas and in reflection Verify - #9437

Open
professor-moody wants to merge 1 commit into
google:masterfrom
professor-moody:fix-bfbs-vector64-structs-and-verifier
Open

professor-moody wants to merge 1 commit into
google:masterfrom
professor-moody:fix-bfbs-vector64-structs-and-verifier

Conversation

@professor-moody

Copy link
Copy Markdown

A schema can declare a 64-bit vector of structs, for example big_struct_vector:[LeafStruct] (vector64) in tests/64bit/test_64bit.fbs, and flatc --schema -b writes it into the binary schema with base type reflection::Vector64 and the struct's object index. Two readers of binary schemas do not handle that base type yet.

Loading a binary schema (Type::Deserialize in src/idl_parser.cpp). is_series (src/idl_parser.cpp:4387-4388) lists reflection::Vector and reflection::Array but not reflection::Vector64, so for a vector64 of structs the object index is looked up in the enum table instead of structs_:

  • If no enum has that index, Parser::Deserialize returns false. For example, flatc -b tests/64bit/test_64bit.bfbs tests/64bit/test_64bit.json stops with "failed to load binary schema", while the same command with test_64bit.fbs succeeds.
  • If an enum does have that index, the schema loads, but the field gets that enum as enum_def and a null struct_def. Parsing JSON for the field through the loaded schema then reaches ParseTable(*val.type.struct_def, ...) (src/idl_parser.cpp:1458) with a null pointer.

This change adds reflection::Vector64 to is_series, so the element type resolves to the struct as it already does for Vector and Array.

Verifying a buffer with reflection (VerifyObject in src/reflection.cpp). The field switch in VerifyObject (src/reflection.cpp:176) has no case reflection::Vector64:. Such a field reaches the default case (src/reflection.cpp:262-264), which only contains FLATBUFFERS_ASSERT(false), so in a build with NDEBUG defined the field is not checked and flatbuffers::Verify returns true even when the field's 64-bit offset or element count points past the end of the buffer. The new case does what generated code does for a vector64 field (VerifyOffset64(verifier, VT_BIG_STRUCT_VECTOR) and verifier.VerifyVector(big_struct_vector()) at tests/64bit/test_64bit_generated.h:289-290): the element type must be a scalar or a struct, the 64-bit offset is checked with VerifyOffset64 (or VerifyOffset64Required for a required field), and the 64-bit element count times the element size must fit in the buffer (VerifyVectorOrString<uoffset64_t>).

What still works: the parser change only affects fields whose base type is reflection::Vector64, and the verifier change only adds checks for such fields, so well-formed schemas and buffers are not newly rejected (see Validation below). Binary schemas that contain a vector64 of structs now load and parse JSON to the same bytes as the text schema. What is now rejected by flatbuffers::Verify: a buffer whose vector64 field has an offset or an element count that reaches past the end of the buffer.

Tests. BinarySchemaVector64OfStructsTest in tests/test.cpp runs CheckVector64OfStructsSchema on a table with v:[S] (vector64) and w:[ubyte] (vector64), once without an enum and once with an enum whose index equals the struct's object index:

  • The schema is written with Parser::Serialize, read back with Parser::Deserialize, the field's struct_def is checked to be the struct, and { "v": [ { "a": 1 }, { "a": 2 } ], "w": [ 1, 2, 3 ] } parsed with the loaded schema must give the same bytes as the text schema. Without the fix this fails: the first schema does not load, and in the second the field has a null struct_def.
  • flatbuffers::Verify must accept the valid buffer (it does with and without the fix), and must reject a copy in which the vector64 offset of v or w is moved past the end of the buffer, and a copy in which the element count of v or w is set to 0xFFFFFFFF. Without the fix all eight rejections fail in a build with NDEBUG defined.

Validation at 6dda43b with clang 21.1.8, running the full flattests binary with no filter in a Release build (-O3 -DNDEBUG) and in an AddressSanitizer and UndefinedBehaviorSanitizer build (-O1 -g -DNDEBUG):

  • new test without the fix: 11 failed checks in each build, all in the new test;
  • new test with the fix: all tests pass in each build, with no sanitizer reports.

With the fix, Parser::Deserialize also accepts all 5 checked-in .bfbs files and 78 .bfbs files written by flatc --schema -b from the .fbs files under tests/ and samples/ (before the fix, the two copies of test_64bit.bfbs are rejected), and flatbuffers::Verify still accepts tests/64bit/test_64bit.bin against test_64bit.bfbs. flatc -b with test_64bit.bfbs now writes the same bytes as with test_64bit.fbs.

This touches the same files as #9430, #9432 and #9433 but in separate hunks. It applies cleanly before or after each of them, and the full test suite passes with all four changes applied together.

#9147 rewrites the is_series block in Type::Deserialize without adding reflection::Vector64, so the two do not apply on top of each other as is; whichever lands second needs the one added line carried over.

@github-actions github-actions Bot added c++ codegen Involving generating code from schema labels Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ codegen Involving generating code from schema

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant