Repository navigation
reflection: verify nested_flatbuffer and flexbuffer UByte vectors in Verify - #9433
Open
professor-moody wants to merge 1 commit into
Open
professor-moody wants to merge 1 commit into
professor-moody wants to merge 1 commit into
Conversation
…Verify
## Summary
Generated C++ table verifiers (such as `VerifyMonsterBuffer` in `tests/monster_test_generated.h`) validate `[ubyte]` vectors annotated with `nested_flatbuffer` and, `else if` present, `[ubyte]` vectors annotated with `flexbuffer` via `flexbuffers::VerifyNestedFlexBuffer(vec, v)`.
In contrast, `flatbuffers::Verify` in `src/reflection.cpp` handled `case reflection::UByte:` inside `VerifyVector` by checking only the outer vector framing (`v.VerifyVector(flatbuffers::GetFieldV<int8_t>(table, vec_field))`) without inspecting `vec_field.attributes()`. As a result, a buffer whose `nested_flatbuffer` or `flexbuffer` `[ubyte]` vector contained a truncated or corrupt payload was rejected by generated verifiers such as `VerifyMonsterBuffer` but accepted by `flatbuffers::Verify(schema, *schema.root_table(), buf, len)`.
## Fix
1. **`src/reflection.cpp` (`LookupObjectByName`, `VerifyVector`, and `VerifyObject`)**:
- Include `"flatbuffers/flex_flat_util.h"`.
- Add `LookupObjectByName`, which resolves a `nested_flatbuffer` target type name against `schema.objects()` by walking enclosing namespace prefixes of `parent_obj.name()` from innermost to outermost (`parent_name.substr(0, dot + 1) + name` via `schema.objects()->LookupByKey(qualified.c_str())`) before falling back to `schema.objects()->LookupByKey(name)` for global or already-qualified names.
- Pass `const reflection::Object& parent_obj` from `VerifyObject` (`case reflection::Vector:`) into `VerifyVector`.
- Separate `case reflection::UByte:` in `VerifyVector` so that after `v.VerifyVector(vec)` succeeds on `flatbuffers::GetFieldV<uint8_t>(table, vec_field)` and when `vec` and `vec_field.attributes()` are non-null:
- If `vec_field.attributes()->LookupByKey("nested_flatbuffer")` is present, require a non-null `nested_attr->value()`, resolve `nested_obj` via `LookupObjectByName(schema, parent_obj, nested_attr->value()->c_str())`, require `nested_obj && !nested_obj->is_struct()` and `v.Check(vec->size() >= FLATBUFFERS_MIN_BUFFER_SIZE)`, construct `Verifier nested_v(vec->Data(), vec->size())`, forward `nested_v.SetFlexReuseTracker(v.GetFlexReuseTracker())`, verify the root offset via `nested_v.VerifyOffset<uoffset_t>(0)`, and recursively verify the nested table via `VerifyObject(nested_v, schema, *nested_obj, flatbuffers::GetAnyRoot(vec->Data()), /*required=*/true)`.
- Otherwise (`else if (vec_field.attributes()->LookupByKey("flexbuffer"))`), verify the FlexBuffer payload via `flexbuffers::VerifyNestedFlexBuffer(vec, v)`.
2. **`tests/reflection_test.cpp` (`ReflectionTest`)**:
- Add unit tests in `ReflectionTest` covering valid `testnestedflatbuffer` and `flex` vectors, 1-byte truncated (`{0}`) and 16-byte corrupt `testnestedflatbuffer` vectors, 1-byte invalid `flex` vectors, and `LookupObjectByName` edge cases (unresolvable object name `"Unknown"`, struct target `"Vec3"`, and global table `"TableA"` resolved via fallback `LookupByKey(name)`).
## Scope and Behavior Notes
- **Binary schema builtin attributes (`--bfbs-builtins`)**: Because `nested_flatbuffer` and `flexbuffer` are builtin schema attributes, `flatc` serializes them into `.bfbs` binary schemas only when `--bfbs-builtins` (`binary_schema_builtins`) is enabled (as used to generate `tests/monster_test.bfbs`). On `.bfbs` schemas serialized without `--bfbs-builtins`, `vec_field.attributes()` does not contain these builtin attributes and `VerifyVector` only validates the outer `[ubyte]` vector framing.
- **Nested verifier limits (known limitation)**: `Verifier nested_v(vec->Data(), vec->size())` constructs a sub-verifier with default `Verifier::Options (max_depth = 64, max_tables = 1000000)` and recurses once per nested FlatBuffer level; it does not inherit the caller's `max_depth` or `max_tables` budgets passed to `flatbuffers::Verify` (whereas generated `VerifyNestedFlatBuffer` forwards the caller's `opts_`, and open PR google#9418 updates only the template `VerifyNestedFlatBuffer` path in include/flatbuffers/verifier.h, so propagating caller depth and table budgets through `src/reflection.cpp` would require a separate follow-up).
## Coordination Note
- This PR modifies `src/reflection.cpp` and `tests/reflection_test.cpp`, which are also touched by open PR google#9430 (root/vtable/offset/union bounds in `src/reflection.cpp`). The hunks in this PR are placed at non-overlapping anchors and apply cleanly (`git apply --check` exit `0`) both on clean `master` (`6dda43bbaa9961ad913ce949f8ebd08f3062fae2`) and on top of google#9430.
- Related open PRs: google#9418 (template `VerifyNestedFlatBuffer` depth/table budget propagation in include/flatbuffers/verifier.h) and google#9429 (`src/reflection.cpp` nested buffer alignment during resize/copy).
## Testing
- Built and ran `flattests` (`ReflectionTest`) at `6dda43bbaa9961ad913ce949f8ebd08f3062fae2` with this change applied across four CMake configurations (`Release`, `Debug`, `Release` + ASan/UBSan, and `Debug` + ASan/UBSan): `ALL TESTS PASSED` (`exit 0`) in all four configurations.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generated C++ table verifiers (such as
VerifyMonsterBufferintests/monster_test_generated.h) validate[ubyte]vectors annotated withnested_flatbufferand,else ifpresent,[ubyte]vectors annotated withflexbufferviaflexbuffers::VerifyNestedFlexBuffer(vec, v).In contrast,
flatbuffers::Verifyinsrc/reflection.cpphandledcase reflection::UByte:insideVerifyVectorby checking only the outer vector framing (v.VerifyVector(flatbuffers::GetFieldV<int8_t>(table, vec_field))) without inspectingvec_field.attributes(). As a result, a buffer whosenested_flatbufferorflexbuffer[ubyte]vector contained a truncated or corrupt payload was rejected by generated verifiers such asVerifyMonsterBufferbut accepted byflatbuffers::Verify(schema, *schema.root_table(), buf, len).Fix
src/reflection.cpp(LookupObjectByName,VerifyVector, andVerifyObject):"flatbuffers/flex_flat_util.h".LookupObjectByName, which resolves anested_flatbuffertarget type name againstschema.objects()by walking enclosing namespace prefixes ofparent_obj.name()from innermost to outermost (parent_name.substr(0, dot + 1) + nameviaschema.objects()->LookupByKey(qualified.c_str())) before falling back toschema.objects()->LookupByKey(name)for global or already-qualified names.const reflection::Object& parent_objfromVerifyObject(case reflection::Vector:) intoVerifyVector.case reflection::UByte:inVerifyVectorso that afterv.VerifyVector(vec)succeeds onflatbuffers::GetFieldV<uint8_t>(table, vec_field)and whenvecandvec_field.attributes()are non-null:vec_field.attributes()->LookupByKey("nested_flatbuffer")is present, require a non-nullnested_attr->value(), resolvenested_objviaLookupObjectByName(schema, parent_obj, nested_attr->value()->c_str()), requirenested_obj && !nested_obj->is_struct()andv.Check(vec->size() >= FLATBUFFERS_MIN_BUFFER_SIZE), constructVerifier nested_v(vec->Data(), vec->size()), forwardnested_v.SetFlexReuseTracker(v.GetFlexReuseTracker()), verify the root offset vianested_v.VerifyOffset<uoffset_t>(0), and recursively verify the nested table viaVerifyObject(nested_v, schema, *nested_obj, flatbuffers::GetAnyRoot(vec->Data()), /*required=*/true).else if (vec_field.attributes()->LookupByKey("flexbuffer"))), verify the FlexBuffer payload viaflexbuffers::VerifyNestedFlexBuffer(vec, v).tests/reflection_test.cpp(ReflectionTest):ReflectionTestcovering validtestnestedflatbufferandflexvectors, 1-byte truncated ({0}) and 16-byte corrupttestnestedflatbuffervectors, 1-byte invalidflexvectors, andLookupObjectByNameedge cases (unresolvable object name"Unknown", struct target"Vec3", and global table"TableA"resolved via fallbackLookupByKey(name)).Scope and Behavior Notes
--bfbs-builtins): Becausenested_flatbufferandflexbufferare builtin schema attributes,flatcserializes them into.bfbsbinary schemas only when--bfbs-builtins(binary_schema_builtins) is enabled (as used to generatetests/monster_test.bfbs). On.bfbsschemas serialized without--bfbs-builtins,vec_field.attributes()does not contain these builtin attributes andVerifyVectoronly validates the outer[ubyte]vector framing.Verifier nested_v(vec->Data(), vec->size())constructs a sub-verifier with defaultVerifier::Options (max_depth = 64, max_tables = 1000000)and recurses once per nested FlatBuffer level; it does not inherit the caller'smax_depthormax_tablesbudgets passed toflatbuffers::Verify(whereas generatedVerifyNestedFlatBufferforwards the caller'sopts_, and open PR verifier: propagate max_depth and max_tables budgets across VerifyNestedFlatBuffer #9418 updates only the templateVerifyNestedFlatBufferpath in include/flatbuffers/verifier.h, so propagating caller depth and table budgets throughsrc/reflection.cppwould require a separate follow-up).Coordination Note
src/reflection.cppandtests/reflection_test.cpp, which are also touched by open PR Harden reflection Verifier against out-of-bounds reads on truncated buffers, vtable offsets, and unions #9430 (root/vtable/offset/union bounds insrc/reflection.cpp). The hunks in this PR are placed at non-overlapping anchors and apply cleanly (git apply --checkexit0) both on cleanmaster(6dda43bbaa9961ad913ce949f8ebd08f3062fae2) and on top of Harden reflection Verifier against out-of-bounds reads on truncated buffers, vtable offsets, and unions #9430.VerifyNestedFlatBufferdepth/table budget propagation in include/flatbuffers/verifier.h) and Fix reflection buffer-editing bugs and alignment in reflection and the Dart, Rust and Swift builders #9429 (src/reflection.cppnested buffer alignment during resize/copy).Testing
flattests(ReflectionTest) at6dda43bbaa9961ad913ce949f8ebd08f3062fae2with this change applied across four CMake configurations (Release,Debug,Release+ ASan/UBSan, andDebug+ ASan/UBSan):ALL TESTS PASSED(exit 0) in all four configurations.