Skip to content

reflection: verify nested_flatbuffer and flexbuffer UByte vectors in Verify - #9433

Open
professor-moody wants to merge 1 commit into
google:masterfrom
professor-moody:fix-reflection-verify-nested-ubyte-vectors
Open

professor-moody wants to merge 1 commit into
google:masterfrom
professor-moody:fix-reflection-verify-nested-ubyte-vectors

Conversation

@professor-moody

Copy link
Copy Markdown

Summary

Generated C++ table verifiers (such as VerifyMonsterBuffer in tests/monster_test_generated.h) validate [ubyte] vectors annotated with nested_flatbuffer and, else if present, [ubyte] vectors annotated with flexbuffer via flexbuffers::VerifyNestedFlexBuffer(vec, v).

In contrast, flatbuffers::Verify in src/reflection.cpp handled case reflection::UByte: inside VerifyVector by checking only the outer vector framing (v.VerifyVector(flatbuffers::GetFieldV<int8_t>(table, vec_field))) without inspecting vec_field.attributes(). As a result, a buffer whose nested_flatbuffer or flexbuffer [ubyte] vector contained a truncated or corrupt payload was rejected by generated verifiers such as VerifyMonsterBuffer but accepted by flatbuffers::Verify(schema, *schema.root_table(), buf, len).

Fix

  1. src/reflection.cpp (LookupObjectByName, VerifyVector, and VerifyObject):

    • Include "flatbuffers/flex_flat_util.h".
    • Add LookupObjectByName, which resolves a nested_flatbuffer target type name against schema.objects() by walking enclosing namespace prefixes of parent_obj.name() from innermost to outermost (parent_name.substr(0, dot + 1) + name via schema.objects()->LookupByKey(qualified.c_str())) before falling back to schema.objects()->LookupByKey(name) for global or already-qualified names.
    • Pass const reflection::Object& parent_obj from VerifyObject (case reflection::Vector:) into VerifyVector.
    • Separate case reflection::UByte: in VerifyVector so that after v.VerifyVector(vec) succeeds on flatbuffers::GetFieldV<uint8_t>(table, vec_field) and when vec and vec_field.attributes() are non-null:
      • If vec_field.attributes()->LookupByKey("nested_flatbuffer") is present, require a non-null nested_attr->value(), resolve nested_obj via LookupObjectByName(schema, parent_obj, nested_attr->value()->c_str()), require nested_obj && !nested_obj->is_struct() and v.Check(vec->size() >= FLATBUFFERS_MIN_BUFFER_SIZE), construct Verifier nested_v(vec->Data(), vec->size()), forward nested_v.SetFlexReuseTracker(v.GetFlexReuseTracker()), verify the root offset via nested_v.VerifyOffset<uoffset_t>(0), and recursively verify the nested table via VerifyObject(nested_v, schema, *nested_obj, flatbuffers::GetAnyRoot(vec->Data()), /*required=*/true).
      • Otherwise (else if (vec_field.attributes()->LookupByKey("flexbuffer"))), verify the FlexBuffer payload via flexbuffers::VerifyNestedFlexBuffer(vec, v).
  2. tests/reflection_test.cpp (ReflectionTest):

    • Add unit tests in ReflectionTest covering valid testnestedflatbuffer and flex vectors, 1-byte truncated ({0}) and 16-byte corrupt testnestedflatbuffer vectors, 1-byte invalid flex vectors, and LookupObjectByName edge cases (unresolvable object name "Unknown", struct target "Vec3", and global table "TableA" resolved via fallback LookupByKey(name)).

Scope and Behavior Notes

  • Binary schema builtin attributes (--bfbs-builtins): Because nested_flatbuffer and flexbuffer are builtin schema attributes, flatc serializes them into .bfbs binary schemas only when --bfbs-builtins (binary_schema_builtins) is enabled (as used to generate tests/monster_test.bfbs). On .bfbs schemas serialized without --bfbs-builtins, vec_field.attributes() does not contain these builtin attributes and VerifyVector only validates the outer [ubyte] vector framing.
  • Nested verifier limits (known limitation): Verifier nested_v(vec->Data(), vec->size()) constructs a sub-verifier with default Verifier::Options (max_depth = 64, max_tables = 1000000) and recurses once per nested FlatBuffer level; it does not inherit the caller's max_depth or max_tables budgets passed to flatbuffers::Verify (whereas generated VerifyNestedFlatBuffer forwards the caller's opts_, and open PR verifier: propagate max_depth and max_tables budgets across VerifyNestedFlatBuffer #9418 updates only the template VerifyNestedFlatBuffer path in include/flatbuffers/verifier.h, so propagating caller depth and table budgets through src/reflection.cpp would require a separate follow-up).

Coordination Note

Testing

  • Built and ran flattests (ReflectionTest) at 6dda43bbaa9961ad913ce949f8ebd08f3062fae2 with this change applied across four CMake configurations (Release, Debug, Release + ASan/UBSan, and Debug + ASan/UBSan): ALL TESTS PASSED (exit 0) in all four configurations.

…Verify

## Summary

Generated C++ table verifiers (such as `VerifyMonsterBuffer` in `tests/monster_test_generated.h`) validate `[ubyte]` vectors annotated with `nested_flatbuffer` and, `else if` present, `[ubyte]` vectors annotated with `flexbuffer` via `flexbuffers::VerifyNestedFlexBuffer(vec, v)`.

In contrast, `flatbuffers::Verify` in `src/reflection.cpp` handled `case reflection::UByte:` inside `VerifyVector` by checking only the outer vector framing (`v.VerifyVector(flatbuffers::GetFieldV<int8_t>(table, vec_field))`) without inspecting `vec_field.attributes()`. As a result, a buffer whose `nested_flatbuffer` or `flexbuffer` `[ubyte]` vector contained a truncated or corrupt payload was rejected by generated verifiers such as `VerifyMonsterBuffer` but accepted by `flatbuffers::Verify(schema, *schema.root_table(), buf, len)`.

## Fix

1. **`src/reflection.cpp` (`LookupObjectByName`, `VerifyVector`, and `VerifyObject`)**:
   - Include `"flatbuffers/flex_flat_util.h"`.
   - Add `LookupObjectByName`, which resolves a `nested_flatbuffer` target type name against `schema.objects()` by walking enclosing namespace prefixes of `parent_obj.name()` from innermost to outermost (`parent_name.substr(0, dot + 1) + name` via `schema.objects()->LookupByKey(qualified.c_str())`) before falling back to `schema.objects()->LookupByKey(name)` for global or already-qualified names.
   - Pass `const reflection::Object& parent_obj` from `VerifyObject` (`case reflection::Vector:`) into `VerifyVector`.
   - Separate `case reflection::UByte:` in `VerifyVector` so that after `v.VerifyVector(vec)` succeeds on `flatbuffers::GetFieldV<uint8_t>(table, vec_field)` and when `vec` and `vec_field.attributes()` are non-null:
     - If `vec_field.attributes()->LookupByKey("nested_flatbuffer")` is present, require a non-null `nested_attr->value()`, resolve `nested_obj` via `LookupObjectByName(schema, parent_obj, nested_attr->value()->c_str())`, require `nested_obj && !nested_obj->is_struct()` and `v.Check(vec->size() >= FLATBUFFERS_MIN_BUFFER_SIZE)`, construct `Verifier nested_v(vec->Data(), vec->size())`, forward `nested_v.SetFlexReuseTracker(v.GetFlexReuseTracker())`, verify the root offset via `nested_v.VerifyOffset<uoffset_t>(0)`, and recursively verify the nested table via `VerifyObject(nested_v, schema, *nested_obj, flatbuffers::GetAnyRoot(vec->Data()), /*required=*/true)`.
     - Otherwise (`else if (vec_field.attributes()->LookupByKey("flexbuffer"))`), verify the FlexBuffer payload via `flexbuffers::VerifyNestedFlexBuffer(vec, v)`.

2. **`tests/reflection_test.cpp` (`ReflectionTest`)**:
   - Add unit tests in `ReflectionTest` covering valid `testnestedflatbuffer` and `flex` vectors, 1-byte truncated (`{0}`) and 16-byte corrupt `testnestedflatbuffer` vectors, 1-byte invalid `flex` vectors, and `LookupObjectByName` edge cases (unresolvable object name `"Unknown"`, struct target `"Vec3"`, and global table `"TableA"` resolved via fallback `LookupByKey(name)`).

## Scope and Behavior Notes

- **Binary schema builtin attributes (`--bfbs-builtins`)**: Because `nested_flatbuffer` and `flexbuffer` are builtin schema attributes, `flatc` serializes them into `.bfbs` binary schemas only when `--bfbs-builtins` (`binary_schema_builtins`) is enabled (as used to generate `tests/monster_test.bfbs`). On `.bfbs` schemas serialized without `--bfbs-builtins`, `vec_field.attributes()` does not contain these builtin attributes and `VerifyVector` only validates the outer `[ubyte]` vector framing.
- **Nested verifier limits (known limitation)**: `Verifier nested_v(vec->Data(), vec->size())` constructs a sub-verifier with default `Verifier::Options (max_depth = 64, max_tables = 1000000)` and recurses once per nested FlatBuffer level; it does not inherit the caller's `max_depth` or `max_tables` budgets passed to `flatbuffers::Verify` (whereas generated `VerifyNestedFlatBuffer` forwards the caller's `opts_`, and open PR google#9418 updates only the template `VerifyNestedFlatBuffer` path in include/flatbuffers/verifier.h, so propagating caller depth and table budgets through `src/reflection.cpp` would require a separate follow-up).

## Coordination Note

- This PR modifies `src/reflection.cpp` and `tests/reflection_test.cpp`, which are also touched by open PR google#9430 (root/vtable/offset/union bounds in `src/reflection.cpp`). The hunks in this PR are placed at non-overlapping anchors and apply cleanly (`git apply --check` exit `0`) both on clean `master` (`6dda43bbaa9961ad913ce949f8ebd08f3062fae2`) and on top of google#9430.
- Related open PRs: google#9418 (template `VerifyNestedFlatBuffer` depth/table budget propagation in include/flatbuffers/verifier.h) and google#9429 (`src/reflection.cpp` nested buffer alignment during resize/copy).

## Testing

- Built and ran `flattests` (`ReflectionTest`) at `6dda43bbaa9961ad913ce949f8ebd08f3062fae2` with this change applied across four CMake configurations (`Release`, `Debug`, `Release` + ASan/UBSan, and `Debug` + ASan/UBSan): `ALL TESTS PASSED` (`exit 0`) in all four configurations.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ codegen Involving generating code from schema

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant