Skip to content

refactor(core): use sqlalchemy orm and table reflection in sql resource store - #1485

Merged
AlessandroPomponio merged 12 commits into
mainfrom
ap_1484_rework_metastore_orm
Oct 2, 2026
Merged

AlessandroPomponio merged 12 commits into
mainfrom
ap_1484_rework_metastore_orm

Conversation

@AlessandroPomponio

@AlessandroPomponio AlessandroPomponio commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR reworks the metastore's SQL layer to replace ad-hoc raw SQL string queries and manual table-existence bookkeeping with SQLAlchemy ORM constructs. The result is a cleaner, more maintainable, and more type-safe database layer that no longer depends on dialect-specific raw SQL for introspection.

High-level Changes

  • SQLAlchemy ORM queries throughout: All raw sqlalchemy.text(...) queries in the resource store (filtering, joining, traversal, relationship lookups) have been replaced with ORM-style constructs using Table, select, and_, column operators, and explicit aliases.
  • Schema reflection replaces manual table-existence checks: The _tables_exist_cache mechanism and the ensureExists parameter have been removed. Instead, create_sql_resource_store now returns a MetaData object that is cached via _reflected_metadata_cache, giving the store direct, typed references to its tables (_resources_table, _relationships_table) without extra round-trips.
  • table_exists_query and check_table_exists removed: The dialect-specific raw SQL helpers for checking table existence have been deleted along with the now-unnecessary statements.py export surface.
  • json_extract_field_as_string utility added: A new helper wraps the ->> JSON extraction operator in a cast(..., String) to prevent SQLAlchemy's JSON type processor from attempting to deserialise already-extracted scalar values, fixing a latent JSONDecodeError.
  • Pickle/unpickle support updated: __getstate__/__setstate__ now include the table references so they are properly dropped and re-reflected across Ray serialisation boundaries.
  • Tests updated: Existing tests are updated to match the new interface; new tests cover the resource store and updated SQL utility helpers.

Impact

These are internal implementation changes with no effect on the public ado API, CLI behaviour, or resource schemas. Performance is unchanged or slightly improved by eliminating redundant table-existence network queries. Any code that previously passed ensureExists to SQLResourceStore directly must be updated, but that parameter was not part of the public API.


AI Disclosure: The code changes and this PR description were generated using IBM Bob and were reviewed manually.

@AlessandroPomponio
AlessandroPomponio force-pushed the ap_1484_rework_metastore_orm branch from 22f78d2 to 9ac2ba9 Compare September 30, 2026 13:19
@AlessandroPomponio AlessandroPomponio added the ci Enables CI integration label Sep 30, 2026
@AlessandroPomponio
AlessandroPomponio marked this pull request as ready for review October 1, 2026 07:54
@AlessandroPomponio
AlessandroPomponio added this pull request to stack #1489 October 1, 2026 08:39
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
… constructs

Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
… constructs

Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
… constructs

Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
@AlessandroPomponio
AlessandroPomponio force-pushed the ap_1484_rework_metastore_orm branch from c2551f3 to 4228018 Compare October 1, 2026 10:35
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
Signed-off-by: Alessandro Pomponio <alessandro.pomponio1@ibm.com>
@AlessandroPomponio
AlessandroPomponio added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 7058fc1 Oct 2, 2026
19 checks passed
@AlessandroPomponio
AlessandroPomponio deleted the ap_1484_rework_metastore_orm branch October 2, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Enables CI integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(core): use sqlalchemy orm and table reflection in sql resource store

2 participants