Skip to content

Expose per-record delete/forget, external-id lookup, and relationships via the REST server (API parity) #75

Description

@dcfocus

Motivation

The REST server added in #66 exposes only a subset of what the core library and
the Python Context API already support. Applications that adopt the server as
their integration surface cannot perform several first-class record operations
and must fall back to the in-process API or open the Lance dataset directly.

Current server routes

GET    /api/v1/health
POST   /api/v1/contexts                      # create
GET    /api/v1/contexts                      # list
GET    /api/v1/contexts/{name}
DELETE /api/v1/contexts/{name}               # delete whole context
POST   /api/v1/contexts/{name}/records       # add batch
GET    /api/v1/contexts/{name}/records       # list
GET    /api/v1/contexts/{name}/records/{id}  # get by internal id
POST   /api/v1/contexts/{name}/search
POST   /api/v1/contexts/{name}/retrieve
GET    /api/v1/contexts/{name}/version
POST   /api/v1/contexts/{name}/checkout
POST   /api/v1/contexts/{name}/compact
GET    /api/v1/contexts/{name}/compact/stats

Capabilities present in the library but missing from the server

The core ContextStore / Python Context provide these, with no equivalent
server route:

  • Per-record delete / forget: delete_by_id, delete_by_external_id, and
    Python forget(...) logically delete a record by writing a tombstone. The
    server can delete an entire context, but not a single record.
  • Get by external id: get_by_external_id(...) resolves the caller-supplied
    stable identifier documented in the Python API and README. The server's
    records/{id} route resolves internal ids only.
  • Relationship traversal: list_related(...) / Python related(...) from
    the relationships feature is not exposed through REST.

The same gap exists at the lance-context-api trait level, which currently
declares only add / get / list / search / retrieve / version / checkout / compact / compaction_stats.

Proposal

Bring the server, remote client, and API trait to parity with the core library
for these existing record-level operations:

  • DELETE /api/v1/contexts/{name}/records/{id}: logically delete a record by
    internal id.
  • DELETE /api/v1/contexts/{name}/records?external_id=...: logically delete a
    record by caller-supplied external id.
  • GET /api/v1/contexts/{name}/records/by-external-id?external_id=...: fetch a
    record by external id.
  • GET /api/v1/contexts/{name}/records/related?target_id=...&relation=...&limit=...:
    list records with relationships pointing at a target, honoring lifecycle
    visibility options.

Query parameters are preferable for external_id and relationship targets
because these caller-supplied identifiers often look like source URIs or paths
and may contain /, #, or ?.

Scope for a first PR

  • Add the routes above with request/response DTOs consistent with existing
    records/search handlers.
  • Extend the lance-context-api trait and remote client so callers can use the
    same operations through either local or remote stores.
  • Add focused tests for delete by id, delete by external id, external-id lookup,
    and related-record traversal.

Non-goals

Backward compatibility

Purely additive; existing routes and payloads are unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions