Skip to content

Improve performance on derivation group subscription - #1976

Open
JosephVolosin wants to merge 2 commits into
developfrom
fix/improve-derivation-group-load
Open

Improve performance on derivation group subscription#1976
JosephVolosin wants to merge 2 commits into
developfrom
fix/improve-derivation-group-load

Conversation

@JosephVolosin

@JosephVolosin JosephVolosin commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

This PR modifies the derivation group subscription to remove the external event count lookups. The change removes the aggregation count from the main Hasura subscription and adds it as a standalone query, GET_EXTERNAL_SOURCE_EXTERNAL_EVENT_COUNTS.

In our LRO deployment, we have been collecting live external event data for multiple months. These in total make up approx. 1 million external events across ~1,200 external sources. At this size, the subscription is very slow to complete and will hold up the UI. This is most notable in the 'Manage Derivation Group' modal in the plan view which will show 'No Derivation Groups' for an extended period of time while the subscription retrieves the source information. With some rough estimates using our data set, the current subscription takes approx. 90 seconds. With the change, the derivation group subscription takes approx. 3 seconds.

The event count can still be a helpful tool so to cover that this PR adds a new query, GET_EXTERNAL_SOURCE_EXTERNAL_EVENT_COUNTS which is called in the new component ExternalSourceDetails.svelte to gather the counts for the specific external source the component is created for. This component fulfills the same role as some prior code that was copy/pasted between the places the derivation group information was used.

Testing

  1. Upload at least 1 external source + external event schema
  2. Upload at least 1 external source that uses the uploaded source + event schemas
  3. Create a plan
  4. Open the External Sources panel and click Manage Derivation Groups
  5. Derivation groups should be show - this should be performant now even with lots of external events
  6. Hover over a derivation group and click the 'View' button
  7. Event counts should be shown next to the sources in the derivation group

Note there is some e2e test changes included as the common component now enforces a time format that doesn't match how some of these components formatted times previously.

@JosephVolosin
JosephVolosin force-pushed the fix/improve-derivation-group-load branch from e8c5211 to c095697 Compare July 21, 2026 20:45
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant