Implement endpoint to retrieve extension changes feed

[gnugomez]

There is currently no robust way to track extension removals or lifecycle for mirroring, auditing, or integrations since deleted extensions (and their versions) are removed from the database. A reliable and efficient endpoint to retrieve registry changes is needed.

This feature will enable easier monitoring for security researchers.

Related to EclipseFdn/open-vsx.org#7867

[gnugomez]

My proposal here would be to:

GET /api/-/changes

Parameter	Type	Description
offset	int	Number of entries to skip (default: 0).
size	int	Max entries to return (default: 100).
since	ISO-8601 timestamp	Return only entries with lastUpdated >= since.
until	ISO-8601 timestamp	Return only entries with lastUpdated < until.

since and until can be combined to define a time window. offset and limit paginate within that window.

Sample response

Each row will represent a version with it's metadata. If an extension gets updated a new entry gets added, if an older version gets removed the old version record gets updated and the state is changed to REMOVED together with an updated lastUpdated date.

{
  "offset": 0,
  "totalSize": 1000,
  "extensions": [
    {
      "namespace": "foo",
      "name": "bar",
      "version": "1.2.3",
      "targetPlatform": "linux-x64",
      "state": "REMOVED",
      "lastUpdated": "2024-06-01T12:34:56Z"
      ... rest of metadata
    },
    {
      "namespace": "bar",
      "name": "foobar",
      "version": "3.2.1",
      "targetPlatform": "linux-x64",
      "state": "ACTIVE",
      "lastUpdated": "2022-06-01T12:34:56Z"
      ... rest of metadata
    }
    ...
  ]
}

[gnugomez]

We would need to update ExtensionService#deleteExtension to mark state = REMOVED and update the timestamp instead of deleting the row.

[rchatrath7]

@gnugomez love this idea!

if an older version gets removed the old version record gets updated and the state is changed to REMOVED together with an updated lastUpdated date.

I'd like to propose that a new record is inserted so that the event feed remains append-only. This way you can canonically reconstruct the events over time in the registry starting from the beginning. More importantly, if maintaining a live replica, you can easily track removals of existing versions based on a last stored offset rather than having to pull down previous events, which will work very nicely with the since and until params (love these!)

[mixmix]

Yes, very keen to see this. I agree that it'd be great if this was append-only, and the ordering is then oldest-newest such that "offset" is stable over time (will always refer to the same entry).

We probably need to agree what "changes" we'd like to track:

• new package/ version/ platform (version built for a particular architecture)
• removal of a package/ version/ platform
• change of metadata to a package/ version/ platform (e.g. maintainers)

e.g. (sketched in a lot of detail to hopefully stimulate some discussion. )

{
  offset: 9000,
  totalSize: 1500345,
  changes: [
    { id: 9000, timestamp: 1779754099, type: 'NEW_PACKAGE',       data: { namespace: 'redhat', name: 'java-utils' } },
    { id: 9001, timestamp: 1779754100, type: 'UNPUBLISH_VERSION', data: { namespace: 'mix', name: 'demo', version: '1.19.20' } }
    { id: 9002, timestamp: 1779754101, type: 'NEW_VERSION',       data: { namespace: 'mix', name: 'demo', version: '1.19.21' } },
    { id: 9003, timestamp: 1779754101, type: 'NEW_PLATFORM',      data: { namespace: 'mix', name: 'demo', version: '1.19.21', platform: 'linux-x64' } },
    { id: 9004, timestamp: 1779754255, type: 'UPDATE_METADATA',   data: { /* not sure what metadata updates need to be supported so format unclear */ } }
  ]
}

I don't personally know if we need to see all the metadata for a particular package in here... or if we do I'd like you to show what exactly that metadata is.

[mixmix]

One problem with a feed like this is how large it gets over time. Specifically

• if you wanted to ingest the whole lot and get up to date, how do you do that efficiently?
• how do you know where the current HEAD offset is? (is that your totalSize?)

Some registries take an approach which is like a live feed combined with archived/ bundled epochs. e.g. when we get to 1,000,000 updates 0-999,999 are considered the first "epoch" and are bundled into /api/-/changes/epoch/1.tar.gz, then the default offset changes to 1,000,000 (until that epoch is also archived). Perhaps before premature over-engineering, the prudent way start is for results to have feedVersion: 1 included, and if that version ever changes you know the feed format has changed/ the append-only log has been wiped, and you need to re-fetch everything.

[gnugomez]

@mixmix @rchatrath7 thanks for getting involved ◡̈

I see you're leaning more towards an immutable stream of events. The initial idea here was to return a single entry that would get updated per extension/version.

Since right now extensions are hard-deleted we cannot get this data from the past, following your approach would also mean that the data returned by this endpoint and the new table we will need to build will contain a lot of data. This could become hard to manage, but I see it has it's advantages.

Is there any specific reason why you think this approach makes more sense?

[mixmix]

We're solving for the case of "a record was just added/mutated/deleted, we want to know about that ASAP so that we can update our local records and alert users of potential malware". A stream of events solves for that case very well.

Perhaps I didn't understand what you were proposing. Were you suggesting that each entry in your changes feed would be "the most recent mutation" for a particular package/version/platform? So I could query entry at offset 41,005 and it would always be for the same "record" and i'd be able to see the state there? If that's what you meant, I think there would have to be entries for namespaces, for packages, for each version, and for each targetPlatform of a package-version, because potentially each of these could be mutated independently? Can you clarify your proposal?

I think we're in agreement that at minimum we need "a source of info for removed records". What use case are you solving for?

[gnugomez]

Yeah so that basically was the idea, you nailed it.

It would be an endpoint returning all the uploaded versions of all extensions (each version+target will be a different entry) and it's current status sorted by updated date. Newer versions will represent a new entry. If an extension is deleted, the existing entry gets updated, meaning that the endpoint will sort it upwards.

Right now we don't have any event's system implemented, so the easiest way for us to expose that would be this one.

[mixmix]

Challenges I see with your proposal

1. sorted by updated-date?
   • this would mean the offsets would be non-constant and could be changing under you, so offset option is very unclear to me
   • I guess you'd order from newest -> oldest and paginate till you hit a date you'd already seen?
2. your list is also very large
   • assuming most packages/versions/targets are NOT mutated/deleted, your list of entries will be about the same length as the one I proposed (a total list of events)

If you agree with (2), then I would much prefer a total list of events, because anyone could replay + audit the events.

I think we have two ideas here which might meet both our needs. My usual preference for this sort of thing would be to jump on a call and put all possible solutions on the table, and both talk around the strengths/ merits/ challenges of each. It might be good to have someone from OpenVSX who has the context around cost of implementing given the current setup. Outside of that, then perhaps we focus on writing up the requirements we have, so they can come up with a solution internally

[mixmix]

I've reached out to a contact at OpenVSX for feedback.
FWIW, I'm in New Zealand timezone. What time zone are you in @gnugomez ?

Edit: Apologies, I hadn't realised you worked for OpenVSX @gnugomez !