[Feature Request]: Download a single database out of a multi-database archive

[kayvanaarssen]

Problem / Motivation

When a backup job covers several databases from one source, the result is a single unified TAR archive. Today the only way to get one database out of it is POST /api/storage/{id}/download-url, which returns a token for the entire archive, decrypted. There is no way to fetch just one dump.

Two consequences:

1. Disproportionate transfer for a routine task. Anyone running a multi-database job who needs a single dump has to pull and decrypt the whole archive - tens of GB moved to obtain a few hundred MB. On a busy shared database server this is the normal case, not an edge case.

2. It makes delegated access impossible. We are an IT company managing servers for clients and resellers. Some clients own a whole server; many own a single site, and therefore a single database, on a shared server - often with an external developer maintaining it. Those developers regularly need a dump before a deploy or to reproduce something locally, and today every one of those requests goes through us by hand (currently: mount the Restic repo, extract, put the file somewhere).

We would like to put a small portal in front of the DBackup API so they can serve themselves. The restore half already works: POST /api/storage/{id}/restore accepts a databaseMapping array with per-entry selected flags, so restoring exactly one database out of a shared archive is expressible today. The download half is the blocker - handing a client the whole archive would hand them every other client's data on that server.

Splitting into one job per database would solve it on paper, but we manage a large number of servers and most are not client-specific, so that multiplies job count across the whole estate to solve a problem that affects only a subset of databases.

Proposed Solution

Expose the per-database selection that the archive format already supports.

Preferred shape - extend the existing endpoint. POST /api/storage/{id}/restore-files already streams a selection as a gzipped tar to target: {kind: "download"}, and already mints single-use tokens via prepare: true. Adding a database dimension to its schema would cover this without a new surface:

POST /api/storage/{id}/restore-files
{
  "file": "backups/shared-mysql-01/2026-08-02.tar",
  "databases": ["customer_shop_prod"],   // new, alongside the existing `selections`
  "target": { "kind": "download" }
}

Alternative shape - a dedicated endpoint mirroring download-url, if that is cleaner:

POST /api/storage/{id}/download-database
{ "file": "...", "database": "customer_shop_prod", "decrypt": true }
-> { "url": "...", "expiresIn": "5 minutes", "singleUse": true }

In the UI, the natural home is the restore dialog: a "Download dump" action per database, matching the Download selection (.tar.gz) that directory sources already have.

On permissions, storage:download would be the intuitive gate. It would be genuinely useful if this worked without storage:restore, so a download-only integration does not need a key that can also write to databases. (Note that analyze and browse-archive currently require storage:restore, which makes a read-only integration hard to build.)

Why this may be smaller than it looks

Reading main at the time of writing, most of the mechanism appears to exist already:

• IndexDatabaseLine (k: "db" in src/lib/archive/types.ts) maps each dump 1:1 onto a physical entry with a byte offset and size, and v2 archives are seekable - so a single dump is addressable without reading the rest.
• ArchiveSelection already declares databaseNames?: string[].
• src/lib/archive/extract.ts already filters index.databases against selection.databaseNames - this is what a selective restore uses.
• restore-files already handles streaming a selection to a download target, including the single-use token path.

The gap seems to be that src/services/restore/file-restore.ts and the restore-files Zod schema are directory-oriented and have no notion of db entries. Apologies if this reading is off - it is from the public source, not from running it.

Alternatives Considered

• One backup job per database. Correct in isolation, but our servers host many clients' databases; this would multiply jobs across the whole estate to serve the subset that actually needs delegated access.
• Download the full archive in our own middleware, extract one dump, serve that. Possible today, but it transfers and decrypts the entire archive per request, and means our portal temporarily holds every client's plaintext data. That is precisely the exposure we are trying to design away.
• Restore to a scratch server, then dump from there. Extra infrastructure, an extra copy of client data at rest, and slow.
• Keep our existing Restic pipeline and build the portal against that instead. Restic gives us the file but not restore-into-a-database, which is the part of DBackup we actually want.

Area

API

Additional Context

Where we would eventually like to get to. The endpoint above unblocks us on its own - we are happy to build the client mapping and authorization layer in our own application. But the direction we are heading is a client-facing view with multi-level access: a client, reseller, or their developer linked to specific databases and/or servers, able to list, download, and where appropriate restore only those, with restore targets pinned by us. Access granularity down to a single database on a shared server is the requirement, because that is how a lot of our hosting actually looks - some clients have a whole server, others just one site on a shared one, maintained by their own developer.

We understand that is a data-model change and not something to ask for casually. If a smaller primitive were ever on the table, generic labels/tags on sources, jobs and databases would go a long way: we could attach our own client mapping without DBackup needing to know what a client is, and other users would get environment, team, or criticality tagging out of it. Label-scoped permissions would then be a natural later step. Not part of this request - just context for where a per-database download would lead.

Today's permission model is type-level (storage:restore grants restore on everything), so all scoping has to live in our middleware regardless. That is a trade-off we accept; the single-database download is the one piece we cannot work around.

Related: #137.

Happy to test against a branch or help however is useful.

[Skyfay]

Hey @kayvanaarssen

This is one of the best-researched feature requests I have received, thank you for taking the time to read the source before writing it.

Your reading is correct almost everywhere. IndexDatabaseLine does map each dump 1:1 onto a physical entry with an ordinal and a size, extract.ts already filters index.databases against selection.databaseNames, and restore-files already gates a download target on storage:download while everything else needs storage:restore. So the permission shape you are asking for is already how that endpoint behaves. Databases are in fact a simpler case than files, because small files get packed into shared bundles and have to be unpacked from a common entry, while a database dump always owns its entry exclusively.

Some history explains the gap you found. The seekable archive format is new, it shipped in v3.0.0 together with File & Folder Backups, and it was designed for exactly one problem: restoring a single file out of a 100 GB archive without downloading the whole thing. It was built for directory sources, and honestly I never once thought about pointing the same mechanism at database dumps. Your request is precisely that idea, and it is a very good one. A multi-database job has the identical problem, one dump out of a shared archive, and the format already solves it. I just never connected the two.

There is one thing you could not see from the public source, and it changes the size of this quite a bit.

Seekable v2 archives are only written when a job has at least one directory source. A database-only job, which is the overwhelming majority and exactly your case of a shared MySQL server with many customer databases, takes a different path in the runner and produces a flat tar with no index and no seekability at all. That is not a deliberate decision against databases, it is simply where the feature stopped when it was built. So the machinery you found is real, but the archives you would be pointing it at do not currently use it. That is why this is not a schema change on restore-files. The actual work is making database-only jobs write v2 archives in the first place.

I would rather do that than bolt a special case onto the old format, because everything else follows from it. Once a multi-database job produces a real v2 archive, the per-database download is close to free, and you also get browsing a backup's contents before restoring, per-entry checksums, and a real size per dump rather than just a total.

On where this leads, and its relation to #137. I had been thinking of that issue and this one as separate, and they are not. Per-database scoping is not realistic on the current format. On a flat tar you have a list of names in the sidecar and nothing else. On v2 you have names, sizes and byte offsets as indexed data. So the format work is the prerequisite, and after it the scoping question becomes a data model question rather than an archive question.

I also want to be honest that per-database scoping is a genuinely harder problem than per-job scoping, for reasons that are worth naming now rather than discovering later:

• The permission unit stops being the file. Scoping by job means an archive is shown or hidden as a whole, which is cheap. Scoping by database means one archive is visible to forty customers at once, each seeing a different subset of it, which the current listing and its cache are not shaped for.
• Restore is the dangerous half. A download is a read, so validating the requested database name is enough. A restore writes into a live shared server, so the target name has to be validated independently, otherwise one customer can restore over another customer's database. That is a data loss bug rather than an information leak, and it deserves more care than the read path.
• Deletion, locking and retention stay admin-only for shared archives. Nobody can delete an archive that holds thirty-nine other customers' data.

None of that is a reason not to do it. It is a reason to build it on top of a format that carries per-database identity properly, which is the same piece of work this issue needs.

One thing I could do sooner, if it helps you in the meantime. Even on the current flat tar, the server can stream the archive, decrypt it, and emit only the requested dump. That still reads the whole archive from storage, so it does not solve the transfer cost, but it does mean your portal never holds anyone else's plaintext. Reading your issue, that exposure is the part you actually cannot work around, and this would close it on every existing backup without waiting for a format change. Tell me if that reading is right and I will consider it as an interim step.

The analyze and browse-archive endpoints requiring storage:restore is a straightforward oversight, both are read-only. I will fix that independently of everything above.

On labels and tags as a smaller primitive: I like that idea a lot, and it fits the direction well. Keeping DBackup out of the business of knowing what a customer is, while still letting you attach your own mapping, is the right instinct. I will keep it in mind when the scoping work gets designed.

No timeline promises. I am finishing the Docker volume backup implementation now and had the DBackup agent planned next, but this reframing moves it up in my thinking, because it turns out to unblock two things rather than one.

I will take you up on the offer to test against a branch when there is something worth testing.

If you think of anything else on this topic in the meantime, I'd love for you to share it.

Thanks :)

[kayvanaarssen]

On mobile just to hook into the streaming whole archive. No issues on bandwidth. But its more time related. If you need to do a restore the faster the better, since there might be major downtime (please don't say backup again before etc.) we do it happens been there done that you never know how it goes. Okay back to the topic.

Restores need to be as fast as possible. So if we have a server that has for example 10GB of archive. And we only need to restore a DB of a client that is 200MB loading the 10GB how fast the lines might be, are a waste of maybe valueable time.

Also I would make the offert to have a 1:1 meeting with you over which ever platform you like to explain our usecase as an MSP more. So you get some more real world insight. Again willing to put in the time and testing.

Check my git, company you will easy find me online hit me up on the platform you like best. Also on Discord in many servers as Kavaa

Also in discord of Ploi (Hosting panel we use) https://discord.gg/ploi-465764339174080512

Where many other users are that do hosting just an example but i'm also in Homelab: https://discord.gg/homelab

[Skyfay]

Speaking also for #137

Thanks for the detail, that genuinely helps.

On the call: I would rather keep this in writing. English is not my first language and I am more precise in text than in a live conversation, and written exchanges leave something I can come back to when I actually sit down to build. If a faster back and forth helps you, my Discord is https://dc.skyfay.ch and you are welcome to DM me from there. I would still want anything we work out to land back in these issues, which you suggested yourself, so that other people with similar needs can read along and push back where they see it differently.

One thing I want to set out plainly, because it shapes what I can realistically promise.

DBackup is built for self-hosters and homelabs, and for small to mid-sized companies running their own infrastructure. That is the audience it is designed around and the one I test against. MSP work is a different world with different requirements, and I do not have day to day exposure to your side of it, which is exactly why the detail you are providing is useful to me.

The technical ground is not the new part. I work as a system engineer, so running servers, taking backups and doing restores under time pressure are part of my day job rather than a side interest. When I say something is hard, that is a statement about the trade-offs involved, not a sign that the scenario is unfamiliar to me. I would rather say that once and clearly than let it sit unspoken underneath the rest of the conversation.

Where that leaves us: if there are features an MSP would benefit from that also make sense for the project as a whole, I am genuinely open to building them, and this looks like exactly that kind of feature. It improves multi-database restores for everyone, not only for your setup. What I cannot do is commit to a requirement set or a timeline. This is free software that I maintain alongside a full time job, and anything I add has to be something I can support for every installation out there, not only for one deployment shape.

So please keep the input coming, it is welcome and it has already changed how I look at the archive format.

To move the access question forward, there is still one thing I need from your last answer. You described the granularity well, whole server for some resellers, restore and download only for others, single backups on a shared server for the rest. What I still do not know is whether filtered visibility on one shared instance would be enough, meaning everyone logs into the same DBackup and each only sees their own resources, or whether you need genuine separation with their own user management and audit trail per customer. That distinction decides whether this is a feature I can build or whether it lands in the multi-tenancy territory I ruled out earlier.

Thank you for your understanding :)

[kayvanaarssen]

Thanks, again on mobile so answer will be somewhat shorter/more direct.

Agree on all points i'll try to do a full writeup tomorrow. The Netherlands time!

The goal is to run 1 DBackup instance for all servers etc. Or maybe 2 one internal one external etc. But that aside. So all custeromers would login to the same one and indeed based on the scope/role they see things or they don't so on reseller servers thats mostly easy: Run Job, Restore Backups, Run manual backup of specific DB, see history etc. The whole deal based on server ID (Server > Jobs > Databases) level to keep things simple i think.

Shared is bit more manual but we can automate this.

User X is linked to server 3838 with Database/Site 48483 what ever the number is. There the user can kick off a backup of that db specific with runtime now, can see history from that specific DB, and restore/download the DB.

This can also be multiple if its a normal end customer, could be they have more than one site/company so its not reseller or something. So when they would login they see x number of jobs/DB's and they can do the approved actions on those.

So that's a bit of the MSP standpoint we would use it now.

And to be honnest we have internal Devs but did not have the time to get around this in our internal Portal. We also use React as frontend. Laravel as backend for us but the idea is already there for some time. And now saw this project, and was about on par with what we need. And thus way i dove into the API part since no multi-tenant / level was available now. So we would just run your DBackup and make a frontend/wrapper around it with the API for our client portal.

That would be the fastes way for us now without changing to much right away on DBackup so the seperate (loose) download of databases via the API would help and thus not streaming the whole archive.