From 67b9eb8243f95695c1c0b3ad634f2d17b9ae4e68 Mon Sep 17 00:00:00 2001 From: abdulsaheel Date: Sun, 26 Jul 2026 11:18:17 +0530 Subject: [PATCH] docs: rewrite the README to describe the worker that actually exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The code was slimmed to a companion in 2026-06-29 ("slim companion backend (free-tier)"). The README was not, and still opened with: "This is the part that does the actual thinking... The app on your phone is mostly a screen; the work happens here." None of that is true. There is no /ingest/batch, no decode.ts, no rollup.ts, no analytics cron, no minute table, no R2 raw archive. The worker is 749 lines across appconfig / auth / health / reads / telemetry, and index.ts's own header comment has been accurate the whole time — the README was the only thing still describing the old architecture. This matters beyond tidiness. OpenStrap's entire pitch, and a good part of its legal posture, is that health data is computed and stored on-device and no server ever sees it. A public repo in the same org claiming to receive raw frames and compute recovery server-side undercuts that, and this repo is linked from the org profile. Rewritten from the actual route table: what each endpoint is for, which are required (almost none), and that the two data-collecting endpoints are opt-in, off by default, and compiled out of store builds entirely. Keeps the self-host instructions and points at migrate_to_slim.sql and git history for anyone who wants the old design. --- README.md | 240 ++++++++++++++---------------------------------------- 1 file changed, 60 insertions(+), 180 deletions(-) diff --git a/README.md b/README.md index 883ca2d..5b5eef0 100644 --- a/README.md +++ b/README.md @@ -1,191 +1,71 @@ # OpenStrap backend -This is the part that does the actual thinking. The phone pulls raw bytes off the band -and ships them here. This server decodes those bytes, files the originals away untouched, -and on a schedule turns them into the numbers you actually look at: recovery, strain, -sleep, stress, all of it. The app on your phone is mostly a screen; the work happens here. - -It's one Cloudflare Worker. Not a fleet of microservices, not Kubernetes, one Worker with -a SQLite database (D1) and an object bucket (R2) hanging off it. That's on purpose, it -keeps the whole thing cheap enough to run on Cloudflare's free tier and simple enough that -one person can hold it in their head. - -> Not affiliated with WHOOP in any way. Run your own copy, or use mine. More on that at -> the bottom. - -## How a byte becomes a number - -Here's the whole life of your data, start to finish. - -Your phone connects to the band, drains whatever records it buffered to flash, and POSTs -them to `/ingest/batch` as raw hex strings. It never sends me a "heart rate of 72", it -sends me the actual frame the band emitted and I figure out the 72 myself. That matters -because if I get smarter about decoding later, I can re-run everything against the bytes I -already have. - -When a batch lands, `ingest.ts` does four things in order: - -1. **Rate-limits you.** A token bucket per user, 0.5 tokens a second, burst of 30. Stops - a runaway client from hammering the thing. Lives in the `rate_limit` table. -2. **Decodes the frames.** `decode.ts` walks each hex string, works out the packet type - and record type, and pulls out what it can: timestamp, heart rate, a motion magnitude, - a step increment, whether the band was on your wrist. The 1 Hz record (`parse_r24`, - borrowed from the protocol package) and the IMU-bearing R10 are the ones that carry - real signal. -3. **Saves the raw bytes to R2** under `raw/{you}/{device}/{when}-{first}-{last}.txt`, - one frame per line — the system of record for re-decoding. Retention is short and - layered: raw R2 objects expire at **14 days** (an R2 bucket lifecycle rule, matching - `RAW_RETENTION_DAYS` — the re-decode horizon), the per-minute `minute` table and the - device `events` table are pruned at **10 days**, and the *derived* tables (`daily`, - `sleep`, `baselines`) are **kept permanently** — that's what the long-window trend - metrics read from, so nothing needs re-decoding past 10 days. -4. **Rolls everything into minutes.** `rollup.ts` buckets the decoded samples by - `floor(ts/60)*60` and writes them to the `minute` table. - -The `minute` table is the one clever bit worth understanding before you touch anything. -It doesn't store an average heart rate, it stores the running pieces: `hr_sum`, `hr_n`, -`act_sum`, `act_n`, plus min/max. The upsert adds the new pieces onto whatever's already -there. The reason is that uploads aren't clean. The phone retries and batches overlap, so -a day's frames arrive split across many POSTs in no particular order. Storing running sums -makes the minute totals **order- and partition-independent**: chop the same frames into any -set of batches, in any order, and the minute rollups come out identical. If I stored -averages instead, the split would corrupt them. - -One honest caveat, because it's easy to misread this as "duplicate-proof": the upsert is -*additive*, so the **same frame delivered in two separate POSTs is counted twice.** This is -not idempotent against duplicate sends on its own — it relies on the client deduping frames -before they ever hit here. The app does exactly that: `raw_records` has the frame hex as a -primary key (`INSERT OR IGNORE`) and only deletes a record after a 200, so each frame is -POSTed once. Keep that contract; if you ever ingest from a client without it, dedup frames -server-side before the rollup. Don't break this. - -Once minutes are written, the user gets flagged dirty (or pushed onto a queue if you've -got the paid plan), and that's where ingest stops. The heavy math is deliberately not on -the request path. - -## Where the metrics actually get computed - -`analytics.ts` is the brain, and it runs on a cron. Two schedules, both in -`wrangler.toml`: - -- **Every 30 minutes** (`*/30`): a light sweep — re-derive every dirty user (their daily - numbers, sleep, incremental steps), and the moment a night actually finishes, kick off - that night's HRV. Cheap work only; the heavy R2 re-decodes are fanned out onto a queue, - one bounded `(user, day)` unit per consumer invocation. -- **Every night at 3:30** (`30 3`): the backstop — re-decode HRV and respiratory rate - only for the recent nights still *missing* them (so a night is never decoded twice, and - a night the wake-time run left empty gets retried), true up steps, and prune minute rows - past their retention. Anything already computed is skipped. - -HRV is real now: the beat-to-beat R-R intervals live in the 1 Hz (V24) records, so -`biometrics.ts` re-decodes them from the raw bytes in R2 (off the request path) to drive -recovery, readiness, and HRV-based stress. - -`processUser` is where it happens for one person. It reads their minutes, pulls their -baselines, and calls into the [analytics package](https://github.com/OpenStrap/analytics) -for each metric: resting HR, strain, HR zones, calories, sleep detection, sleep -regularity, workout detection, training load, fitness trend, readiness, the anomaly -signal, the coach plan, stress, nocturnal heart. The results land in `daily`, `sleep`, -`sessions`, and `baselines`. I keep this on a trailing window per day so the numbers -actually move day to day instead of collapsing into one flat value, which was a real bug -early on. - -The cross-day metrics that need long windows — training load (EWMA acute:chronic over -7/28 days), Banister fitness/fatigue/form, Foster monotony, sleep regularity — are -**seeded from the permanent `daily`/`sleep` tables**, not just the few days being -recomputed. We never re-decode old days (raw is gone after 14 days); we reuse the -already-derived rows, so a real 28-day ACWR or 14-night regularity actually exists. - -Every number comes back wrapped: a value, a unit, a confidence between 0 and 1, a tier, -and a label. If the inputs aren't there, the value is `null` and the confidence is `0`. I -would rather show you a dash than make something up. The whole project falls apart the -moment it starts inventing numbers, so it doesn't. - -## What's in src - -| File | What it's for | -|------|---------------| -| `index.ts` | The Hono app, the route table, and the cron handler | -| `auth.ts` | JWT signing/verifying, the email OTP flow, sending mail | -| `ingest.ts` | The whole ingest path above | -| `decode.ts` | Hex frames into decoded samples | -| `rollup.ts` | Decoded samples into per-minute buckets | -| `analytics.ts` | The cron brain, `processUser` / `runAnalytics` | -| `queue.ts` | The queue consumer: one bounded `(user, job, day)` unit per invocation, and the wake-time HRV trigger | -| `biometrics.ts` | HRV (RMSSD/SDNN/LF-HF), recovery, stress, relative temp/SpO₂ — re-decoded from the R-R intervals in R2 | -| `steps_imu.ts` | Step counting from the wrist accelerometer (incremental + nightly true-up) | -| `query.ts` | The read endpoints: today, sleep, strain, trends, chart, history | -| `daydetail.ts` | Single-day drill-downs (the strain curve, the hypnogram, the stress band) | -| `history.ts` | Range aggregation and the calendar heatmap | -| `records.ts` | Personal bests, streaks, resting-HR drift | -| `journal.ts` | Your tags and notes, plus a correlation engine that looks for what your tags do to your numbers | -| `notifications.ts` | The notification feed and marking things read | -| `resp.ts` | Respiratory rate from the optical PPG record (gated, only when there's real PPG) | -| `seed.ts` | Synthetic data generator for testing, runs in phases to stay under the free-plan request cap | -| `db/schema.sql` | The whole database, idempotent so you can re-run it | - -## The API - -Everything except sign-up needs `Authorization: Bearer `, and you only ever -see your own data, scoped by the user id baked into the token. The `/admin/*` routes need -the admin token instead. - -Sign in is passwordless: `POST /auth/register`, then `/auth/request-otp` mails you a -six-digit code, then `/auth/verify-otp` trades the code for an access token (24h) and a -refresh token (30d). `/auth/refresh` rotates them. If email isn't configured the code -comes back in the response so you never get locked out during setup. - -Pushing data: `POST /ingest/batch` with `{device_id, records: [hex...]}` and -`/ingest/events` for the device events. - -Reading it back: `/today`, `/sleep`, `/strain`, `/sessions`, `/trends`, `/chart`, -`/history`, the `/day/{strain,sleep,timeline,stress}` drill-downs, `/records`, `/journal` -(and `/journal/insights`), and `/notifications`. There's a `/profile` you can GET and -PATCH. - -Admin stuff for when you run your own: `/admin/run-analytics`, `/admin/run-resp`, -`/admin/seed-demo`, `/admin/issue-token`, `/admin/wipe-raw`, `/admin/prune`. - -## The database - -Thirteen tables. The ones you'll care about: `minute` (the running-sum rollups, pruned at -10 days), `daily` and `sleep` and `sessions` (the derived output, mostly JSON columns for -the structured bits like coach plans and HR zones), `baselines` (your resting HR, max HR, -sleep need, the anchors everything else is measured against), and the auth trio (`users`, -`otps`, `refresh_tokens`). Full DDL is in `src/db/schema.sql`, it's commented, go read it. - -## Running your own +> **This is optional. The app does not need it.** +> +> OpenStrap is local-first: [edge](https://github.com/OpenStrap/edge) pulls the bytes off +> the band over Bluetooth, decodes them with +> [protocol](https://github.com/OpenStrap/protocol), computes every metric with +> [analytics](https://github.com/OpenStrap/analytics), and stores the results on your +> phone. None of that involves this worker, or any other server. + +So what is this? A small companion worker backing a handful of narrow, mostly optional +features — and nothing else. It does **not** receive your raw frames, it does **not** +decode records, and it does **not** compute recovery, strain, sleep, or any other metric. +All of that used to live here. It doesn't anymore. + +## What it actually does + +| Endpoint | What it's for | Required? | +|---|---|---| +| `GET /app/status` | OTA pointer + in-app announcements | No — the app works offline | +| `GET /legal/terms` | Terms text behind the onboarding consent gate | No | +| `POST /auth/{request-otp,verify-otp,refresh}` | Sign-in **for legacy import only** | Only to import | +| `GET /profile`, `/strain`, `/sleep`, `/sessions` | One-time read of an old OpenStrap cloud account | Only to import | +| `POST /consent` | Consent ledger entry | No | +| `POST /telemetry` | **Opt-in** crash/usage telemetry — device-id keyed, no account | No, off by default | +| `POST /health/upload` | **Opt-in** health-data contribution (full local `.db`) | No, off by default | +| `DELETE /data` | Right to be forgotten — erases everything keyed to you | — | + +The two data-collecting endpoints (`/telemetry`, `/health/upload`) are opt-in, off by +default, and are compiled out entirely of App Store and Play Store builds. See +[edge's PRIVACY.md](https://github.com/OpenStrap/edge/blob/main/PRIVACY.md) for the full +two-tier policy. + +## Why it still exists + +Three reasons, none of them "computing your metrics": + +1. **Legacy import.** Early OpenStrap had a real cloud backend. People have data in it. + This lets them pull it down once and then never talk to a server again. +2. **An OTA/announcement pointer**, so a sideloaded app can find out there's a newer build. +3. **Opt-in contribution**, for people who actively want to donate data to improve the + sleep-staging and protocol work. + +## Run your own + +One Cloudflare Worker on the **free** plan — no Queues, no Cron, no Durable Objects, by +deliberate design (see the comments in `wrangler.toml`, and keep it that way). ```bash npm install -npx wrangler d1 create openstrap-db # paste the id into wrangler.toml -npx wrangler r2 bucket create openstrap-raw -npx wrangler d1 execute openstrap-db --file src/db/schema.sql -npx wrangler secret put JWT_SECRET # any long random string -npx wrangler secret put ADMIN_TOKEN # another one, for /admin/* -npx wrangler deploy +wrangler d1 create openstrap-v3-slim-db +wrangler r2 bucket create openstrap-v3-health +wrangler d1 execute openstrap-v3-slim-db --file=src/db/schema.sql +wrangler secret put JWT_SECRET +wrangler deploy ``` -If you want sign-in emails to actually send, add a `BREVO_API_KEY` or `RESEND_API_KEY` -and an `EMAIL_FROM`. Without them, sign-in still works, the code just comes back in the -API response. Secrets go through `wrangler secret` or `.dev.vars`, never into the repo. -The `wrangler.toml` in here has a placeholder where the D1 id goes, swap in yours. +Then point the app at your worker instead of mine. If you're self-hosting just for +yourself, set `STATIC_OTP` so it doesn't need an email provider — never set that on a +shared deployment. -## About your data, honestly +## History -You can run all of this yourself. That's the whole point of it being open and the backend -URL being a setting, not a constant. Stand up your own Worker, your own D1, your own R2, -and your health data never touches a machine you don't own. +This repo used to be the centre of OpenStrap: it ingested raw frames, stored them in R2, +rolled them into minutes, and computed everything on a cron. The architecture moved +on-device in mid-2026 and the server was cut down to what you see here. If you want the +old design, it's in `src/db/migrate_to_slim.sql` and the git history. -Or don't, and use mine. If you do: what am I going to do with your heart rate? Nothing. I -promise the only thing I'll ever do with it is make the decoders and the math better over -time. I'm not selling it, I'm not building a profile on you, I genuinely do not care that -you ran 5k on Tuesday. But you don't have to take my word for it, that's why the self-host -path exists. +--- -## It's not finished - -There are bugs in here. I know about some of them and not others. The stress thresholds -aren't well calibrated, the sleep stage estimator leans too hard on REM, respiratory rate -only shows up when there's live PPG to work with. If you find something wrong, open an -issue, I'll work through them. This gets better with more people poking at it, not less. +Not affiliated with, endorsed by, or connected to WHOOP. MIT licensed.