Skip to content

fix: four silent-failure paths — backend flag ignored, cloud add 400s, failed chunks checkpointed, LongMemEval token budget - #24

Open
christian4423 wants to merge 4 commits into
mem0ai:mainfrom
christian4423:fix/silent-failure-paths
Open

fix: four silent-failure paths — backend flag ignored, cloud add 400s, failed chunks checkpointed, LongMemEval token budget#24
christian4423 wants to merge 4 commits into
mem0ai:mainfrom
christian4423:fix/silent-failure-paths

Conversation

@christian4423

Copy link
Copy Markdown

Four independent fixes, all found while running BEAM and the mem0-cloud path. Each one turns a
failure into something that looks like a legitimate result, so none of them are visible in the
output they corrupt. Happy to split into separate PRs if you'd prefer.

Follow-up to #23, which fixed the same token-budget defect in the BEAM runner.


1. --backend is unreachable whenever MEM0_BACKEND is set

backend = os.getenv("MEM0_BACKEND", args.backend)

--backend is declared with default="oss", so args.backend is always truthy and the getenv
default is never reached — the environment wins unconditionally. Since load_dotenv(override=True)
pushes .env into the environment, a stale line in a committed .env silently beats an explicit
flag on the command line, and nothing in the logs or results records which backend actually ran.

Cost me ten minutes of a paid run pointed at the wrong backend. Now args.backend or os.getenv("MEM0_BACKEND") or "oss", in both runners. Unchanged when the flag is omitted.

2. Cloud add() posts to /v3/memories/, which is a retrieval endpoint

Returns 400 Filters are required — so --backend cloud ingestion was non-functional, not
merely degraded. Add is /v1/; retrieval correctly stays on /v3/. Both are now env-overridable
so the harness isn't pinned to one API generation. A follow-on: /v1/ wraps its acknowledgement
in a list, so the dict-shaped handler raised AttributeError on the first successful add — also
fixed.

3. A failed ingestion chunk is checkpointed and never retried

chunks_already_done.add(chunk_key) sits outside the if response is not None branch, so a chunk
that failed is recorded as completed and --resume skips it forever. The failure is counted and
logged, but by the time anyone reads that warning the gap is permanent — the only record of which
chunks are missing is the checkpoint that just claimed them.

We hit this on BEAM 1M: one conversation ingested at 6.5 memories/chunk against 15.9 and
15.8 for its two siblings — same domain, same chunk count, slightly higher fact density.
Re-ingesting brought it to 15.8 (1,703 → 4,143 memories), confirming ~55% had been silently
dropped. total_chunks_failed read 0 the entire time.

4. LongMemEval answerer/judge inherit max_tokens=4096

answerer.generate (:593, :759) and judge_llm.generate_structured (:700, :816) pass no
max_tokens, inheriting LLMClient's 4096. --answerer-model and --judge-model both default to
gpt-5, and reasoning models bill reasoning against max_completion_tokens — so a long answer
exhausts the budget on reasoning alone and returns an empty string, which scores 0.0 and is
indistinguishable from a wrong answer.

⚠️ This is the path your published 91.0% is measured through — the README states the LongMemEval
runs used GPT-5 as answerer and judge. On BEAM the identical fix moved five arms by +0.010 to
+0.047
and took empty answers from 20 to 0. Worth a re-run before the next publication.


Verification

Scope is deliberately narrow: three files, no new dependencies, no behaviour change when the
relevant flags and env vars are left alone.

christian4423 and others added 4 commits August 8, 2026 21:10
    backend = os.getenv("MEM0_BACKEND", args.backend)

`--backend` is declared with `default="oss"`, so `args.backend` is always truthy
and the `os.getenv` default is never reached. The environment therefore wins
unconditionally and the CLI flag does nothing whenever MEM0_BACKEND is present.

This is quiet and expensive. `load_dotenv(override=True)` pushes .env into the
environment, so a stale line in a committed .env silently beats an explicit flag
typed on the command line, and neither the logs nor the results record which
backend actually ran. I lost a paid run to it: `--backend cloud` spent ten
minutes against the previous backend before the mismatch became obvious.

Precedence is now explicit — CLI flag, then MEM0_BACKEND, then the "oss" default:

    backend = args.backend or os.getenv("MEM0_BACKEND") or "oss"

Same fix in both runners (benchmarks/beam/run.py, benchmarks/longmemeval/run.py).
Behaviour is unchanged when the flag is omitted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cloud add path posted to POST /v3/memories/ and failed 400 on every call with
"Filters are required and cannot be empty. Please refer to .../get-memories".
Probed against api.mem0.ai on 2026-08-07:

  POST /v1/memories/         200  [{"status":"PENDING","event_id":...}]   <- add
  POST /v3/memories/         400  filters required (v3 POST is RETRIEVAL, not create)
  POST /v3/memories/search/  200  {"results":[...]}                       <- search
  POST /v1/memories/search/  400  v1 expects user_id, not the filters payload

Only ADD was mis-versioned; SEARCH at v3 was always correct and is unchanged in
behaviour (its version is now merely parameterised at the same default). The
surrounding add code already expected v1 semantics — it reads event_id from the
response — so the v3 URL was the mismatch.

Second symptom on the same path: v1 wraps its acknowledgement in a list,
[{"status":"PENDING","event_id":...}], while the code called .get() on it directly
and raised "'list' object has no attribute 'get'" — retried five times, then
returned None. Now accepts either envelope.

Both versions are overridable via MEM0_CLOUD_ADD_VERSION / MEM0_CLOUD_SEARCH_VERSION
so this does not hard-code today's API surface.

Verified end to end through Mem0Client itself (not curl): add returns an event_id,
the memory appears via the retrieval endpoint, and search returns it with score
0.6302. Note mem0 processes asynchronously — the probe took ~3 minutes from add
acknowledgement to becoming searchable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`chunks_already_done.add(chunk_key)` and the checkpoint write sit outside the
`if response is not None` branch, so a chunk whose ingestion failed is recorded
as completed. `--resume` then skips it forever.

The failure is counted in total_failed and logged, but by the time anyone reads
that warning the gap is already permanent: the only record of which chunks are
missing is the checkpoint that just claimed them as done. Nothing downstream can
tell a conversation that legitimately produced few memories from one that lost a
run of chunks to transient 5xx.

We hit this on BEAM 1M: one conversation ingested at 6.5 memories/chunk against
15.9 and 15.8 for its two siblings — same domain, same chunk count, slightly
higher fact density. Re-ingesting after fixing the server side brought it to 15.8
(1,703 -> 4,143 memories), confirming ~55% had been silently dropped. The
checkpoint had recorded total_chunks_failed = 0 throughout, because every failure
had been counted as a success upstream of it.

Only successful chunks are checkpointed now, so a resume retries the rest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oring runs 0.0

Same defect as mem0ai#23, in the LongMemEval runner. mem0ai#23 fixed the BEAM call sites;
these four were not touched.

`answerer.generate(...)` (:593, :759) and `judge_llm.generate_structured(...)`
(:700, :816) pass no max_tokens, so they inherit LLMClient's default of 4096.
`--answerer-model` and `--judge-model` both default to gpt-5, and reasoning
models bill their reasoning against max_completion_tokens — so a long answer can
exhaust the budget on reasoning alone and return an EMPTY string. That scores 0.0
and is indistinguishable from a wrong answer in the results.

This matters more here than it did on BEAM: the README states the LongMemEval
runs used GPT-5 as answerer and judge, so the published 91.0% is measured through
this path. On BEAM the same fix moved five arms by +0.010 to +0.047 and took
empty answers from 20 to 0.

Budgets match the BEAM fix: 16384 for answers, 8192 for the judge (small JSON
output, but its reasoning is billed the same way).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@christian4423
christian4423 force-pushed the fix/silent-failure-paths branch from 71a38ad to 1b15330 Compare August 9, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant