fix: four silent-failure paths — backend flag ignored, cloud add 400s, failed chunks checkpointed, LongMemEval token budget - #24
Open
christian4423 wants to merge 4 commits into
Conversation
backend = os.getenv("MEM0_BACKEND", args.backend)
`--backend` is declared with `default="oss"`, so `args.backend` is always truthy
and the `os.getenv` default is never reached. The environment therefore wins
unconditionally and the CLI flag does nothing whenever MEM0_BACKEND is present.
This is quiet and expensive. `load_dotenv(override=True)` pushes .env into the
environment, so a stale line in a committed .env silently beats an explicit flag
typed on the command line, and neither the logs nor the results record which
backend actually ran. I lost a paid run to it: `--backend cloud` spent ten
minutes against the previous backend before the mismatch became obvious.
Precedence is now explicit — CLI flag, then MEM0_BACKEND, then the "oss" default:
backend = args.backend or os.getenv("MEM0_BACKEND") or "oss"
Same fix in both runners (benchmarks/beam/run.py, benchmarks/longmemeval/run.py).
Behaviour is unchanged when the flag is omitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cloud add path posted to POST /v3/memories/ and failed 400 on every call with
"Filters are required and cannot be empty. Please refer to .../get-memories".
Probed against api.mem0.ai on 2026-08-07:
POST /v1/memories/ 200 [{"status":"PENDING","event_id":...}] <- add
POST /v3/memories/ 400 filters required (v3 POST is RETRIEVAL, not create)
POST /v3/memories/search/ 200 {"results":[...]} <- search
POST /v1/memories/search/ 400 v1 expects user_id, not the filters payload
Only ADD was mis-versioned; SEARCH at v3 was always correct and is unchanged in
behaviour (its version is now merely parameterised at the same default). The
surrounding add code already expected v1 semantics — it reads event_id from the
response — so the v3 URL was the mismatch.
Second symptom on the same path: v1 wraps its acknowledgement in a list,
[{"status":"PENDING","event_id":...}], while the code called .get() on it directly
and raised "'list' object has no attribute 'get'" — retried five times, then
returned None. Now accepts either envelope.
Both versions are overridable via MEM0_CLOUD_ADD_VERSION / MEM0_CLOUD_SEARCH_VERSION
so this does not hard-code today's API surface.
Verified end to end through Mem0Client itself (not curl): add returns an event_id,
the memory appears via the retrieval endpoint, and search returns it with score
0.6302. Note mem0 processes asynchronously — the probe took ~3 minutes from add
acknowledgement to becoming searchable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`chunks_already_done.add(chunk_key)` and the checkpoint write sit outside the `if response is not None` branch, so a chunk whose ingestion failed is recorded as completed. `--resume` then skips it forever. The failure is counted in total_failed and logged, but by the time anyone reads that warning the gap is already permanent: the only record of which chunks are missing is the checkpoint that just claimed them as done. Nothing downstream can tell a conversation that legitimately produced few memories from one that lost a run of chunks to transient 5xx. We hit this on BEAM 1M: one conversation ingested at 6.5 memories/chunk against 15.9 and 15.8 for its two siblings — same domain, same chunk count, slightly higher fact density. Re-ingesting after fixing the server side brought it to 15.8 (1,703 -> 4,143 memories), confirming ~55% had been silently dropped. The checkpoint had recorded total_chunks_failed = 0 throughout, because every failure had been counted as a success upstream of it. Only successful chunks are checkpointed now, so a resume retries the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oring runs 0.0 Same defect as mem0ai#23, in the LongMemEval runner. mem0ai#23 fixed the BEAM call sites; these four were not touched. `answerer.generate(...)` (:593, :759) and `judge_llm.generate_structured(...)` (:700, :816) pass no max_tokens, so they inherit LLMClient's default of 4096. `--answerer-model` and `--judge-model` both default to gpt-5, and reasoning models bill their reasoning against max_completion_tokens — so a long answer can exhaust the budget on reasoning alone and return an EMPTY string. That scores 0.0 and is indistinguishable from a wrong answer in the results. This matters more here than it did on BEAM: the README states the LongMemEval runs used GPT-5 as answerer and judge, so the published 91.0% is measured through this path. On BEAM the same fix moved five arms by +0.010 to +0.047 and took empty answers from 20 to 0. Budgets match the BEAM fix: 16384 for answers, 8192 for the judge (small JSON output, but its reasoning is billed the same way). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
christian4423
force-pushed
the
fix/silent-failure-paths
branch
from
August 9, 2026 01:10
71a38ad to
1b15330
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four independent fixes, all found while running BEAM and the mem0-cloud path. Each one turns a
failure into something that looks like a legitimate result, so none of them are visible in the
output they corrupt. Happy to split into separate PRs if you'd prefer.
Follow-up to #23, which fixed the same token-budget defect in the BEAM runner.
1.
--backendis unreachable wheneverMEM0_BACKENDis set--backendis declared withdefault="oss", soargs.backendis always truthy and thegetenvdefault is never reached — the environment wins unconditionally. Since
load_dotenv(override=True)pushes
.envinto the environment, a stale line in a committed.envsilently beats an explicitflag on the command line, and nothing in the logs or results records which backend actually ran.
Cost me ten minutes of a paid run pointed at the wrong backend. Now
args.backend or os.getenv("MEM0_BACKEND") or "oss", in both runners. Unchanged when the flag is omitted.2. Cloud
add()posts to/v3/memories/, which is a retrieval endpointReturns 400
Filters are required— so--backend cloudingestion was non-functional, notmerely degraded. Add is
/v1/; retrieval correctly stays on/v3/. Both are now env-overridableso the harness isn't pinned to one API generation. A follow-on:
/v1/wraps its acknowledgementin a list, so the dict-shaped handler raised
AttributeErroron the first successful add — alsofixed.
3. A failed ingestion chunk is checkpointed and never retried
chunks_already_done.add(chunk_key)sits outside theif response is not Nonebranch, so a chunkthat failed is recorded as completed and
--resumeskips it forever. The failure is counted andlogged, but by the time anyone reads that warning the gap is permanent — the only record of which
chunks are missing is the checkpoint that just claimed them.
We hit this on BEAM 1M: one conversation ingested at 6.5 memories/chunk against 15.9 and
15.8 for its two siblings — same domain, same chunk count, slightly higher fact density.
Re-ingesting brought it to 15.8 (1,703 → 4,143 memories), confirming ~55% had been silently
dropped.
total_chunks_failedread 0 the entire time.4. LongMemEval answerer/judge inherit
max_tokens=4096answerer.generate(:593,:759) andjudge_llm.generate_structured(:700,:816) pass nomax_tokens, inheritingLLMClient's 4096.--answerer-modeland--judge-modelboth default togpt-5, and reasoning models bill reasoning against
max_completion_tokens— so a long answerexhausts the budget on reasoning alone and returns an empty string, which scores 0.0 and is
indistinguishable from a wrong answer.
runs used GPT-5 as answerer and judge. On BEAM the identical fix moved five arms by +0.010 to
+0.047 and took empty answers from 20 to 0. Worth a re-run before the next publication.
Verification
--backend cloudwithMEM0_BACKEND=ossnow resolvesto
cloud(previouslyoss); env still honoured with no flag; default preserved with neither.avg_score=0.559.Before the fix every cloud
addfailed with the 400 above.Scope is deliberately narrow: three files, no new dependencies, no behaviour change when the
relevant flags and env vars are left alone.