All my graph maintenances are failing

[maikokan]

Hi Everyone,

I checked my hindsight control panel after a while and noticed that all my graph maintenances are failing. Retain and consolidations are all working perfectly. Any ideas why this might be the case? I am on v0.8.3

[nicoloboschi]

do you have any logs to share?

[maikokan]

Here are some logs from running journalctl -fu hindsight.service | grep --line-buffered graph_maintenance.

root@vmi3152678:~# journalctl -fu hindsight.service | grep --line-buffered graph_maintenance
Jul 05 21:07:30 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:07:30,599 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:07:50 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:07:50,206 - INFO - hindsight_api.worker.poller - Worker vmi3152678 claimed 1 tasks (0 consolidation): graph_maintenance:1 (schemas: default)
Jul 05 21:08:01 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:01,073 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=1/10 | reserved: [consolidation=0/2(avail=2)] | shared=1/8(avail=7) | global: pending=1 (queried=1/1 schemas: default) | others: none | pool: size=6 limits=5-100 idle=5 in_use=1 free_holders=99 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(1)
Jul 05 21:08:01 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:01,073 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:08:01 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:01,073 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1e103ad6-1f2d-4448-98ab-53f54cea5eb7 type=graph_maintenance op_type=graph_maintenance bank=hermes age=11s stage=task.graph_maintenance stage_age=11s
Jul 05 21:08:22 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:22,349 - INFO - hindsight_api.worker.poller - Worker vmi3152678 claimed 1 tasks (0 consolidation): graph_maintenance:1 (schemas: default)
Jul 05 21:08:31 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:31,451 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=2/10 | reserved: [consolidation=0/2(avail=2)] | shared=2/8(avail=6) | global: pending=0 (queried=1/1 schemas: default) | others: none | pool: size=7 limits=5-100 idle=5 in_use=2 free_holders=98 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(2)
Jul 05 21:08:31 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:31,452 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1e103ad6-1f2d-4448-98ab-53f54cea5eb7 type=graph_maintenance op_type=graph_maintenance bank=hermes age=41s stage=task.graph_maintenance stage_age=41s
Jul 05 21:08:31 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:31,452 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1c78c0e5-bd47-4319-83ce-dcfe61fdadfa type=graph_maintenance op_type=graph_maintenance bank=hermes age=9s stage=task.graph_maintenance stage_age=9s
Jul 05 21:08:51 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:08:51,065 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Jul 05 21:08:51 vmi3152678 hindsight-launcher.sh[538017]:     await self._handle_graph_maintenance(task_dict)
Jul 05 21:08:51 vmi3152678 hindsight-launcher.sh[538017]:   File "/usr/local/lib/python3.12/dist-packages/hindsight_api/engine/memory_engine.py", line 1778, in _handle_graph_maintenance
Jul 05 21:08:51 vmi3152678 hindsight-launcher.sh[538017]:     return await run_graph_maintenance_job(
Jul 05 21:08:51 vmi3152678 hindsight-launcher.sh[538017]:   File "/usr/local/lib/python3.12/dist-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
Jul 05 21:09:01 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:01,636 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=1/10 | reserved: [consolidation=0/2(avail=2)] | shared=1/8(avail=7) | global: pending=1 (queried=1/1 schemas: default) | others: none | pool: size=7 limits=5-100 idle=6 in_use=1 free_holders=99 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(1)
Jul 05 21:09:01 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:01,636 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:09:01 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:01,636 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1c78c0e5-bd47-4319-83ce-dcfe61fdadfa type=graph_maintenance op_type=graph_maintenance bank=hermes age=39s stage=task.graph_maintenance stage_age=39s
Jul 05 21:09:31 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:31,992 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=1/10 | reserved: [consolidation=0/2(avail=2)] | shared=1/8(avail=7) | global: pending=1 (queried=1/1 schemas: default) | others: none | pool: size=7 limits=5-100 idle=6 in_use=1 free_holders=99 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(1)
Jul 05 21:09:31 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:31,992 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:09:31 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:31,992 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1c78c0e5-bd47-4319-83ce-dcfe61fdadfa type=graph_maintenance op_type=graph_maintenance bank=hermes age=70s stage=task.graph_maintenance stage_age=70s
Jul 05 21:09:51 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:51,131 - INFO - hindsight_api.worker.poller - Worker vmi3152678 claimed 1 tasks (0 consolidation): graph_maintenance:1 (schemas: default)
Jul 05 21:09:51 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:09:51,461 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Jul 05 21:09:51 vmi3152678 hindsight-launcher.sh[538017]:     await self._handle_graph_maintenance(task_dict)
Jul 05 21:09:51 vmi3152678 hindsight-launcher.sh[538017]:   File "/usr/local/lib/python3.12/dist-packages/hindsight_api/engine/memory_engine.py", line 1778, in _handle_graph_maintenance
Jul 05 21:09:51 vmi3152678 hindsight-launcher.sh[538017]:     return await run_graph_maintenance_job(
Jul 05 21:09:51 vmi3152678 hindsight-launcher.sh[538017]:   File "/usr/local/lib/python3.12/dist-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
Jul 05 21:10:02 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:02,490 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=1/10 | reserved: [consolidation=0/2(avail=2)] | shared=1/8(avail=7) | global: pending=1 (queried=1/1 schemas: default) | others: none | pool: size=7 limits=5-100 idle=6 in_use=1 free_holders=99 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(1)
Jul 05 21:10:02 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:02,491 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:10:02 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:02,492 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1e103ad6-1f2d-4448-98ab-53f54cea5eb7 type=graph_maintenance op_type=graph_maintenance bank=hermes age=11s stage=task.graph_maintenance stage_age=11s
Jul 05 21:10:32 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:32,506 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=1/10 | reserved: [consolidation=0/2(avail=2)] | shared=1/8(avail=7) | global: pending=1 (queried=1/1 schemas: default) | others: none | pool: size=7 limits=5-100 idle=6 in_use=1 free_holders=99 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(1)
Jul 05 21:10:32 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:32,506 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:10:32 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:32,506 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1e103ad6-1f2d-4448-98ab-53f54cea5eb7 type=graph_maintenance op_type=graph_maintenance bank=hermes age=41s stage=task.graph_maintenance stage_age=41s
Jul 05 21:10:51 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:51,528 - INFO - hindsight_api.worker.poller - Worker vmi3152678 claimed 1 tasks (0 consolidation): graph_maintenance:1 (schemas: default)
Jul 05 21:10:51 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:10:51,786 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Jul 05 21:10:51 vmi3152678 hindsight-launcher.sh[538017]:     await self._handle_graph_maintenance(task_dict)
Jul 05 21:10:51 vmi3152678 hindsight-launcher.sh[538017]:   File "/usr/local/lib/python3.12/dist-packages/hindsight_api/engine/memory_engine.py", line 1778, in _handle_graph_maintenance
Jul 05 21:10:51 vmi3152678 hindsight-launcher.sh[538017]:     return await run_graph_maintenance_job(
Jul 05 21:10:51 vmi3152678 hindsight-launcher.sh[538017]:   File "/usr/local/lib/python3.12/dist-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
Jul 05 21:11:02 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:11:02,817 - INFO - hindsight_api.worker.poller - [WORKER_STATS] worker=vmi3152678 slots=1/10 | reserved: [consolidation=0/2(avail=2)] | shared=1/8(avail=7) | global: pending=1 (queried=1/1 schemas: default) | others: none | pool: size=7 limits=5-100 idle=6 in_use=1 free_holders=99 pending_acquires=0 | proc: rss_mb=2631 | my_active: graph_maintenance:hermes(1)
Jul 05 21:11:02 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:11:02,817 - INFO - hindsight_api.worker.poller - [PENDING_BREAKDOWN] graph_maintenance: total=1 claimable=0 payload_null=0 retry_blocked=1 assigned=0
Jul 05 21:11:02 vmi3152678 hindsight-launcher.sh[538017]: 2026-07-05 21:11:02,817 - INFO - hindsight_api.worker.poller - [WORKER_TASK] op=1c78c0e5-bd47-4319-83ce-dcfe61fdadfa type=graph_maintenance op_type=graph_maintenance bank=hermes age=11s stage=task.graph_maintenance stage_age=11s

[maikokan]

Additional logs from running journalctl -u hindsight.service -o cat | grep -A 20 "Task execution failed: graph_maintenance"

2026-06-16 10:21:36,434 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-16 10:23:37,033 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-16 10:25:37,629 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-16 10:27:38,016 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-17 02:20:21,089 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-17 02:22:21,750 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-17 02:24:22,246 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(
--
2026-06-17 02:26:22,633 - ERROR - hindsight_api.engine.memory_engine - Task execution failed: graph_maintenance, error:
Traceback (most recent call last):
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1719, in execute_task
    await self._handle_graph_maintenance(task_dict)
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/memory_engine.py", line 1567, in _handle_graph_maintenance
    return await run_graph_maintenance_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/graph_maintenance.py", line 220, in run_graph_maintenance_job
    result.stale_cooccurrences_pruned = await ops.prune_stale_cooccurrences(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/ops_postgresql.py", line 419, in prune_stale_cooccurrences
    result = await conn.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/hindsight_api/engine/db/postgresql.py", line 36, in execute
    return await self._conn.execute(query, *args, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 357, in execute
    _, status, _ = await self._execute(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/asyncpg/connection.py", line 1873, in _execute
    result, _ = await self.__execute(

Seems like a timeout error

[benfrank241]

Thanks — that's enough to pin it down. Your instinct is right: this is the stale_cooccurrences prune hitting the database statement_timeout and getting cancelled, which fails the whole graph_maintenance job (it runs prune_orphan_entities + prune_stale_cooccurrences in a single transaction, so if the second DELETE is cancelled the job fails and just keeps retrying — that's why every run shows as failing, and why the poller shows retry_blocked=1).

Retain/consolidation keep working because they touch small incremental batches; graph_maintenance does a bank-wide sweep of the co-occurrence graph, and on the hermes bank that graph has grown big enough for the DELETE to blow past the timeout.

Can you confirm the exact exception? The -A 20 window cut off the actual error line. Please run:

journalctl -u hindsight.service -o cat | grep -A 40 "Task execution failed: graph_maintenance" | grep -i "timeout\|QueryCanceled\|canceling statement"

I expect asyncpg.exceptions.QueryCanceledError: canceling statement due to statement timeout — that's the 600s default statement_timeout we set on every connection.

Two things to try, in order:

1. Refresh planner stats first — this is usually the real fix. A bank-wide DELETE that "suddenly" times out while incremental ops stay fast is almost always a stale-statistics bad plan (nested loop instead of a hash/merge). Run against your DB:

VACUUM ANALYZE entity_cooccurrences;
VACUUM ANALYZE unit_entities;
VACUUM ANALYZE entities;

Then let the next graph_maintenance fire. There's a decent chance it now completes in seconds. While you're there, check autovacuum has been keeping up:

SELECT relname, n_live_tup, last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname IN ('entity_cooccurrences','unit_entities','entities');

2. If it's genuinely just large, give the first sweep more room. Temporarily raise (or disable) the statement timeout so the backlog can drain once, then restart:

HINDSIGHT_API_DB_STATEMENT_TIMEOUT=0     # 0 disables; or e.g. 3600 for a 1h cap

After the first successful full prune, later runs only handle the small delta and finish well under the default, so you can set it back to 600.

Separately, I'll harden this on our side — the bank-wide co-occurrence prune should be batched so it can never exceed statement_timeout and wedge graph_maintenance into a permanent retry loop. I'll open an issue to track that. Let me know what the confirmed exception and the ANALYZE run show.