Repository navigation
Conversation
…the vector DB used by `occ context_chat:wipe` Signed-off-by: kyteinsky <kyteinsky@gmail.com> Assisted-by: Github Copilot:claude-opus-5-5
Signed-off-by: kyteinsky <kyteinsky@gmail.com> Assisted-by: Github Copilot:claude-opus-5-5
Signed-off-by: kyteinsky <kyteinsky@gmail.com> Assisted-by: Github Copilot:claude-opus-5-5
don't fetch the chunk ids into the python app Signed-off-by: kyteinsky <kyteinsky@gmail.com> Assisted-by: Github Copilot:claude-opus-5-5
Synchronous counterpart of the delete_provider_id action, so context_chat:wipe --provider can report the result and reindex right after. The bg threads are stopped while it runs. Signed-off-by: kyteinsky <kyteinsky@gmail.com> Assisted-by: Github Copilot:claude-opus-5-5
| Any batch the threads are working on is finished first, so nothing they fetched | ||
| before can be written to the vector db while the caller runs. | ||
| ''' | ||
| with BG_THREADS_PAUSE_LOCK: |
There was a problem hiding this comment.
nice catch!
yeah this will just be received in one pod, we don't know about others.
see the context_chat part of the change: nextcloud/context_chat#290, we can turn the tap off there and wait for the locked docs to be done before starting a wipe, that might work well.
there is one issue that some docs are locked for 24h due to failures, they won't clear up in the timeout of 30 mins. that we wait for the batch the complete.
this code can probably be removed from CCB then I suppose.
|
Claude says:
|
|
|
||
| # The embeddings table is shared by all the langchain collections in this db. Lock it so no | ||
| # other collection can add rows between the check and the truncate, the lock is released on commit. | ||
| session.execute(sa.text(f'LOCK TABLE {embeddings_table} IN ACCESS EXCLUSIVE MODE')) |
There was a problem hiding this comment.
Claude says:
the code truncates docs/access_list (exclusive lock) first and then locks the embeddings table. Another worker or pod in add_indocuments may already hold a lock on the embeddings table and be waiting for docs. Postgres would detect the deadlock and abort one side, so it fails cleanly but it does fail. Locking all three tables at the start, in a fixed order, avoids this.
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
bbbca5c to
401aa3d
Compare
and some performance improvements.
it needs context chat changes to pass the tests.
related: #359 and nextcloud/context_chat#175
🤖 AI (if applicable)