Skip to content

[BUG]: Data conflicting issue with multiple jivas replicas in a kuberentes deployment #145

@ypkang

Description

@ypkang

Describe the bug
In a kubernetes deployment, when there are multiple jivas replicas sharing the same database, if two replicas execute init_agents at the same time, it will cause some data conflicting issue as two instances re-creating part of the same graph and saving to the database. This will cause one of the replicas not initing the agent properly.

The error will be something like the following

uccessfully installed langchain-0.3.26 langchain-community-0.3.27 langchain-core-0.3.68 langchain-openai-0.3.27 ollama-0.3.3 openai-1.93.1 typesense-1.1.1                                                                                                                                                                                                                                                                                                                                         
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.                                                                                                                         
INFO:     Package dependencies installed successfully.                                                                                                                                                                                                                                                                                                                                                                                                                                              
INFO:     downloaded action package: jivas/intro_interact_action 0.0.2                                                                                                                                                                                                                                                                                                                                                                                                                              
INFO:     jivas/intro_interact_action 0.0.2 loaded locally                                                                                                                                                                                                                                                                                                                                                                                                                                          
INFO:     downloaded action package: jivas/typesense_vector_store_action >=0.0.1                                                                                                                                                                                                                                                                                                                                                                                                                    
INFO:     jivas/typesense_vector_store_action 0.0.3 loaded locally                                                                                                                                                                                                                                                                                                                                                                                                                                  
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.                                                                                                                                                                                                                                                                                                                                      
INFO:     downloaded action package: jivas/typesense_retrieval_interact_action 0.0.2                                                                                                                                                                                                                                                                                                                                                                                                                
INFO:     jivas/typesense_retrieval_interact_action 0.0.2 loaded locally                                                                                                                                                                                                                                                                                                                                                                                                                            
INFO:     downloaded action package: jivas/agent_utils_action 0.0.8                                                                                                                                                                                                                                                                                                                                                                                                                                 
INFO:     jivas/agent_utils_action 0.0.8 loaded locally                                                                                                                                                                                                                                                                                                                                                                                                                                             
ERROR:     an exception occurred, Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                                                                                
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/core/init_agents.jac", line 50, in on_agents                                                                                                                                                                                                                                                                                                                                                                                            
    here spawn import_agent(descriptor = descriptor, reporting = self.reporting);                                                                                                                                                                                                                                                                                                                                                                                                                   
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.12/site-packages/jaclang/__init__.py", line 170, in spawn                                                                                                                                                                                                                                                                                                                                                                                                            
    return Jac.spawn_call(self, archi)  # type: ignore [arg-type, return-value]                                                                                                                                                                                                                                                                                                                                                                                                                     
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.12/site-packages/jaclang/plugin/feature.py", line 185, in spawn_call                                                                                                                                                                                                                                                                                                                                                                                                 
    return plugin_manager.hook.spawn_call(op1=op1, op2=op2)                                                                                                                                                                                                                                                                                                                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_hooks.py", line 513, in __call__                                                                                                                                                                                                                                                                                                                                                                                             
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_manager.py", line 120, in _hookexec                                                                                                                                                                                                                                                                                                                                                                                          
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                            
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 139, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    raise exception.with_traceback(exception.__traceback__)                                                                                                                                                                                                                                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 103, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    res = hook_impl.function(*args)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
          ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.12/site-packages/jac_cloud/plugin/jaseci.py", line 562, in spawn_call                                                                                                                                                                                                                                                                                                                                                                                                
    walker.returns.append(i.func(warch, current_node))                                                                                                                                                                                                                                                                                                                                                                                                                                              
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                               
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/core/import_agent.jac", line 62, in on_agents                                                                                                                                                                                                                                                                                                                                                                                           
    if(agent_node := self.import_from_descriptor(here, self.descriptor)) {                                                                                                                                                                                                                                                                                                                                                                                                                          
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                            
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/core/import_agent.jac", line 151, in import_from_descriptor                                                                                                                                                                                                                                                                                                                                                                             
    if(agent_node := agent_node.update(                                                                                                                                                                                                                                                                                                                                                                                                                                                             
                      ^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/core/agent.jac", line 146, in update                                                                                                                                                                                                                                                                                                                                                                                                    
    self.get_actions().install_actions(                                                                                                                                                                                                                                                                                                                                                                                                                                                             
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/action/actions.jac", line 274, in install_actions                                                                                                                                                                                                                                                                                                                                                                                       
    self.deregister_actions();                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
     ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/action/actions.jac", line 317, in deregister_actions                                                                                                                                                                                                                                                                                                                                                                                    
    for action_node in self.get_all() {                                                                                                                                                                                                                                                                                                                                                                                                                                                             
                        ^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/action/actions.jac", line 78, in get_all                                                                                                                                                                                                                                                                                                                                                                                                
    return ((self spawn _get_actions(filter_enabled=only_enabled)).action_nodes);                                                                                                                                                                                                                                                                                                                                                                                                                   
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                  
  File "/usr/local/lib/python3.12/site-packages/jaclang/__init__.py", line 170, in spawn                                                                                                                                                                                                                                                                                                                                                                                                            
    return Jac.spawn_call(self, archi)  # type: ignore [arg-type, return-value]                                                                                                                                                                                                                                                                                                                                                                                                                     
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.12/site-packages/jaclang/plugin/feature.py", line 185, in spawn_call                                                                                                                                                                                                                                                                                                                                                                                                 
    return plugin_manager.hook.spawn_call(op1=op1, op2=op2)                                                                                                                                                                                                                                                                                                                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_hooks.py", line 513, in __call__                                                                                                                                                                                                                                                                                                                                                                                             
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_manager.py", line 120, in _hookexec                                                                                                                                                                                                                                                                                                                                                                                          
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                            
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 139, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    raise exception.with_traceback(exception.__traceback__)                                                                                                                                                                                                                                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 103, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    res = hook_impl.function(*args)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
          ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.12/site-packages/jac_cloud/plugin/jaseci.py", line 562, in spawn_call                                                                                                                                                                                                                                                                                                                                                                                                
    walker.returns.append(i.func(warch, current_node))                                                                                                                                                                                                                                                                                                                                                                                                                                              
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                               
  File "/usr/local/lib/python3.12/site-packages/jivas/agent/action/actions.jac", line 731, in on_actions                                                                                                                                                                                                                                                                                                                                                                                            
    visit [-->](`?Action);                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
           ^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  File "/usr/local/lib/python3.12/site-packages/jaclang/__init__.py", line 224, in refs                                                                                                                                                                                                                                                                                                                                                                                                             
    ret = plugin_manager.hook.edge_ref(                                                                                                                                                                                                                                                                                                                                                                                                                                                             
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_hooks.py", line 513, in __call__                                                                                                                                                                                                                                                                                                                                                                                             
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_manager.py", line 120, in _hookexec                                                                                                                                                                                                                                                                                                                                                                                          
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                            
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 139, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    raise exception.with_traceback(exception.__traceback__)                                                                                                                                                                                                                                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 103, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    res = hook_impl.function(*args)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
          ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.12/site-packages/jaclang/plugin/default.py", line 1006, in edge_ref                                                                                                                                                                                                                                                                                                                                                                                                  
    nodes = Jac.edges_to_nodes(                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
            ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  File "/usr/local/lib/python3.12/site-packages/jaclang/plugin/feature.py", line 132, in edges_to_nodes                                                                                                                                                                                                                                                                                                                                                                                             
    return plugin_manager.hook.edges_to_nodes(                                                                                                                                                                                                                                                                                                                                                                                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_hooks.py", line 513, in __call__                                                                                                                                                                                                                                                                                                                                                                                             
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_manager.py", line 120, in _hookexec                                                                                                                                                                                                                                                                                                                                                                                          
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)                                                                                                                                                                                                                                                                                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                            
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 139, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    raise exception.with_traceback(exception.__traceback__)                                                                                                                                                                                                                                                                                                                                                                                                                                         
  File "/usr/local/lib/python3.12/site-packages/jaclang/vendor/pluggy/_callers.py", line 103, in _multicall                                                                                                                                                                                                                                                                                                                                                                                         
    res = hook_impl.function(*args)                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
          ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.12/site-packages/jac_cloud/plugin/jaseci.py", line 214, in edges_to_nodes                                                                                                                                                                                                                                                                                                                                                                                            
    return JacFeatureImpl.edges_to_nodes(                                                                                                                                                                                                                                                                                                                                                                                                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  File "/usr/local/lib/python3.12/site-packages/jaclang/plugin/default.py", line 267, in edges_to_nodes                                                                                                                                                                                                                                                                                                                                                                                             
    (source := anchor.source)                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
               ^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
  File "/usr/local/lib/python3.12/site-packages/jaclang/runtimelib/architype.py", line 104, in __getattr__                                                                                                                                                                                                                                                                                                                                                                                          
    self.populate()                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  File "/usr/local/lib/python3.12/site-packages/jac_cloud/core/architype.py", line 480, in populate                                                                                                                                                                                                                                                                                                                                                                                                 
    raise ValueError(                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
ValueError: EdgeAnchor [e::686c4097d05e16e7155471c4] is not a valid reference!                                                                                                                                                                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
INFO:     Returning call from admin@jivas.com to init_agents with payload: {"query":{},"files":{},"body":{"reporting":false}} at entry node: None with response: {"status":200}      

To Reproduce
Steps to reproduce the behavior:

  1. Start a kuberentes deployment of jivas with at least 2 replicas
  2. Spam init_agent calls
  3. Observe logs to see the error
  4. This happens pretty deterministically

Expected behavior
Some kind of locking around init_agents so that only one replica is modifying the graph structure at a time.

At least, init_agents request should return 500 when this happens. Currently returns 200.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):
jivas version 2.0.0a51

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions