Skip to content

memory-system.md

Latest commit

 

History

History
230 lines (187 loc) · 9.61 KB

memory-system.md

File metadata and controls

230 lines (187 loc) · 9.61 KB
Raw

Memory System

Semantic memory system for the Chief of Staff that stores facts, learnings, observations, decisions, and user preferences with vector embeddings for intelligent retrieval.

Architecture

  • Memory Backend (server/services/memoryBackend.js): Selects the storage backend — memoryDB.js (PostgreSQL + pgvector) for real installs; memory.js (JSON files) is a test-only escape hatch (MEMORY_BACKEND=file / NODE_ENV=test), not a supported deployment mode
  • Memory DB Service (server/services/memoryDB.js): Primary CRUD, search, and lifecycle operations against PostgreSQL
  • Embeddings Service (server/services/memoryEmbeddings.js): LM Studio integration for vector generation
  • Memory Extractor (server/services/memoryExtractor.js): Extract memories from agent output
  • Memory Classifier (server/services/memoryClassifier.js): LLM-based quality filtering
  • Memory Retriever (server/services/memoryRetriever.js): Context injection for agent prompts
  • Memory Routes (server/routes/memory.js): REST API endpoints
  • Memory Tab (ChiefOfStaff.jsx): UI with list, timeline, and graph views

Features

  1. Six Memory Types: fact, learning, observation, decision, preference, context
  2. Semantic Search: LM Studio embeddings for similarity-based retrieval
  3. LLM Classification: Intelligent memory extraction with quality filtering (M31)
  4. Auto-Extraction: Memories extracted from successful agent task completions
  5. Auto-Injection: Relevant memories injected into agent prompts before execution
  6. Importance Decay: Time-based decay with access-based boosts
  7. Memory Consolidation: Merge similar memories automatically
  8. Real-time Updates: WebSocket events for memory changes
  9. Graph Visualization: D3.js relationship graph (planned)

Persistent Mind memory protection

Persistent Mind memories have an explicit cleanup protection level, separate from numerical importance: standard, important, or core-identity. The Memories panel lets the user set it when creating or editing a memory. Existing records remain standard until explicitly marked; no identity is inferred from private conversation history during installation or startup.

Protected memories remain active through manual and self-triggered mind cleanup, background decay, expiration, and duplicate consolidation. Cleanup reports both archived and preserved counts. Clearing history still removes facts stored only in that history, so save lasting identity and knowledge as protected memories first. Protected memories take priority in the bounded context; protection is a retention policy, not a grant of authority or an unlimited context budget.

The mind can emit protection in a memory candidate. With its existing manageMind grant it can call mind.protect-memory using an existing owned memory ID. That tool only adds protection; it cannot remove protection or demote core identity. Only an explicit user edit to Standard makes a protected memory eligible for bulk cleanup again. Direct record deletion in Brain remains a separate user action.

Storage uses the existing tags field: mind:core-identity and mind:important. Both PostgreSQL and the test-only file backend already round-trip tags, so no new record format, seed, migration, or federation surface is introduced. The Mind API projects those tags as protection; older clients editing ordinary tags preserve the protection unless they explicitly supply a new protection level. The cleanup guards require this version of PortOS; downgrading to an older build cannot provide these retention guarantees.

Memory Schema

{
  id: string,              // UUID
  type: 'fact' | 'learning' | 'observation' | 'decision' | 'preference' | 'context',
  content: string,         // Full memory content
  summary: string,         // Short summary
  category: string,        // e.g., 'codebase', 'workflow', 'tools'
  tags: string[],          // Auto-extracted and user-defined
  relatedMemories: string[], // Linked memory IDs
  sourceTaskId: string,    // Origin task
  sourceAgentId: string,   // Origin agent
  embedding: number[],     // Vector (768 dims for nomic-embed)
  confidence: number,      // 0.0-1.0
  importance: number,      // 0.0-1.0 (decays over time)
  accessCount: number,
  lastAccessed: string,
  createdAt: string,
  status: 'active' | 'archived' | 'expired'
}

Data Storage

Memories and links live in PostgreSQL — the memories and memory_links tables, with pgvector columns for embeddings (see docs/STORAGE.md).

The test-only file backend (MEMORY_BACKEND=file / NODE_ENV=test) uses this JSON layout instead:

./data/cos/memory/
├── index.json         # Lightweight metadata for listing/filtering
├── embeddings.json    # Vector storage for semantic search
└── memories/          # Full memory content
    └── {id}/
        └── memory.json

LLM-Based Classification (M31)

The memory classifier uses LM Studio's gptoss-20b model to intelligently evaluate agent output and extract only genuinely useful memories.

Good Memories

  • Codebase facts: File locations, architecture patterns, dependencies
  • User preferences: Coding style, tool preferences, workflow patterns
  • Learnings: Discovered behaviors, gotchas, workarounds
  • Decisions: Architectural choices with reasoning

Rejected Memories

  • Task echoes: Just restating what the task was
  • Generic summaries: "The task was successful"
  • Temporary info: Session-specific data, timestamps
  • Truncated/incomplete content

Configuration

{
  "enabled": true,
  "provider": "lmstudio",
  "endpoint": "http://localhost:1234/v1/chat/completions",
  "model": "gptoss-20b",
  "timeout": 60000,
  "maxOutputLength": 10000,
  "minConfidence": 0.6,
  "fallbackToPatterns": true
}

Memory Extraction

Memories are extracted from agent output:

  1. LLM Classification: gptoss-20b analyzes task and output, extracts quality memories
  2. Fallback Patterns: If LLM unavailable, falls back to pattern matching
  3. High confidence (>0.8): Auto-saved
  4. Medium confidence (0.5-0.8): Queued for user approval

Memory Injection

Before agent task execution:

  1. Generate embedding for task description
  2. Find semantically similar memories (>0.7 relevance)
  3. Include high-importance user preferences
  4. Include relevant codebase facts
  5. Format as markdown section in agent prompt

API Endpoints

Route Description
GET /api/memory List memories with filters
GET /api/memory/:id Get single memory
POST /api/memory Create memory
PUT /api/memory/:id Update memory
DELETE /api/memory/:id Delete (soft) memory
POST /api/memory/search Semantic search
GET /api/memory/categories List categories
GET /api/memory/tags List tags
GET /api/memory/timeline Timeline view data
GET /api/memory/graph Graph visualization data
GET /api/memory/stats Memory statistics
POST /api/memory/link Link two memories
POST /api/memory/consolidate Merge similar memories
POST /api/memory/decay Apply importance decay
DELETE /api/memory/expired Clear expired memories
GET /api/memory/embeddings/status LM Studio connection status
GET /api/memory/backend/status Active storage backend status
GET /api/memory/sync File↔DB sync status
POST /api/memory/sync Trigger file↔DB sync
POST /api/memory/:id/approve Approve a pending memory
POST /api/memory/:id/reject Reject a pending memory

WebSocket Events

Event Description
cos:memory:created New memory created
cos:memory:updated Memory updated
cos:memory:deleted Memory deleted
cos:memory:extracted Memories extracted from agent
cos:memory:approval-needed Medium-confidence memories pending approval

Setup Requirements

LM Studio must be running with an embedding model loaded:

  1. Download and install LM Studio
  2. Load an embedding model: text-embedding-nomic-embed-text-v2-moe (recommended)
  3. Load a classification model: gptoss-20b or similar
  4. Start the local server on port 1234 (default)
  5. The memory system will automatically connect

LM Studio Configuration

memory: {
  enabled: true,
  embeddingProvider: 'lmstudio',
  embeddingEndpoint: 'http://localhost:1234/v1/embeddings',
  embeddingModel: 'text-embedding-nomic-embed-text-v2-moe',
  embeddingDimension: 768,
  maxContextTokens: 2000,
  minRelevanceThreshold: 0.7,
  autoExtractEnabled: true
}

Implementation Files

File Purpose
server/lib/memoryValidation.js Zod schemas for memory operations
server/lib/vectorMath.js Cosine similarity, clustering helpers
server/services/memoryBackend.js Backend switcher (Postgres primary, file test-only)
server/services/memoryDB.js Core CRUD, search, lifecycle (PostgreSQL + pgvector)
server/services/memory.js File-backed CRUD (test-only backend)
server/services/memoryEmbeddings.js LM Studio embedding generation
server/services/memoryExtractor.js Extract memories from agent output
server/services/memoryClassifier.js LLM-based classification service
server/services/memoryRetriever.js Retrieve and format for injection
server/routes/memory.js REST API endpoints
client/src/pages/ChiefOfStaff.jsx MemoryTab, MemoryTimeline, MemoryGraph
client/src/services/api.js Memory API client functions
data/prompts/stages/memory-evaluate.md Memory evaluation prompt template

Related Features