Skip to main content
The Context Management System uses a “pull” strategy where the AI model naturally requests context through tool calls rather than having context pushed to it. This ensures seamless integration with the Gemini Live API.

​
Recent Updates

Fixes

Fixed getContext tool registration and added automatic system prompt augmentation for context tool usage. The AI now proactively calls the tool when needing user information, page context, or memories.

​
Architecture Overview

1

Context Sources

Multiple sources feed into the context system:
  • Page Metadata (URL, title, domain)
  • User Information (preferences, settings)
  • Memory Search (semantic search)
  • Custom Context (application-specific)
2

Context State Manager

Centralized state storage with:
  • Context formatting for LLM
  • Update tracking and management
  • Type-safe state access
3

Context Memory Manager

Automatic memory handling:
  • Transcription-based searches
  • Debouncing and thresholds
  • Memory backend integration
4

GetContext Tool

Pull-based retrieval:
  • Synchronous tool execution
  • INTERRUPT scheduling for immediate use
  • No API calls - just formatting

​
Key Components

​
SystemPromptAugmentor

The SystemPromptAugmentor is a singleton that manages all system prompt modifications, ensuring the AI knows about available tools and context.
const augmentor = SystemPromptAugmentor.getInstance();

// Set language for the session
augmentor.setLanguage('es-ES');

// Add custom augmentation
augmentor.addAugmentation({
  id: 'custom-rules',
  content: '## Custom Rules\n\nAlways be polite and professional.',
  priority: 40,
  enabled: true,
});

// Apply augmentations to base prompt
const finalPrompt = augmentor.augmentPrompt(basePrompt);

​
ContextStateManager

The central state management class for all contextual information:
  • Page Metadata
  • User Information
  • Custom Context
  • Format Output
contextManager.updatePageMetadata({
  url: 'https://example.com/dashboard',
  title: 'User Dashboard',
  domain: 'example.com',
  path: '/dashboard'
});

​
ContextMemoryManager

Handles automatic memory searches based on transcriptions: 
const memoryManager = new ContextMemoryManager(
  memoryService,
  contextStateManager,
  {
    userInputThreshold: 20,      // Min chars for user input
    agentOutputThreshold: 50,    // Min chars for agent output
    searchLimit: 3,              // Max memories to return
    similarityThreshold: 0.2     // Min relevance score
  }
);

​
Automatic Context Tracking

SAMMY Three provides hooks for automatic context updates without manual intervention:

​
useContextUpdater Hook

The useContextUpdater hook automatically tracks page changes and updates context in real-time.
import { useContextUpdater } from '@sammy-labs/sammy-three';

function MyComponent() {
  // Automatically track page changes
  useContextUpdater({
    trackPageChanges: true,    // Monitor URL and title changes
    updateInterval: 5000,       // Check for changes every 5 seconds
    includeMetadata: true,      // Include page metadata
    customContext: {            // Add custom context data
      userId: 'user-123',
      sessionId: 'session-456',
      theme: 'dark',
      permissions: ['read', 'write'],
    },
  });

  return <div>Content tracked automatically</div>;
}

​
Manual Context Updates

For more granular control, update context manually: 
import { ContextStateManager } from '@sammy-labs/sammy-three';

const contextManager = new ContextStateManager();

// Update page information
contextManager.updatePageMetadata({
  url: window.location.href,
  title: document.title,
  timestamp: Date.now(),
  viewport: {
    width: window.innerWidth,
    height: window.innerHeight,
  },
});

​
Context Lifecycle

Initialization

Context managers are created when the agent starts

Automatic Updates

Page changes and user actions trigger context updates

Memory Search

Transcriptions automatically trigger memory searches

AI Retrieval

AI calls getContext tool when needed

Cleanup

Context is cleared when agent stops

​
Integration Guide

​
Basic Setup

Context management is automatically initialized when creating a new agent. No manual setup required!
const agent = new SammyAgentCore({
  // ... other options
});

// Context management is initialized automatically
// ✅ Transcriptions trigger memory searches
// ✅ GetContext tool is registered
// ✅ System prompt includes context instructions

​
Manual Context Updates

  • Page Context
  • User Context
  • Custom Context
agent.updatePageContext({
  url: window.location.href,
  title: document.title
});

​
React Hook Usage

For React applications, use the useContextUpdater hook: 
import { useContextUpdater } from '@sammy-labs/sammy-three';

function MyApp() {
  const { 
    startMonitoring, 
    updateUserContext, 
    setCustomContext 
  } = useContextUpdater({
    updateInterval: 1000,    // Check every second
    includeTitle: true,      // Track page title
    includeDomain: true,     // Track domain
    includePath: true        // Track URL path
  });

  // Start monitoring when agent is ready
  useEffect(() => {
    if (agentCoreRef.current) {
      startMonitoring(agentCoreRef.current);
    }
  }, [agentCoreRef.current]);

  // Manual updates
  const handleUserLogin = (user) => {
    updateUserContext({
      name: user.name,
      preferences: user.preferences
    });
  };

  // Custom context
  const handleTaskChange = (task) => {
    setCustomContext('currentTask', task);
  };
}

​
Context Flow Example

1

Page Navigation

User navigates to a new page
  • URL change detected by context updater
  • Page metadata updated in ContextStateManager
2

User Speech

User speaks: “What is my name?”
  • Transcription processed by ContextMemoryManager
  • Memory search triggered after 20+ characters
  • Relevant memories stored in context state
3

AI Response

AI recognizes it needs context
  • System prompt instructs AI to use getContext tool
  • AI calls getContext with query “user information”
  • Tool retrieves formatted context from ContextStateManager
  • Context returned with INTERRUPT scheduling
  • AI uses context: “Your name is Joseph Marinio”

​
System Prompt Augmentations

​
System Prompt Augmentation Details

  • Context Tool Instructions (Priority: 10)
  • Language Instructions (Priority: 5)
  • Memory Management (Priority: 20)
  • Tool Usage Guidelines (Priority: 30)
## Context Retrieval Tool

IMPORTANT: When the user asks questions about themselves, 
their preferences, or their context, you MUST use the 
getContext tool to retrieve this information. 

The getContext tool provides:
- User information (name, preferences, settings)
- Current page context (URL, title, domain)
- Relevant memories from past interactions
- Custom contextual information

Call getContext with queries like:
- "user information" - for user details
- "current page" - for page context
- "everything" - for all available context
- "memories about [topic]" - for specific memories

Always use this tool when you need contextual 
information to provide personalized assistance.

​
Context Formatting

​
Tool Response Format

Context Retrieved

The system formats context with clear sections for easy AI parsing:
[CONTEXT RETRIEVED]
📍 Current Page:
   URL: https://example.com/dashboard
   Title: User Dashboard
   Timestamp: 2024-01-15T10:30:00Z

👤 User Information:
   Name: Joseph Marinio
   Preferences: {"theme":"dark"}

💭 Relevant Knowledge:
   1. User prefers dark theme (0.95)
   2. User is an admin (0.87)
   3. Last login was yesterday (0.72)

🔧 Additional Context:
   currentTask: "reviewing invoices"

Use this context to provide personalized 
and accurate assistance.

​
Custom Formatting Options

const formatted = contextManager.formatContext({
  includeMemories: true,
  includePageMetadata: true,
  includeUserPreferences: true,
  includeCustomContext: true,
  memoryLimit: 5,
  format: 'detailed' // or 'compact'
});

​
Configuration Reference

​
Memory Search Configuration

interface MemorySearchConfig {
  userInputThreshold?: number;     // Default: 20
  agentOutputThreshold?: number;   // Default: 50
  searchLimit?: number;            // Default: 3
  similarityThreshold?: number;    // Default: 0.2
  useFeatureIdFilter?: boolean;    // Default: true
  searchGlobalOnly?: boolean;      // Default: false
  userContextPrefix?: string;      // Default: '[LIVE CONTEXT]'
  agentContextPrefix?: string;     // Default: '[VALIDATION]'
}

​
Context State Structure

interface ContextState {
  memories: MemoryEntry[];
  pageMetadata: PageMetadata | null;
  userPreferences: UserPreferences | null;
  customContext: Record<string, any>;
  lastUpdated: {
    memories?: Date;
    pageMetadata?: Date;
    userPreferences?: Date;
  };
}

​
Best Practices

Update Proactively

  • Use context updater hook for automatic page tracking
  • Update user context after authentication
  • Set custom context for important state changes

Optimize Memory Search

  • Adjust thresholds based on your use case
  • Use debouncing to prevent excessive searches
  • Consider feature-scoped vs global searches

Use Custom Context

  • Store workflow states
  • Track user actions
  • Add temporary contextual data

Performance Tips

  • Context is only formatted when requested
  • Memory searches are debounced
  • State updates are lightweight

​
Troubleshooting

​
GetContext Tool Not Being Called

If the AI isn’t calling the getContext tool, follow these steps:
1

Check Registration

Run agent.debugContextSystem() to verify the tool is registered
2

Verify System Prompt

Check console logs to ensure system prompt includes context instructions
3

Test Manually

Try asking explicit questions like “What is my name?” or “What page am I on?”
4

Check Logs

Look for these log messages:
  • 🔧 [SAMMY-AGENT-CORE] Registered getContext tool
  • ✅ [SAMMY-AGENT-CORE] getContext tool is present in declarations

​
Memory Search Not Working

1

Check Thresholds

Ensure transcription meets minimum character thresholds
2

Verify API Connection

Check memory service is properly initialized
3

Monitor Logs

Look for 💭 [CONTEXT-MEMORY-MANAGER] messages

​
Debug Logging

Enable debug logging to see context updates: 
🎯 [CONTEXT-STATE-MANAGER] initialized
📚 [CONTEXT-STATE-MANAGER] Updating memories: 3 entries
🌐 [CONTEXT-STATE-MANAGER] Updating page metadata
🔍 [GET-CONTEXT-TOOL] Called with query: user information
💭 [CONTEXT-MEMORY-MANAGER] Searching memories for user: What is my name...
🔧 [SAMMY-AGENT-CORE] Registered getContext tool
✅ [SAMMY-AGENT-CORE] getContext tool is present in declarations

​
Migration from Legacy System

The old MemoryManager has been completely removed. All memory management now goes through the centralized context system.
// ❌ Don't use this anymore
memoryManager.search(query)
memoryManager.processTranscription(text)

​
Future Enhancements

When Google fixes the sendClientContent bug, the system can easily switch from tool-based to text-based injection:
The modular design ensures a smooth migration path without changing the core context management logic.
⌘I