Observability System - SAMMY Labs Docs

The observability system tracks ALL data flowing into and out of the Gemini Live API, including session events, audio/video streams, tool calls, memory searches, transcriptions, and errors with context.

Quick Start

Minimal Configuration

import { SammyAgentConfig, ObservabilityConfig } from '@sammy-labs/sammy-three';

const config: SammyAgentConfig = {
  observability: {
    enabled: true,
  },
  // ... 
};

Architecture Overview

System Architecture

The observability system uses a worker-based architecture for optimal performance:

Key Components

ObservabilityManager

Central event tracking and session management

ObservabilityWorker

Background thread for API communication (optional)

AudioAggregator

PCM audio buffering and flushing

TraceEvents

Strongly-typed event definitions

Configuration Reference

Default Values

The observability system uses sensible defaults when values are not specified:

const defaults = {
  enabled: false,                   // Must be explicitly enabled
  logToConsole: false,              // No console logging by default
  includeSystemPrompt: true,        // Include system prompts
  includeAudioData: true,           // Include raw audio data by default
  includeImageData: true,           // Include image data by default
  useWorker: true,                  // Worker mode enabled by default
  
  // Audio aggregation defaults
  audioAggregation: {
    flushIntervalMs: 10000,         // 10 seconds
  },
  
  // Worker configuration defaults
  workerConfig: {
    batchSize: 50,                  // 50 events per batch
    batchIntervalMs: 5000,          // 5 seconds between batches
  },
};

Complete Configuration Interface

ObservabilityConfig
Development Config

export interface ObservabilityConfig {
  /**
   * Enable/disable observability tracking
   */
  enabled: boolean;

  /**
   * Use web worker for non-blocking API calls (recommended)
   */
  useWorker?: boolean;

  /**
   * Worker-specific configuration
   */
  workerConfig?: {
    batchSize?: number;        // Default: 50 events
    batchIntervalMs?: number;  // Default: 5000ms
  };

  /**
   * Audio aggregation configuration
   */
  audioAggregation?: {
    flushIntervalMs?: number;  // Default: 10000ms
    onFlush?: (data: FlushData) => Promise<void>; // Optional with worker
  };

  /**
   * Custom callback for each event (runs on main thread)
   */
  callback?: (event: TraceEvent) => Promise<void> | void;

  /**
   * Event types to filter out
   */
  disableEventTypes?: TraceEventType[];

  /**
   * Privacy controls
   */
  includeAudioData?: boolean;     // Include raw audio in events
  includeImageData?: boolean;     // Include screenshots in events
  includeSystemPrompt?: boolean;  // Include system prompts

  /**
   * Debug logging to console
   */
  logToConsole?: boolean;

  /**
   * Additional metadata for all events
   */
  metadata?: Record<string, any>;
}

Worker Mode Setup

Worker mode moves all API communication to a background thread for optimal performance.

Benefits

Zero UI Blocking

High-frequency events don’t affect UI

Automatic Batching

Reduces API calls by 10-50x

Built-in Retry

Failed requests with exponential backoff

Efficient Transfer

Zero-copy audio transfer

Configuration Example

const config: SammyAgentConfig = {
  observability: {
    enabled: true,
    useWorker: true,  // Enable worker mode
    
    workerConfig: {
      batchSize: 50,         // Events per batch
      batchIntervalMs: 5000, // Send interval
    },
    
    // Audio handling is automatic with worker
    audioAggregation: {
      flushIntervalMs: 30000, // 30 seconds
      // No onFlush needed - worker handles it
    },
  },
};

CSP Requirements

Worker Mode CSP

The observability worker uses Data URLs to bypass CSP restrictions - no configuration required!

// Workers are loaded via data: URLs, not blob: or worker-src
const dataUrl = `data:application/javascript;base64,${base64Code}`;
this.worker = new Worker(dataUrl);

Required CSP Headers

Content-Security-Policy: 
  connect-src 'self' https://api.sammylabs.com https://your-api.com;

When using absolute URLs (recommended), all observability endpoints are derived from the baseUrl [[memory:5135225]].

Complete Production Example

/**
 * Custom hook for Sammy authentication
 */
export const useSammyAuth = ({ isInternal = false }: { isInternal: boolean }) => {
  const [jwtToken, setJwtToken] = useState<string | null>(null);
  const [authError, setAuthError] = useState<string | null>(null);
  const [isRefreshing, setIsRefreshing] = useState<boolean>(false);

  const refreshToken = async () => {
    if (isRefreshing) {
      console.log('[Auth] Token refresh already in progress, skipping...');
      return;
    }

    try {
      setIsRefreshing(true);
      console.log('[Auth] Refreshing JWT token...');

      const tokenData = await fetchJWTToken(isInternal);
      setJwtToken(tokenData.token);
      setAuthError(null);

      console.log('[Auth] JWT token refreshed successfully');
    } catch (error) {
      console.error('[Auth] Failed to refresh JWT token:', error);
      setAuthError('Failed to refresh authentication token');
    } finally {
      setIsRefreshing(false);
    }
  };

  const handleTokenExpired = async () => {
    console.log('[Auth] Token expired, attempting to refresh...');
    await refreshToken();
  };

  useEffect(() => {
    const initializeAuth = async () => {
      try {
        const tokenData = await fetchJWTToken(isInternal);
        setJwtToken(tokenData.token);
        setAuthError(null);
      } catch (error) {
        console.error('Failed to initialize JWT token:', error);
        setAuthError('Failed to authenticate with Sammy Agent');
      }
    };

    initializeAuth();
  }, [isInternal]);

  return {
    jwtToken,
    authError,
    isRefreshing,
    refreshToken,
    handleTokenExpired,
  };
};

API Endpoints

When using worker mode, the observability system automatically calls these endpoints:

Trace Endpoint

POST /api/v1/sammy-three/trace/
Content-Type: application/json

{
  events: TraceEvent[],
  conversationData?: {
    // Only for session.start events
    sessionId: string,
    agentMode: 'USER' | 'ADMIN',
    model: string,
    externalUserId?: string,
  },
  metadata?: Record<string, any>
}

Audio Flush Endpoint

POST /api/v1/sammy-three/trace/audio/flush
  ?conversationId=X
  &speaker=Y
  &sampleRate=Z
  &startTime=A
  &endTime=B
  &totalBytes=C
Content-Type: multipart/form-data

FormData:
- audio: Blob (PCM format, not WAV)

Event Types Reference

Core Event Categories

// Session lifecycle
'session.start' | 'session.end'

// Configuration
'config.set' | 'system_prompt.set'

// Content flow
'content.send' | 'content.receive'
'transcription.input' | 'transcription.output'
'turn.complete'

// Audio
'audio.send' | 'audio.receive'
'audio.recording_start' | 'audio.recording_stop'
'audio.volume_change' | 'audio.gate_state_change'

// Screen capture
'screen_capture.send' | 'screen_capture.critical'
'screen_capture.start' | 'screen_capture.stop'

// Tools
'tool.register' | 'tool.call' | 'tool.response'

// Memory
'memory.search' | 'memory.inject'

// Errors
'error' | 'connection.error' | 'audio.recording_error'

// Agent control
'agent.mute' | 'agent.unmute'
'agent.streaming_start' | 'agent.streaming_stop'

Filtering Events

// Filter out noisy events
observability: {
  disableEventTypes: [
    'audio.send',
    'audio.receive',
    'transcription.input',
    'transcription.output',
  ],
}

Advanced Features

High-Resolution Timestamps

Precise Event Ordering

Events use dual approaches for precise ordering:

Microsecond timestamps via performance.now()
Sequence numbers for guaranteed ordering

interface TraceEvent {
  timestamp: Date;         // High-resolution timestamp
  sequenceNumber: number;  // Guaranteed ordering (0, 1, 2...)
}

Session Statistics

const agent = useSammyAgentContext();
const stats = agent.agentCoreRef.current?.getObservabilityStatistics();

console.log({
  duration: stats.duration,
  messagesSent: stats.messagesSent,
  audioBytesSent: stats.audioBytesSent,
  toolCalls: stats.toolCalls,
  errors: stats.errors,
});

Custom Event Tracking

const agent = useSammyAgentContext();
const observability = agent.agentCoreRef.current?.observabilityManager;

await observability?.trackEvent({
  type: 'custom.event',
  data: {
    action: 'user_clicked_button',
    details: { buttonId: 'submit' },
  },
});

Export Trace Data

const trace = agent.agentCoreRef.current?.getObservabilityTrace();
const json = agent.agentCoreRef.current?.observabilityManager?.exportAsJson();

// Save to file
const blob = new Blob([json], { type: 'application/json' });
const url = URL.createObjectURL(blob);
// ... download logic

Troubleshooting

Common Issues

Worker Not Sending Events
Events Not Batching
Audio Not Flushing
High Memory Usage

Troubleshooting Steps:

Check enabled: true in config
Verify useWorker: true is set
Ensure proper authentication token
Check browser console for worker errors

Debugging Tips

// Enable debug mode
observability: {
  enabled: true,
  logToConsole: true,  // See all events in console
  useWorker: false,    // Disable worker to see errors
}

// Check worker status
// Look for: [ObservabilityWorker] Trace events sent successfully:

// Monitor performance
const stats = performance.getEntriesByType('measure');

Migration from Callback Mode

Before (Manual)
After (Worker)

observability: {
  callback: async (event) => {
    // Manual API call
    await fetch('/api/trace', { 
      body: JSON.stringify(event) 
    });
  },
  audioAggregation: {
    onFlush: async (data) => {
      // Manual audio upload
      await uploadAudio(data);
    },
  },
}

Performance Considerations

Main Thread Impact

Without Worker: Each event blocks during JSON serializationWith Worker: Events sent via postMessage (microseconds)

Memory Usage

Audio uses transferable objects (zero-copy)
Events batched efficiently in worker
Automatic cleanup on session end

Network Optimization

Batching reduces API calls by 10-50x
Automatic retry with exponential backoff
Failed events don’t block new ones

Known Limitations

Be aware of these current limitations:

Worker Initialization: Worker mode requires proper authentication setup and may fail silently if auth is misconfigured
Audio Format: Audio is sent as PCM data, which requires server-side processing to convert to playable formats
Memory Usage: High-frequency events can consume significant memory if not properly filtered
Browser Compatibility: Worker mode uses data URLs which work in all modern browsers but may have issues in some extensions

Best Practices

Start Simple

Begin with basic observability enabled and add worker mode later

Filter Events

Always use disableEventTypes to filter out noisy events in production

Monitor Performance

Watch for memory usage and batch sizes in production

Test Worker Mode

Thoroughly test worker initialization in your deployment environment

Quick Start Checklist

Implementation Checklist

Set observability.enabled: true
Configure authentication with baseUrl
Add event filtering for production
Test worker mode if using high-frequency events
Monitor console logs during development
Verify API endpoints are receiving data

Get Started

Features

Integrations

​Quick Start

​Minimal Configuration

​Architecture Overview

System Architecture

​Key Components

ObservabilityManager

ObservabilityWorker

AudioAggregator

TraceEvents

​Configuration Reference

​Default Values

​Complete Configuration Interface

​Worker Mode Setup

​Benefits

Zero UI Blocking

Automatic Batching

Built-in Retry

Efficient Transfer

​Configuration Example

​CSP Requirements

​Worker Mode CSP

​Required CSP Headers

​Complete Production Example

​API Endpoints

​Trace Endpoint

​Audio Flush Endpoint

​Event Types Reference

​Core Event Categories

​Filtering Events

​Advanced Features

​High-Resolution Timestamps

Precise Event Ordering

​Session Statistics

​Custom Event Tracking

​Export Trace Data

​Troubleshooting

​Common Issues

​Debugging Tips

​Migration from Callback Mode

​Performance Considerations

Main Thread Impact

Memory Usage

Network Optimization

​Known Limitations

​Best Practices

​Quick Start Checklist

Implementation Checklist

Quick Start

Minimal Configuration

Architecture Overview

Key Components

Configuration Reference

Default Values

Complete Configuration Interface

Worker Mode Setup

Benefits

Configuration Example

CSP Requirements

Worker Mode CSP

Required CSP Headers

Complete Production Example

API Endpoints

Trace Endpoint

Audio Flush Endpoint

Event Types Reference

Core Event Categories

Filtering Events

Advanced Features

High-Resolution Timestamps

Session Statistics

Custom Event Tracking

Export Trace Data

Troubleshooting

Common Issues

Debugging Tips

Migration from Callback Mode

Performance Considerations

Known Limitations

Best Practices

Quick Start Checklist