> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sammylabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability System

> Complete implementation guide for adding observability and traces to your application with @sammy-labs/sammy-three

<Note>
  The observability system tracks ALL data flowing into and out of the Gemini Live API, including session events, audio/video streams, tool calls, memory searches, transcriptions, and errors with context.
</Note>

## Quick Start

### Minimal Configuration

```typescript theme={null}
import { SammyAgentConfig, ObservabilityConfig } from '@sammy-labs/sammy-three';

const config: SammyAgentConfig = {
  observability: {
    enabled: true,
  },
  // ... 
};
```

## Architecture Overview

<Card title="System Architecture" icon="sitemap">
  The observability system uses a worker-based architecture for optimal performance:

  ```mermaid theme={null}
  graph LR
      A[Main Thread] --> B[ObservabilityMgr]
      B --> C[TraceEvents]
      B --> D[Worker Thread]
      D --> E[Batch Events]
      D --> F[API Calls]
      D --> G[Audio Flush]
  ```
</Card>

### Key Components

<Columns cols={2}>
  <Card title="ObservabilityManager" icon="database">
    Central event tracking and session management
  </Card>

  <Card title="ObservabilityWorker" icon="gear">
    Background thread for API communication (optional)
  </Card>

  <Card title="AudioAggregator" icon="microphone">
    PCM audio buffering and flushing
  </Card>

  <Card title="TraceEvents" icon="code">
    Strongly-typed event definitions
  </Card>
</Columns>

## Configuration Reference

### Default Values

<Info>
  The observability system uses sensible defaults when values are not specified:
</Info>

```typescript theme={null}
const defaults = {
  enabled: false,                   // Must be explicitly enabled
  logToConsole: false,              // No console logging by default
  includeSystemPrompt: true,        // Include system prompts
  includeAudioData: true,           // Include raw audio data by default
  includeImageData: true,           // Include image data by default
  useWorker: true,                  // Worker mode enabled by default
  
  // Audio aggregation defaults
  audioAggregation: {
    flushIntervalMs: 10000,         // 10 seconds
  },
  
  // Worker configuration defaults
  workerConfig: {
    batchSize: 50,                  // 50 events per batch
    batchIntervalMs: 5000,          // 5 seconds between batches
  },
};
```

### Complete Configuration Interface

<Tabs>
  <Tab title="ObservabilityConfig">
    ```typescript theme={null}
    export interface ObservabilityConfig {
      /**
       * Enable/disable observability tracking
       */
      enabled: boolean;

      /**
       * Use web worker for non-blocking API calls (recommended)
       */
      useWorker?: boolean;

      /**
       * Worker-specific configuration
       */
      workerConfig?: {
        batchSize?: number;        // Default: 50 events
        batchIntervalMs?: number;  // Default: 5000ms
      };

      /**
       * Audio aggregation configuration
       */
      audioAggregation?: {
        flushIntervalMs?: number;  // Default: 10000ms
        onFlush?: (data: FlushData) => Promise<void>; // Optional with worker
      };

      /**
       * Custom callback for each event (runs on main thread)
       */
      callback?: (event: TraceEvent) => Promise<void> | void;

      /**
       * Event types to filter out
       */
      disableEventTypes?: TraceEventType[];

      /**
       * Privacy controls
       */
      includeAudioData?: boolean;     // Include raw audio in events
      includeImageData?: boolean;     // Include screenshots in events
      includeSystemPrompt?: boolean;  // Include system prompts

      /**
       * Debug logging to console
       */
      logToConsole?: boolean;

      /**
       * Additional metadata for all events
       */
      metadata?: Record<string, any>;
    }
    ```
  </Tab>

  <Tab title="Development Config">
    ```typescript theme={null}
    const config: SammyAgentConfig = {
      auth: {
        token: 'your-jwt-token',
        baseUrl: process.env.NEXT_PUBLIC_BASE_URL || 'https://api.sammylabs.com',
        onTokenExpired: async () => {
          // Handle token refresh
        },
      },
      captureMethod: 'render',
      debugLogs: true,
      
      observability: {
        enabled: true,
        logToConsole: true,
        includeSystemPrompt: true,
        includeAudioData: false,  // Don't include large audio data
        includeImageData: true,   // Include screenshots for debugging
        
        // Filter out noisy events
        disableEventTypes: [
          'audio.send',
          'audio.receive',
        ],
        
        metadata: {
          environment: 'development',
          timestamp: new Date().toISOString(),
        },
        
        // Custom callback for development logging
        callback: async (event: TraceEvent) => {
          if (event.type === 'error') {
            console.error('[Observability] Error:', event.data);
          }
        },
      },
    };
    ```
  </Tab>
</Tabs>

## Worker Mode Setup

<Note>
  Worker mode moves all API communication to a background thread for optimal performance.
</Note>

### Benefits

<Columns cols={2}>
  <Card title="Zero UI Blocking" icon="gauge">
    High-frequency events don't affect UI
  </Card>

  <Card title="Automatic Batching" icon="layer-group">
    Reduces API calls by 10-50x
  </Card>

  <Card title="Built-in Retry" icon="refresh">
    Failed requests with exponential backoff
  </Card>

  <Card title="Efficient Transfer" icon="exchange">
    Zero-copy audio transfer
  </Card>
</Columns>

### Configuration Example

```typescript theme={null}
const config: SammyAgentConfig = {
  observability: {
    enabled: true,
    useWorker: true,  // Enable worker mode
    
    workerConfig: {
      batchSize: 50,         // Events per batch
      batchIntervalMs: 5000, // Send interval
    },
    
    // Audio handling is automatic with worker
    audioAggregation: {
      flushIntervalMs: 30000, // 30 seconds
      // No onFlush needed - worker handles it
    },
  },
};
```

## CSP Requirements

### Worker Mode CSP

<Tip>
  The observability worker uses **Data URLs** to bypass CSP restrictions - no configuration required!
</Tip>

```typescript theme={null}
// Workers are loaded via data: URLs, not blob: or worker-src
const dataUrl = `data:application/javascript;base64,${base64Code}`;
this.worker = new Worker(dataUrl);
```

### Required CSP Headers

```http theme={null}
Content-Security-Policy: 
  connect-src 'self' https://api.sammylabs.com https://your-api.com;
```

<Note>
  When using absolute URLs (recommended), all observability endpoints are derived from the baseUrl \[\[memory:5135225]].
</Note>

## Complete Production Example

<CodeGroup>
  ```typescript Authentication Hook theme={null}
  /**
   * Custom hook for Sammy authentication
   */
  export const useSammyAuth = ({ isInternal = false }: { isInternal: boolean }) => {
    const [jwtToken, setJwtToken] = useState<string | null>(null);
    const [authError, setAuthError] = useState<string | null>(null);
    const [isRefreshing, setIsRefreshing] = useState<boolean>(false);

    const refreshToken = async () => {
      if (isRefreshing) {
        console.log('[Auth] Token refresh already in progress, skipping...');
        return;
      }

      try {
        setIsRefreshing(true);
        console.log('[Auth] Refreshing JWT token...');

        const tokenData = await fetchJWTToken(isInternal);
        setJwtToken(tokenData.token);
        setAuthError(null);

        console.log('[Auth] JWT token refreshed successfully');
      } catch (error) {
        console.error('[Auth] Failed to refresh JWT token:', error);
        setAuthError('Failed to refresh authentication token');
      } finally {
        setIsRefreshing(false);
      }
    };

    const handleTokenExpired = async () => {
      console.log('[Auth] Token expired, attempting to refresh...');
      await refreshToken();
    };

    useEffect(() => {
      const initializeAuth = async () => {
        try {
          const tokenData = await fetchJWTToken(isInternal);
          setJwtToken(tokenData.token);
          setAuthError(null);
        } catch (error) {
          console.error('Failed to initialize JWT token:', error);
          setAuthError('Failed to authenticate with Sammy Agent');
        }
      };

      initializeAuth();
    }, [isInternal]);

    return {
      jwtToken,
      authError,
      isRefreshing,
      refreshToken,
      handleTokenExpired,
    };
  };
  ```

  ```typescript Configuration Factory theme={null}
  /**
   * Create Sammy Provider configuration with full observability
   */
  export const createSammyProviderConfig = ({
    jwtToken,
    onTokenExpired,
    captureMethod,
    enableWorkerMode = true,
    enableAudioAggregation = true,
    debugMode = false,
  }: SammyProviderConfigParams): SammyAgentConfig => {
    
    // Get base URL from environment [[memory:5135225]]
    const baseUrl = process.env.NEXT_PUBLIC_BASE_URL || 'https://api.sammylabs.com';
    
    // Build observability configuration
    const observabilityConfig: ObservabilityConfig = {
      // Core settings
      enabled: true,
      logToConsole: debugMode || process.env.NODE_ENV === 'development',
      
      // Privacy settings
      includeSystemPrompt: true,   // Include prompts for debugging
      includeAudioData: false,      // Don't include raw audio (large)
      includeImageData: true,       // Include screenshots for visual context
      
      // Worker mode configuration
      useWorker: enableWorkerMode,
      workerConfig: enableWorkerMode
        ? {
            batchSize: 50,          // Events per batch
            batchIntervalMs: 5000,  // 5 seconds
          }
        : undefined,
      
      // Filter noisy events to reduce log spam
      disableEventTypes: [
        'audio.send' as TraceEventType,
        'audio.receive' as TraceEventType,
      ],
      
      // Metadata for all events
      metadata: {
        environment: process.env.NODE_ENV || 'development',
        timestamp: new Date().toISOString(),
        version: process.env.NEXT_PUBLIC_APP_VERSION || '1.0.0',
        userAgent: typeof window !== 'undefined' ? window.navigator.userAgent : 'server',
        deployment: process.env.NEXT_PUBLIC_DEPLOYMENT || 'unknown',
      },
      
      // Audio aggregation configuration
      audioAggregation: enableAudioAggregation
        ? {
            flushIntervalMs: 30000, // 30 seconds
          }
        : undefined,
    };
    
    return {
      // Basic configuration
      debugLogs: debugMode,
      captureMethod,
      model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
      
      // Authentication with base URL [[memory:5135225]]
      auth: {
        token: jwtToken,
        baseUrl: baseUrl,  // All endpoints derived from this
        onTokenExpired,
      },
      
      // Apply observability configuration
      observability: observabilityConfig,
    };
  };
  ```

  ```tsx Provider Wrapper theme={null}
  /**
   * Example usage in a React component
   */
  export const SammyProviderWrapper: React.FC<{ children: React.ReactNode }> = ({ 
    children 
  }) => {
    const { jwtToken, handleTokenExpired } = useSammyAuth({ isInternal: false });
    
    if (!jwtToken) {
      return <div>Loading authentication...</div>;
    }
    
    const config = createSammyProviderConfig({
      jwtToken,
      onTokenExpired: handleTokenExpired,
      captureMethod: 'render',
      enableWorkerMode: true,      // Use worker for performance
      enableAudioAggregation: true, // Enable audio tracking
      debugMode: false,             // Set to true for debugging
    });
    
    return (
      <SammyAgentProvider config={config}>
        {children}
      </SammyAgentProvider>
    );
  };
  ```
</CodeGroup>

## API Endpoints

When using worker mode, the observability system automatically calls these endpoints:

### Trace Endpoint

```http theme={null}
POST /api/v1/sammy-three/trace/
Content-Type: application/json

{
  events: TraceEvent[],
  conversationData?: {
    // Only for session.start events
    sessionId: string,
    agentMode: 'USER' | 'ADMIN',
    model: string,
    externalUserId?: string,
  },
  metadata?: Record<string, any>
}
```

### Audio Flush Endpoint

```http theme={null}
POST /api/v1/sammy-three/trace/audio/flush
  ?conversationId=X
  &speaker=Y
  &sampleRate=Z
  &startTime=A
  &endTime=B
  &totalBytes=C
Content-Type: multipart/form-data

FormData:
- audio: Blob (PCM format, not WAV)
```

## Event Types Reference

### Core Event Categories

```typescript theme={null}
// Session lifecycle
'session.start' | 'session.end'

// Configuration
'config.set' | 'system_prompt.set'

// Content flow
'content.send' | 'content.receive'
'transcription.input' | 'transcription.output'
'turn.complete'

// Audio
'audio.send' | 'audio.receive'
'audio.recording_start' | 'audio.recording_stop'
'audio.volume_change' | 'audio.gate_state_change'

// Screen capture
'screen_capture.send' | 'screen_capture.critical'
'screen_capture.start' | 'screen_capture.stop'

// Tools
'tool.register' | 'tool.call' | 'tool.response'

// Memory
'memory.search' | 'memory.inject'

// Errors
'error' | 'connection.error' | 'audio.recording_error'

// Agent control
'agent.mute' | 'agent.unmute'
'agent.streaming_start' | 'agent.streaming_stop'
```

### Filtering Events

```typescript theme={null}
// Filter out noisy events
observability: {
  disableEventTypes: [
    'audio.send',
    'audio.receive',
    'transcription.input',
    'transcription.output',
  ],
}
```

## Advanced Features

### High-Resolution Timestamps

<Card title="Precise Event Ordering" icon="clock">
  Events use dual approaches for precise ordering:

  1. **Microsecond timestamps** via `performance.now()`
  2. **Sequence numbers** for guaranteed ordering

  ```typescript theme={null}
  interface TraceEvent {
    timestamp: Date;         // High-resolution timestamp
    sequenceNumber: number;  // Guaranteed ordering (0, 1, 2...)
  }
  ```
</Card>

### Session Statistics

```typescript theme={null}
const agent = useSammyAgentContext();
const stats = agent.agentCoreRef.current?.getObservabilityStatistics();

console.log({
  duration: stats.duration,
  messagesSent: stats.messagesSent,
  audioBytesSent: stats.audioBytesSent,
  toolCalls: stats.toolCalls,
  errors: stats.errors,
});
```

### Custom Event Tracking

```typescript theme={null}
const agent = useSammyAgentContext();
const observability = agent.agentCoreRef.current?.observabilityManager;

await observability?.trackEvent({
  type: 'custom.event',
  data: {
    action: 'user_clicked_button',
    details: { buttonId: 'submit' },
  },
});
```

### Export Trace Data

```typescript theme={null}
const trace = agent.agentCoreRef.current?.getObservabilityTrace();
const json = agent.agentCoreRef.current?.observabilityManager?.exportAsJson();

// Save to file
const blob = new Blob([json], { type: 'application/json' });
const url = URL.createObjectURL(blob);
// ... download logic
```

## Troubleshooting

### Common Issues

<Tabs>
  <Tab title="Worker Not Sending Events">
    **Troubleshooting Steps:**

    1. Check `enabled: true` in config
    2. Verify `useWorker: true` is set
    3. Ensure proper authentication token
    4. Check browser console for worker errors
  </Tab>

  <Tab title="Events Not Batching">
    **Default Settings:**

    * Default batch interval: 5 seconds
    * Default batch size: 50 events
    * Events sent immediately when batch is full
  </Tab>

  <Tab title="Audio Not Flushing">
    **Troubleshooting Steps:**

    1. Verify `audioAggregation.flushIntervalMs` is set
    2. Check authentication is working
    3. Look for console logs: `[ObservabilityWorker] Audio flush sent successfully:`
    4. Ensure worker mode is enabled if using automatic audio flushing
  </Tab>

  <Tab title="High Memory Usage">
    **Solutions:**

    * Disable `includeAudioData` and `includeImageData`
    * Increase `disableEventTypes` filter
    * Reduce `batchSize` in worker config
  </Tab>
</Tabs>

### Debugging Tips

```typescript theme={null}
// Enable debug mode
observability: {
  enabled: true,
  logToConsole: true,  // See all events in console
  useWorker: false,    // Disable worker to see errors
}

// Check worker status
// Look for: [ObservabilityWorker] Trace events sent successfully:

// Monitor performance
const stats = performance.getEntriesByType('measure');
```

## Migration from Callback Mode

<Tabs>
  <Tab title="Before (Manual)">
    ```typescript theme={null}
    observability: {
      callback: async (event) => {
        // Manual API call
        await fetch('/api/trace', { 
          body: JSON.stringify(event) 
        });
      },
      audioAggregation: {
        onFlush: async (data) => {
          // Manual audio upload
          await uploadAudio(data);
        },
      },
    }
    ```
  </Tab>

  <Tab title="After (Worker)">
    ```typescript theme={null}
    observability: {
      enabled: true,
      useWorker: true,  // That's it!
      // Worker handles all API calls automatically
    }
    ```
  </Tab>
</Tabs>

## Performance Considerations

<Columns cols={3}>
  <Card title="Main Thread Impact" icon="gauge">
    **Without Worker**: Each event blocks during JSON serialization

    **With Worker**: Events sent via postMessage (microseconds)
  </Card>

  <Card title="Memory Usage" icon="memory">
    * Audio uses transferable objects (zero-copy)
    * Events batched efficiently in worker
    * Automatic cleanup on session end
  </Card>

  <Card title="Network Optimization" icon="network-wired">
    * Batching reduces API calls by 10-50x
    * Automatic retry with exponential backoff
    * Failed events don't block new ones
  </Card>
</Columns>

## Known Limitations

<Warning>
  Be aware of these current limitations:
</Warning>

1. **Worker Initialization**: Worker mode requires proper authentication setup and may fail silently if auth is misconfigured
2. **Audio Format**: Audio is sent as PCM data, which requires server-side processing to convert to playable formats
3. **Memory Usage**: High-frequency events can consume significant memory if not properly filtered
4. **Browser Compatibility**: Worker mode uses data URLs which work in all modern browsers but may have issues in some extensions

## Best Practices

<Steps>
  <Step title="Start Simple">
    Begin with basic observability enabled and add worker mode later
  </Step>

  <Step title="Filter Events">
    Always use `disableEventTypes` to filter out noisy events in production
  </Step>

  <Step title="Monitor Performance">
    Watch for memory usage and batch sizes in production
  </Step>

  <Step title="Test Worker Mode">
    Thoroughly test worker initialization in your deployment environment
  </Step>
</Steps>

## Quick Start Checklist

<Card title="Implementation Checklist" icon="list-check">
  * [ ] Set `observability.enabled: true`
  * [ ] Configure authentication with `baseUrl`
  * [ ] Add event filtering for production
  * [ ] Test worker mode if using high-frequency events
  * [ ] Monitor console logs during development
  * [ ] Verify API endpoints are receiving data
</Card>