Performance Optimization

Achieve optimal performance with SAMMY Three’s built-in optimization strategies and worker-based architecture.

Overview

SAMMY Three is designed for high performance with multiple optimization layers working together.

Worker Architecture

Offload heavy processing to background workers

Audio-Aware Capture

Intelligent throttling during audio playback

Critical Renders

Automatic capture at conversation boundaries

Worker Architecture

SAMMY Three uses multiple workers to prevent main thread blocking.

Worker Types

Canvas Worker
Observability Worker
DOM Capture Workers

Handles image encoding and compression.

// Automatic - no configuration needed
// Processes screen captures in background
// Falls back to main thread if unavailable

Operations:

JPEG/PNG encoding
Image compression
Resolution scaling
Format conversion

Batches and sends analytics events.

const config = {
  observability: {
    useWorker: true, // Enable worker mode
    workerConfig: {
      batchSize: 50,
      batchIntervalMs: 5000,
    },
  },
};

Operations:

Event batching
API request queuing
Data compression
Retry logic

Generates screenshots efficiently.

// Multiple workers for parallel processing
// Automatic load balancing
// CSP-compliant Data URL approach

Operations:

DOM traversal
Style computation
SVG generation
HTML rendering

Worker Benefits

Non-Blocking UI

Main thread remains responsive during heavy operations

Parallel Processing

Multiple operations execute simultaneously

Automatic Fallback

Gracefully degrades to main thread if workers unavailable

CSP Compliance

Uses Data URLs instead of blob URLs for security

Audio-Aware Capture

Screen capture automatically adapts based on audio state to prevent stuttering.

How It Works

const config = {
  captureConfig: {
    enableAudioAdaptation: true, // Default: true
    quality: 0.8,
  },
};

Normal Mode

When: No audio playing

High-frequency captures (100ms-2s)
Full quality rendering
Maximum detail capture

Audio Mode

When: Agent speaking

Reduced captures (500ms-5s)
Optimized quality
Prevents audio stuttering

Configuration

// Fine-tune audio adaptation
const config = {
  captureConfig: {
    enableAudioAdaptation: true,
    
    // Normal mode settings
    normalInterval: 1000,      // ms between captures
    normalQuality: 0.9,        // JPEG quality
    
    // Audio mode settings
    audioInterval: 3000,       // ms between captures
    audioQuality: 0.7,         // Reduced quality
  },
};

Critical DOM Renders

Ensures fresh visual context at important conversation moments.

Automatic Triggers

Critical renders happen automatically at:

Speech Boundaries

User starts speaking
User stops speaking
Agent starts response
Agent completes response

Interruptions

User interrupts agent
Agent pauses for user
Conversation direction changes

State Changes

Page navigation
Major UI updates
Form submissions
Modal opens/closes

Benefits

Agent always has current visual context

No stale UI information during conversations

Captures happen even when regular capture is throttled

Minimal performance impact with smart timing

Memory Management

Optimize memory usage for long-running sessions.

Automatic Cleanup

// SAMMY Three automatically manages:
- Old screen captures
- Audio buffers
- Processed frames
- Event queues

Manual Optimization

import { useSammyAgentContext } from '@sammy-labs/sammy-three';

function MemoryOptimizedChat() {
  const { clearCache, getMemoryUsage } = useSammyAgentContext();
  
  // Monitor memory usage
  useEffect(() => {
    const interval = setInterval(() => {
      const usage = getMemoryUsage();
      if (usage > 100 * 1024 * 1024) { // 100MB
        clearCache();
      }
    }, 60000); // Check every minute
    
    return () => clearInterval(interval);
  }, []);
  
  return <ChatInterface />;
}

Capture Optimization

Quality Settings

Balance quality and performance based on your needs.

// Best for detailed UIs
const config = {
  captureConfig: {
    quality: 0.95,
    frameRate: 30,
    resolution: {
      maxWidth: 1920,
      maxHeight: 1080,
    },
  },
};

// Good quality with better performance
const config = {
  captureConfig: {
    quality: 0.8,
    frameRate: 15,
    resolution: {
      maxWidth: 1280,
      maxHeight: 720,
    },
  },
};

// Maximum performance for slower devices
const config = {
  captureConfig: {
    quality: 0.6,
    frameRate: 10,
    resolution: {
      maxWidth: 854,
      maxHeight: 480,
    },
    enableAudioAdaptation: true,
  },
};

Capture Methods Comparison

Method	CPU Usage	Quality	Best For
`render`	Low-Medium	High	Web applications
`video`	Medium-High	Variable	Full screen capture

Use render method for most web applications. It’s more efficient and provides consistent quality.

Performance Debugging

Audio Stutter Analysis

Debug and fix audio performance issues.

// Enable debugging
const config = {
  debugAudioPerformance: true,
};

// In browser console
audioStutterAnalyzer.setDebugMode(true);
audioStutterAnalyzer.getAnalysis();

// Returns detailed metrics:
{
  stutterCount: 3,
  underruns: [/* timestamps */],
  longCaptures: [/* durations */],
  averageBufferLevel: 0.8,
  correlationScore: 0.7
}

Performance Monitoring

import { usePerformanceMonitor } from '@sammy-labs/sammy-three';

function PerformanceDebugger() {
  const metrics = usePerformanceMonitor();
  
  return (
    <div className="debug-panel">
      <div>FPS: {metrics.fps}</div>
      <div>Capture Time: {metrics.captureTime}ms</div>
      <div>Memory: {metrics.memoryUsage}MB</div>
      <div>Worker Queue: {metrics.workerQueueSize}</div>
    </div>
  );
}

Optimization Strategies

Device-Based Configuration

Adjust settings based on device capabilities.

function getOptimalConfig() {
  const memory = navigator.deviceMemory || 4; // GB
  const cores = navigator.hardwareConcurrency || 4;
  
  if (memory <= 2 || cores <= 2) {
    // Low-end device
    return {
      captureConfig: {
        quality: 0.6,
        frameRate: 10,
        enableAudioAdaptation: true,
      },
      observability: {
        useWorker: false, // Avoid worker overhead
      },
    };
  }
  
  if (memory <= 4 || cores <= 4) {
    // Mid-range device
    return {
      captureConfig: {
        quality: 0.8,
        frameRate: 15,
        enableAudioAdaptation: true,
      },
    };
  }
  
  // High-end device
  return {
    captureConfig: {
      quality: 0.95,
      frameRate: 30,
      enableAudioAdaptation: false, // Can handle both
    },
  };
}

Network Optimization

Reduce bandwidth usage for slow connections.

function getNetworkOptimizedConfig() {
  const connection = navigator.connection;
  const effectiveType = connection?.effectiveType || '4g';
  
  const configs = {
    'slow-2g': { quality: 0.4, frameRate: 5 },
    '2g': { quality: 0.5, frameRate: 8 },
    '3g': { quality: 0.7, frameRate: 12 },
    '4g': { quality: 0.9, frameRate: 20 },
  };
  
  return {
    captureConfig: configs[effectiveType] || configs['4g'],
  };
}

Best Practices

Profile First: Use browser DevTools to identify bottlenecks before optimizing

Start Conservative: Begin with lower quality settings and increase as needed

Monitor Metrics: Track performance in production to catch issues early

Test on Target Devices: Always test on actual devices your users will use

Use Workers: Keep worker mode enabled unless you have specific reasons not to

Batch Operations: Group multiple operations together when possible

Performance Benchmarks

Expected Performance

Metric	Good	Acceptable	Poor
Capture Time	< 50ms	50-150ms	> 150ms
Audio Latency	< 100ms	100-300ms	> 300ms
Memory Usage	< 50MB	50-150MB	> 150MB
CPU Usage	< 30%	30-60%	> 60%
FPS (during capture)	> 30	15-30	< 15

Optimization Checklist

Advanced Techniques

Custom Worker Implementation

// Create custom worker for specific tasks
class CustomProcessor {
  constructor() {
    this.worker = new Worker(
      URL.createObjectURL(new Blob([`
        self.onmessage = function(e) {
          // Custom processing logic
          const result = processData(e.data);
          self.postMessage(result);
        }
      `], { type: 'application/javascript' }))
    );
  }
  
  process(data) {
    return new Promise((resolve) => {
      this.worker.onmessage = (e) => resolve(e.data);
      this.worker.postMessage(data);
    });
  }
}

Throttling and Debouncing

import { useThrottle, useDebounce } from '@sammy-labs/sammy-three';

function OptimizedCapture() {
  // Throttle captures to max once per second
  const throttledCapture = useThrottle(captureScreen, 1000);
  
  // Debounce UI updates to reduce re-renders
  const debouncedUpdate = useDebounce(updateUI, 300);
  
  return (
    <div onChange={debouncedUpdate}>
      {/* UI components */}
    </div>
  );
}

Audio Processing

Optimize audio for better performance

Observability

Monitor performance in production

Error Handling

Handle performance-related errors

Screen Capture VAD

​Performance Optimization

​Overview

Worker Architecture

Audio-Aware Capture

Critical Renders

​Worker Architecture

​Worker Types

​Worker Benefits

​Audio-Aware Capture

​How It Works

Normal Mode

Audio Mode

​Configuration

​Critical DOM Renders

​Automatic Triggers

​Benefits

​Memory Management

​Automatic Cleanup

​Manual Optimization

​Capture Optimization

​Quality Settings

​Capture Methods Comparison

​Performance Debugging

​Audio Stutter Analysis

​Performance Monitoring

​Optimization Strategies

​Device-Based Configuration

​Network Optimization

​Best Practices

​Performance Benchmarks

​Expected Performance

​Optimization Checklist

​Advanced Techniques

​Custom Worker Implementation

​Throttling and Debouncing

​Related Features

Audio Processing

Observability

Error Handling

Performance Optimization

Overview

Worker Architecture

Worker Types

Worker Benefits

Audio-Aware Capture

How It Works

Configuration

Critical DOM Renders

Automatic Triggers

Benefits

Memory Management

Automatic Cleanup

Manual Optimization

Capture Optimization

Quality Settings

Capture Methods Comparison

Performance Debugging

Audio Stutter Analysis

Performance Monitoring

Optimization Strategies

Device-Based Configuration

Network Optimization

Best Practices

Performance Benchmarks

Expected Performance

Optimization Checklist

Advanced Techniques

Custom Worker Implementation

Throttling and Debouncing

Related Features