SAMMY Implementation Guide

The comprehensive guide for integrating @sammy-labs/sammy-three Screen-aware AI agent into your React application.

What is SAMMY?

SAMMY is a production-ready Screen-aware AI agent package that provides:

Voice Conversations

Real-time voice interactions

Screen Capture

Optimized visual context capture with render or video methods

Memory Management

Semantic search and intelligent context injection

Interactive Guides

Built-in walkthrough system for user onboarding

Quick Start

Get up and running with SAMMY in minutes.

Install the package

npm install @sammy-labs/sammy-three
# or
pnpm add @sammy-labs/sammy-three
# or
yarn add @sammy-labs/sammy-three

Set up authentication

// Create your authentication hook
const useAuth = () => {
  return {
    token: 'your-jwt-token',
    baseUrl: 'https://your-api-url.com',
    onTokenExpired: async () => {
      // Handle token refresh
      await refreshYourToken();
    },
  };
};

Wrap your application

import { SammyAgentProvider } from '@sammy-labs/sammy-three';
import '@sammy-labs/sammy-three/styles.css';

function App() {
  const auth = useAuth();

  if (!auth.token) {
    return <div>Loading authentication...</div>;
  }

  return (
    <SammyAgentProvider
      config={{
        auth: auth,
        captureMethod: 'render', // or 'video'
        debugLogs: true,
        model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
      }}
      onError={(error) => console.error('Agent error:', error)}
      onTokenExpired={auth.onTokenExpired}
    >
      <YourApp />
    </SammyAgentProvider>
  );
}

Use in your components

import { useSammyAgentContext } from '@sammy-labs/sammy-three';

function ChatComponent() {
  const {
    startAgent,
    stopAgent,
    sendMessage,
    toggleMuted,
    agentStatus,
    agentVolume,
    userVolume,
  } = useSammyAgentContext();

  const handleStart = async () => {
    const success = await startAgent({
      agentMode: 'user', // 'admin', 'user', or 'sammy'
      sammyThreeOrganisationFeatureId: 'your-org-id', // optional
      guideId: 'guide-123', // optional - for guided experiences
    });
    
    if (success) {
      console.log('Agent started successfully');
    }
  };

  return (
    <div>
      <button onClick={handleStart} disabled={agentStatus === 'connecting'}>
        {agentStatus === 'connecting' ? 'Starting...' : 'Start Agent'}
      </button>
      
      <button onClick={stopAgent} disabled={agentStatus === 'disconnected'}>
        Stop Agent
      </button>
      
      <button onClick={() => sendMessage('Hello!')}>
        Send Message
      </button>
      
      <button onClick={toggleMuted}>
        Mute/Unmute
      </button>

      <div>Status: {agentStatus}</div>
      <div>Agent Volume: {Math.round(agentVolume * 100)}%</div>
      <div>User Volume: {Math.round(userVolume * 100)}%</div>
    </div>
  );
}

Core Configuration

Configure SAMMY to match your application’s needs.

const config = {
  // REQUIRED: Authentication
  auth: {
    token: 'your-jwt-token',
    baseUrl: 'https://your-api.com',
    onTokenExpired: () => refreshToken(),
  },
  
  // Screen capture method
  captureMethod: 'render', // or 'video'
  
  // AI model
  model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
  
  // Debug logging
  debugLogs: false,
  
  // Voice settings
  defaultVoice: 'aoede',
};

const config = {
  auth: { /* ... */ },
  
  // Capture configuration
  captureConfig: {
    frameRate: 30,
    quality: 0.8,
    enableAudioAdaptation: true,
  },
  
  // Audio processing
  audioConfig: {
    noiseSuppression: {
      enabled: true,
      enhancementLevel: 'medium',
    },
    noiseGate: {
      enabled: true,
      threshold: 0.04,
    },
    environmentPreset: 'office',
  },
  
  // Observability
  observability: {
    enabled: true,
    useWorker: true,
    workerConfig: {
      batchSize: 50,
      batchIntervalMs: 5000,
    },
  },
  
  // MCP Integration
  mcp: {
    enabled: true,
    servers: [/* ... */],
  },
  
  // Guides
  guides: {
    enabled: true,
    autoStartFromURL: true,
  },
};

Feature Documentation

Explore the full capabilities of SAMMY.

Audio Processing

Advanced noise suppression, noise gate, and environment presets

Screen Capture

Render-based and video-based capture with optimization strategies

Interactive Guides

Built-in walkthrough system with URL activation and progress tracking

Custom Tools

Extend agent capabilities with custom tools and handlers

MCP Integration

Model Context Protocol support for dynamic tool discovery

Observability

Comprehensive event tracking, analytics, and debugging

Context Management

Automatic context tracking and memory management

Performance

Optimization strategies and worker architecture

Environment Variables

Configure SAMMY using environment variables.

# API Configuration
NEXT_PUBLIC_SAMMY_API_BASE_URL=https://your-api.com
NEXT_PUBLIC_APP_VERSION=1.0.0

# Feature Flags
NEXT_PUBLIC_DISABLE_WORKER_MODE=false

# Audio Processing (optional)
NEXT_PUBLIC_KOALA_ACCESS_KEY=your-key
NEXT_PUBLIC_KOALA_MODEL_PATH=/models/koala.pv

# MCP Integration (optional)
NEXT_PUBLIC_MCP_API_KEY=your-mcp-key

# Debug Settings (development only)
NEXT_PUBLIC_DEBUG_AUDIO=true
NEXT_PUBLIC_DEBUG_OBSERVABILITY=true

Common Issues & Solutions

Authentication Errors
Microphone Issues
Audio Stuttering
Noise Issues

Token Expired

// Ensure onTokenExpired is implemented
const config = {
  auth: {
    token: jwtToken,
    onTokenExpired: async () => {
      await refreshToken();
    },
  },
};

Permission Denied

import { useMicrophonePermission } from '@sammy-labs/sammy-three';

const { permission, requestPermission } = useMicrophonePermission();
if (permission === 'denied') {
  // Show instructions to enable in browser settings
}

Debug & Optimize

// Enable audio debugging
audioStutterAnalyzer.setDebugMode(true);
audioStutterAnalyzer.getAnalysis();

// Reduce capture quality
const config = {
  captureConfig: {
    quality: 0.7,
    enableAudioAdaptation: true,
  },
};

Environment Presets

const config = {
  audioConfig: {
    environmentPreset: 'noisy', // More aggressive filtering
  },
};

Best Practices

Authentication: Always implement token refresh logic and handle expiration gracefully

Audio Configuration: Use environment presets for quick setup and test noise gate threshold

Production: Enable worker mode for better performance and prevent main thread blocking

Capture Methods: Use render for web apps (recommended) and video for full screen needs

Error Handling: Implement both provider-level and component-level error boundaries

Performance: Enable audio-aware capture and use appropriate quality settings

Monitoring: Use observability for production monitoring and debug logs in development

Additional Resources

API Reference

Complete API documentation

Error Handling

Comprehensive error management

Migration Guide

Upgrade from legacy versions

Support

Need help? We’re here for you.

GitHub Repository

View source code and contribute

Contact Support

Email our founders directly

Get Started

Features

Integrations

Implementation Guide

SAMMY Implementation Guide

What is SAMMY?

Voice Conversations

Screen Capture

Memory Management

Interactive Guides

Quick Start

Core Configuration

Feature Documentation

Audio Processing

Screen Capture

Interactive Guides

Custom Tools

MCP Integration

Observability

Context Management

Performance

Environment Variables

Common Issues & Solutions

Token Expired

Permission Denied

Debug & Optimize

Environment Presets

Best Practices

Additional Resources

API Reference

Error Handling

Migration Guide

Support

GitHub Repository

Contact Support

​SAMMY Implementation Guide

​What is SAMMY?

Voice Conversations

Screen Capture

Memory Management

Interactive Guides

​Quick Start

​Core Configuration

​Feature Documentation

Audio Processing

Screen Capture

Interactive Guides

Custom Tools

MCP Integration

Observability

Context Management

Performance

​Environment Variables

​Common Issues & Solutions

​Token Expired

​Permission Denied

​Debug & Optimize

​Environment Presets

​Best Practices

​Additional Resources

API Reference

Error Handling

Migration Guide

​Support

GitHub Repository

Contact Support

SAMMY Implementation Guide

What is SAMMY?

Quick Start

Core Configuration

Feature Documentation

Environment Variables

Common Issues & Solutions

Token Expired

Permission Denied

Debug & Optimize

Environment Presets

Best Practices

Additional Resources

Support