Skip to main content

​
SAMMY Implementation Guide

The comprehensive guide for integrating @sammy-labs/sammy-three Screen-aware AI agent into your React application.

​
What is SAMMY?

SAMMY is a production-ready Screen-aware AI agent package that provides:

Voice Conversations

Real-time voice interactions

Screen Capture

Optimized visual context capture with render or video methods

Memory Management

Semantic search and intelligent context injection

Interactive Guides

Built-in walkthrough system for user onboarding

​
Quick Start

Get up and running with SAMMY in minutes.
1

Install the package

npm install @sammy-labs/sammy-three
# or
pnpm add @sammy-labs/sammy-three
# or
yarn add @sammy-labs/sammy-three
2

Set up authentication

// Create your authentication hook
const useAuth = () => {
  return {
    token: 'your-jwt-token',
    baseUrl: 'https://your-api-url.com',
    onTokenExpired: async () => {
      // Handle token refresh
      await refreshYourToken();
    },
  };
};
3

Wrap your application

import { SammyAgentProvider } from '@sammy-labs/sammy-three';
import '@sammy-labs/sammy-three/styles.css';

function App() {
  const auth = useAuth();

  if (!auth.token) {
    return <div>Loading authentication...</div>;
  }

  return (
    <SammyAgentProvider
      config={{
        auth: auth,
        captureMethod: 'render', // or 'video'
        debugLogs: true,
        model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
      }}
      onError={(error) => console.error('Agent error:', error)}
      onTokenExpired={auth.onTokenExpired}
    >
      <YourApp />
    </SammyAgentProvider>
  );
}
4

Use in your components

import { useSammyAgentContext } from '@sammy-labs/sammy-three';

function ChatComponent() {
  const {
    startAgent,
    stopAgent,
    sendMessage,
    toggleMuted,
    agentStatus,
    agentVolume,
    userVolume,
  } = useSammyAgentContext();

  const handleStart = async () => {
    const success = await startAgent({
      agentMode: 'user', // 'admin', 'user', or 'sammy'
      sammyThreeOrganisationFeatureId: 'your-org-id', // optional
      guideId: 'guide-123', // optional - for guided experiences
    });
    
    if (success) {
      console.log('Agent started successfully');
    }
  };

  return (
    <div>
      <button onClick={handleStart} disabled={agentStatus === 'connecting'}>
        {agentStatus === 'connecting' ? 'Starting...' : 'Start Agent'}
      </button>
      
      <button onClick={stopAgent} disabled={agentStatus === 'disconnected'}>
        Stop Agent
      </button>
      
      <button onClick={() => sendMessage('Hello!')}>
        Send Message
      </button>
      
      <button onClick={toggleMuted}>
        Mute/Unmute
      </button>

      <div>Status: {agentStatus}</div>
      <div>Agent Volume: {Math.round(agentVolume * 100)}%</div>
      <div>User Volume: {Math.round(userVolume * 100)}%</div>
    </div>
  );
}

​
Core Configuration

Configure SAMMY to match your application’s needs. 
const config = {
  // REQUIRED: Authentication
  auth: {
    token: 'your-jwt-token',
    baseUrl: 'https://your-api.com',
    onTokenExpired: () => refreshToken(),
  },
  
  // Screen capture method
  captureMethod: 'render', // or 'video'
  
  // AI model
  model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
  
  // Debug logging
  debugLogs: false,
  
  // Voice settings
  defaultVoice: 'aoede',
};

​
Feature Documentation

Explore the full capabilities of SAMMY.

Audio Processing

Advanced noise suppression, noise gate, and environment presets

Screen Capture

Render-based and video-based capture with optimization strategies

Interactive Guides

Built-in walkthrough system with URL activation and progress tracking

Custom Tools

Extend agent capabilities with custom tools and handlers

MCP Integration

Model Context Protocol support for dynamic tool discovery

Observability

Comprehensive event tracking, analytics, and debugging

Context Management

Automatic context tracking and memory management

Performance

Optimization strategies and worker architecture

​
Environment Variables

Configure SAMMY using environment variables. 
# API Configuration
NEXT_PUBLIC_SAMMY_API_BASE_URL=https://your-api.com
NEXT_PUBLIC_APP_VERSION=1.0.0

# Feature Flags
NEXT_PUBLIC_DISABLE_WORKER_MODE=false

# Audio Processing (optional)
NEXT_PUBLIC_KOALA_ACCESS_KEY=your-key
NEXT_PUBLIC_KOALA_MODEL_PATH=/models/koala.pv

# MCP Integration (optional)
NEXT_PUBLIC_MCP_API_KEY=your-mcp-key

# Debug Settings (development only)
NEXT_PUBLIC_DEBUG_AUDIO=true
NEXT_PUBLIC_DEBUG_OBSERVABILITY=true

​
Common Issues & Solutions

  • Authentication Errors
  • Microphone Issues
  • Audio Stuttering
  • Noise Issues

​
Token Expired

// Ensure onTokenExpired is implemented
const config = {
  auth: {
    token: jwtToken,
    onTokenExpired: async () => {
      await refreshToken();
    },
  },
};

​
Best Practices

Authentication: Always implement token refresh logic and handle expiration gracefully
Audio Configuration: Use environment presets for quick setup and test noise gate threshold
Production: Enable worker mode for better performance and prevent main thread blocking
Capture Methods: Use render for web apps (recommended) and video for full screen needs
Error Handling: Implement both provider-level and component-level error boundaries
Performance: Enable audio-aware capture and use appropriate quality settings
Monitoring: Use observability for production monitoring and debug logs in development

​
Additional Resources

API Reference

Complete API documentation

Error Handling

Comprehensive error management

Migration Guide

Upgrade from legacy versions

​
Support

Need help? We’re here for you.

GitHub Repository

View source code and contribute

Contact Support

Email our founders directly
⌘I