SAMMY Implementation Guide
The comprehensive guide for integrating @sammy-labs/sammy-three Screen-aware AI agent into your React application.
What is SAMMY?
SAMMY is a production-ready Screen-aware AI agent package that provides:
Voice Conversations
Real-time voice interactions
Screen Capture
Optimized visual context capture with render or video methods
Memory Management
Semantic search and intelligent context injection
Interactive Guides
Built-in walkthrough system for user onboarding
Quick Start
Get up and running with SAMMY in minutes.
Install the package
npm install @sammy-labs/sammy-three
# or
pnpm add @sammy-labs/sammy-three
# or
yarn add @sammy-labs/sammy-three
Set up authentication
// Create your authentication hook
const useAuth = () => {
return {
token: 'your-jwt-token',
baseUrl: 'https://your-api-url.com',
onTokenExpired: async () => {
// Handle token refresh
await refreshYourToken();
},
};
};
Wrap your application
import { SammyAgentProvider } from '@sammy-labs/sammy-three';
import '@sammy-labs/sammy-three/styles.css';
function App() {
const auth = useAuth();
if (!auth.token) {
return <div>Loading authentication...</div>;
}
return (
<SammyAgentProvider
config={{
auth: auth,
captureMethod: 'render', // or 'video'
debugLogs: true,
model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
}}
onError={(error) => console.error('Agent error:', error)}
onTokenExpired={auth.onTokenExpired}
>
<YourApp />
</SammyAgentProvider>
);
}
Use in your components
import { useSammyAgentContext } from '@sammy-labs/sammy-three';
function ChatComponent() {
const {
startAgent,
stopAgent,
sendMessage,
toggleMuted,
agentStatus,
agentVolume,
userVolume,
} = useSammyAgentContext();
const handleStart = async () => {
const success = await startAgent({
agentMode: 'user', // 'admin', 'user', or 'sammy'
sammyThreeOrganisationFeatureId: 'your-org-id', // optional
guideId: 'guide-123', // optional - for guided experiences
});
if (success) {
console.log('Agent started successfully');
}
};
return (
<div>
<button onClick={handleStart} disabled={agentStatus === 'connecting'}>
{agentStatus === 'connecting' ? 'Starting...' : 'Start Agent'}
</button>
<button onClick={stopAgent} disabled={agentStatus === 'disconnected'}>
Stop Agent
</button>
<button onClick={() => sendMessage('Hello!')}>
Send Message
</button>
<button onClick={toggleMuted}>
Mute/Unmute
</button>
<div>Status: {agentStatus}</div>
<div>Agent Volume: {Math.round(agentVolume * 100)}%</div>
<div>User Volume: {Math.round(userVolume * 100)}%</div>
</div>
);
}
Core Configuration
Configure SAMMY to match your application’s needs.
const config = {
// REQUIRED: Authentication
auth: {
token: 'your-jwt-token',
baseUrl: 'https://your-api.com',
onTokenExpired: () => refreshToken(),
},
// Screen capture method
captureMethod: 'render', // or 'video'
// AI model
model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
// Debug logging
debugLogs: false,
// Voice settings
defaultVoice: 'aoede',
};
Feature Documentation
Explore the full capabilities of SAMMY.
Audio Processing
Advanced noise suppression, noise gate, and environment presets
Screen Capture
Render-based and video-based capture with optimization strategies
Interactive Guides
Built-in walkthrough system with URL activation and progress tracking
Custom Tools
Extend agent capabilities with custom tools and handlers
MCP Integration
Model Context Protocol support for dynamic tool discovery
Observability
Comprehensive event tracking, analytics, and debugging
Context Management
Automatic context tracking and memory management
Performance
Optimization strategies and worker architecture
Environment Variables
Configure SAMMY using environment variables.
# API Configuration
NEXT_PUBLIC_SAMMY_API_BASE_URL=https://your-api.com
NEXT_PUBLIC_APP_VERSION=1.0.0
# Feature Flags
NEXT_PUBLIC_DISABLE_WORKER_MODE=false
# Audio Processing (optional)
NEXT_PUBLIC_KOALA_ACCESS_KEY=your-key
NEXT_PUBLIC_KOALA_MODEL_PATH=/models/koala.pv
# MCP Integration (optional)
NEXT_PUBLIC_MCP_API_KEY=your-mcp-key
# Debug Settings (development only)
NEXT_PUBLIC_DEBUG_AUDIO=true
NEXT_PUBLIC_DEBUG_OBSERVABILITY=true
Common Issues & Solutions
Authentication Errors
Microphone Issues
Audio Stuttering
Noise Issues
Token Expired
// Ensure onTokenExpired is implemented
const config = {
auth: {
token: jwtToken,
onTokenExpired: async () => {
await refreshToken();
},
},
};
Permission Denied
import { useMicrophonePermission } from '@sammy-labs/sammy-three';
const { permission, requestPermission } = useMicrophonePermission();
if (permission === 'denied') {
// Show instructions to enable in browser settings
}
Debug & Optimize
// Enable audio debugging
audioStutterAnalyzer.setDebugMode(true);
audioStutterAnalyzer.getAnalysis();
// Reduce capture quality
const config = {
captureConfig: {
quality: 0.7,
enableAudioAdaptation: true,
},
};
Environment Presets
const config = {
audioConfig: {
environmentPreset: 'noisy', // More aggressive filtering
},
};
Best Practices
Authentication: Always implement token refresh logic and handle expiration gracefully
Audio Configuration: Use environment presets for quick setup and test noise gate threshold
Production: Enable worker mode for better performance and prevent main thread blocking
Capture Methods: Use render for web apps (recommended) and video for full screen needs
Error Handling: Implement both provider-level and component-level error boundaries
Performance: Enable audio-aware capture and use appropriate quality settings
Monitoring: Use observability for production monitoring and debug logs in development
Additional Resources
API Reference
Complete API documentation
Error Handling
Comprehensive error management
Migration Guide
Upgrade from legacy versions
Support
Need help? We’re here for you.
GitHub Repository
View source code and contribute
Contact Support
Email our founders directly
