SAMMY Implementation Guide
The comprehensive guide for integrating @sammy-labs/sammy-three Screen-aware AI agent into your React application.
What is SAMMY?
SAMMY is a production-ready Screen-aware AI agent package that provides:
Voice Conversations
Real-time voice interactions
Screen Capture
Optimized visual context capture with render or video methods
Memory Management
Semantic search and intelligent context injection
Interactive Guides
Built-in walkthrough system for user onboarding
Quick Start
Get up and running with SAMMY in minutes.
Install the package
npm install @sammy-labs/sammy-three
# or
pnpm add @sammy-labs/sammy-three
# or
yarn add @sammy-labs/sammy-three
Set up authentication
// Create your authentication hook
const useAuth = () => {
return {
token: 'your-jwt-token',
baseUrl: 'https://your-api-url.com',
onTokenExpired: async () => {
// Handle token refresh
await refreshYourToken();
},
};
};
Wrap your application
import { SammyAgentProvider } from '@sammy-labs/sammy-three';
import '@sammy-labs/sammy-three/styles.css';
function App() {
const auth = useAuth();
if (!auth.token) {
return <div>Loading authentication...</div>;
}
return (
<SammyAgentProvider
config={{
auth: auth,
captureMethod: 'render', // or 'video'
debugLogs: true,
model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
}}
onError={(error) => console.error('Agent error:', error)}
onTokenExpired={auth.onTokenExpired}
>
<YourApp />
</SammyAgentProvider>
);
}
Use in your components
import { useSammyAgentContext } from '@sammy-labs/sammy-three';
function ChatComponent() {
const {
startAgent,
stopAgent,
sendMessage,
toggleMuted,
agentStatus,
agentVolume,
userVolume,
} = useSammyAgentContext();
const handleStart = async () => {
const success = await startAgent({
agentMode: 'user', // 'admin', 'user', or 'sammy'
sammyThreeOrganisationFeatureId: 'your-org-id', // optional
guideId: 'guide-123', // optional - for guided experiences
});
if (success) {
console.log('Agent started successfully');
}
};
return (
<div>
<button onClick={handleStart} disabled={agentStatus === 'connecting'}>
{agentStatus === 'connecting' ? 'Starting...' : 'Start Agent'}
</button>
<button onClick={stopAgent} disabled={agentStatus === 'disconnected'}>
Stop Agent
</button>
<button onClick={() => sendMessage('Hello!')}>
Send Message
</button>
<button onClick={toggleMuted}>
Mute/Unmute
</button>
<div>Status: {agentStatus}</div>
<div>Agent Volume: {Math.round(agentVolume * 100)}%</div>
<div>User Volume: {Math.round(userVolume * 100)}%</div>
</div>
);
}
Core Configuration
Configure SAMMY to match your application’s needs.
const config = {
// REQUIRED: Authentication
auth: {
token: 'your-jwt-token',
baseUrl: 'https://your-api.com',
onTokenExpired: () => refreshToken(),
},
// Screen capture method
captureMethod: 'render', // or 'video'
// AI model
model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
// Debug logging
debugLogs: false,
// Voice settings
defaultVoice: 'aoede',
};
Feature Documentation
Explore the full capabilities of SAMMY.
Environment Variables
Configure SAMMY using environment variables.
# API Configuration
NEXT_PUBLIC_SAMMY_API_BASE_URL=https://your-api.com
NEXT_PUBLIC_APP_VERSION=1.0.0
# Feature Flags
NEXT_PUBLIC_DISABLE_WORKER_MODE=false
# Audio Processing (optional)
NEXT_PUBLIC_KOALA_ACCESS_KEY=your-key
NEXT_PUBLIC_KOALA_MODEL_PATH=/models/koala.pv
# MCP Integration (optional)
NEXT_PUBLIC_MCP_API_KEY=your-mcp-key
# Debug Settings (development only)
NEXT_PUBLIC_DEBUG_AUDIO=true
NEXT_PUBLIC_DEBUG_OBSERVABILITY=true
Common Issues & Solutions
Authentication Errors
Microphone Issues
Audio Stuttering
Noise Issues
Token Expired
// Ensure onTokenExpired is implemented
const config = {
auth: {
token: jwtToken,
onTokenExpired: async () => {
await refreshToken();
},
},
};
Permission Denied
import { useMicrophonePermission } from '@sammy-labs/sammy-three';
const { permission, requestPermission } = useMicrophonePermission();
if (permission === 'denied') {
// Show instructions to enable in browser settings
}
Debug & Optimize
// Enable audio debugging
audioStutterAnalyzer.setDebugMode(true);
audioStutterAnalyzer.getAnalysis();
// Reduce capture quality
const config = {
captureConfig: {
quality: 0.7,
enableAudioAdaptation: true,
},
};
Environment Presets
const config = {
audioConfig: {
environmentPreset: 'noisy', // More aggressive filtering
},
};
Best Practices
Authentication: Always implement token refresh logic and handle expiration gracefully
Audio Configuration: Use environment presets for quick setup and test noise gate threshold
Production: Enable worker mode for better performance and prevent main thread blocking
Capture Methods: Use render for web apps (recommended) and video for full screen needs
Error Handling: Implement both provider-level and component-level error boundaries
Performance: Enable audio-aware capture and use appropriate quality settings
Monitoring: Use observability for production monitoring and debug logs in development
Additional Resources
Support
Need help? We’re here for you.
