Documentation Index
Fetch the complete documentation index at: https://docs.sammylabs.com/llms.txt
Use this file to discover all available pages before exploring further.
Quick Start Guide
Get your Screen-aware AI agent running in just a few minutes with this step-by-step guide.
Installation
Install the SAMMY package using your preferred package manager:
npm install @sammy-labs/sammy-three
npm install @sammy-labs/sammy-three-ui-kit
Authentication
SAMMY requires JWT authentication. Set up your authentication configuration:
Get Your JWT Token
Obtain a JWT token from your authentication system or SAMMY Labs API.
Create Auth Hook
const useAuth = () => {
const [token, setToken] = useState(null);
useEffect(() => {
// Fetch your JWT token
fetchAuthToken().then(setToken);
}, []);
return {
token,
baseUrl: process.env.NEXT_PUBLIC_SAMMY_API_BASE_URL || 'https://api.sammylabs.com',
onTokenExpired: async () => {
// Handle token refresh
const newToken = await refreshAuthToken();
setToken(newToken);
},
};
};
Environment Variables
Add these environment variables to your .env.local:NEXT_PUBLIC_SAMMY_API_BASE_URL=https://your-api-url.com
NEXT_PUBLIC_APP_VERSION=1.0.0
Basic Usage
1. Wrap Your App
Import the required components and wrap your application:
import {
SammyAgentProvider,
useSammyAgentContext,
} from '@sammy-labs/sammy-three';
import '@sammy-labs/sammy-three/styles.css';
function App() {
const auth = useAuth();
if (!auth.token) {
return <div>Loading authentication...</div>;
}
return (
<SammyAgentProvider
config={{
auth: auth,
captureMethod: 'render', // or 'video'
debugLogs: true,
model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
}}
onError={(error) => console.error('Agent error:', error)}
onTokenExpired={auth.onTokenExpired}
>
<YourApp />
</SammyAgentProvider>
);
}
2. Create Your First Screen-aware AI Agent
Use the context hook to interact with the agent:
function ChatComponent() {
const {
startAgent,
stopAgent,
sendMessage,
toggleMuted,
agentStatus,
agentVolume,
userVolume,
} = useSammyAgentContext();
const handleStart = async () => {
const success = await startAgent({
agentMode: 'user', // 'admin', 'user', or 'sammy'
});
if (success) {
console.log('Agent started successfully');
}
};
return (
<div>
<button
onClick={handleStart}
disabled={agentStatus === 'connecting'}
>
{agentStatus === 'connecting' ? 'Starting...' : 'Start Agent'}
</button>
<button
onClick={stopAgent}
disabled={agentStatus === 'disconnected'}
>
Stop Agent
</button>
<button onClick={() => sendMessage('Hello!')}>
Send Message
</button>
<button onClick={toggleMuted}>
Mute/Unmute
</button>
<div>Status: {agentStatus}</div>
<div>Agent Volume: {Math.round(agentVolume * 100)}%</div>
<div>User Volume: {Math.round(userVolume * 100)}%</div>
</div>
);
}
Enhanced Configuration
Audio Processing
Add advanced audio processing for better voice quality:
const config = {
auth: authConfig,
captureMethod: 'render',
// Audio processing configuration
audioConfig: {
noiseSuppression: {
enabled: true,
enhancementLevel: 'medium', // 'light', 'medium', 'aggressive'
},
noiseGate: {
enabled: true,
threshold: 0.04,
attackTime: 30,
holdTime: 400,
releaseTime: 150,
},
environmentPreset: 'office', // 'office', 'home', 'noisy', 'studio'
},
};
Learn More About Audio
Explore advanced audio processing features including noise suppression, noise gates, and environment presets.
Screen Capture
Configure screen capture for visual context:
const config = {
// ... other config
captureMethod: 'render', // Recommended for web apps
captureConfig: {
quality: 0.8, // JPEG quality (0.0-1.0)
enableAudioAdaptation: true, // Reduce captures during audio
},
};
Screen Capture Guide
Learn about render-based and video-based screen capture options.
Next Steps
Now that you have a basic agent running, explore these advanced features:
Interactive Guides
Add walkthrough experiences with URL-based guide activation
Custom Tools
Extend agent capabilities with custom tools and actions
Memory Management
Enable conversation memory and context tracking
Observability
Monitor performance and track events in production
Common Configuration Examples
Production Setup
const productionConfig = {
auth: {
token: jwtToken,
baseUrl: process.env.NEXT_PUBLIC_SAMMY_API_BASE_URL,
onTokenExpired: handleTokenRefresh,
},
captureMethod: 'render',
debugLogs: false, // Disable in production
// Audio optimized for office environment
audioConfig: {
environmentPreset: 'office',
},
// Enable observability
observability: {
enabled: true,
useWorker: true,
includeAudioData: false, // Exclude large audio data
includeImageData: true,
},
};
Development Setup
const developmentConfig = {
auth: {
token: jwtToken,
baseUrl: 'http://localhost:3000',
onTokenExpired: handleTokenRefresh,
},
captureMethod: 'render',
debugLogs: true, // Enable debug logging
debugAudioPerformance: true, // Enable audio debugging
// Aggressive audio filtering for noisy dev environment
audioConfig: {
environmentPreset: 'noisy',
},
// Full observability for debugging
observability: {
enabled: true,
logToConsole: true,
includeAudioData: true,
includeImageData: true,
},
};
Troubleshooting
Common Issues
Authentication
Microphone
Audio Quality
Token expired errors:// Ensure onTokenExpired is implemented
const config = {
auth: {
token: jwtToken,
onTokenExpired: async () => {
await refreshToken();
},
},
};
Permission denied:import { useMicrophonePermission } from '@sammy-labs/sammy-three';
const { permission, requestPermission } = useMicrophonePermission();
if (permission === 'denied') {
// Show instructions to enable in browser settings
}
Stuttering or poor quality:// Try different environment presets
const config = {
audioConfig: {
environmentPreset: 'noisy', // More aggressive filtering
},
};
What’s Next?
Need Help?
Visit our comprehensive API reference or explore specific feature documentation for detailed implementation guides. 
