> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sammylabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Implementation Guide

> Complete guide to implementing the SAMMY Screen-aware AI agent in your React application

# SAMMY Implementation Guide

The comprehensive guide for integrating `@sammy-labs/sammy-three` Screen-aware AI agent into your React application.

## What is SAMMY?

SAMMY is a production-ready Screen-aware AI agent package that provides:

<Columns cols={2}>
  <Card title="Voice Conversations" icon="microphone">
    Real-time voice interactions
  </Card>

  <Card title="Screen Capture" icon="desktop">
    Optimized visual context capture with render or video methods
  </Card>

  <Card title="Memory Management" icon="brain">
    Semantic search and intelligent context injection
  </Card>

  <Card title="Interactive Guides" icon="route">
    Built-in walkthrough system for user onboarding
  </Card>
</Columns>

## Quick Start

Get up and running with SAMMY in minutes.

<Steps>
  <Step title="Install the package">
    ```bash theme={null}
    npm install @sammy-labs/sammy-three
    # or
    pnpm add @sammy-labs/sammy-three
    # or
    yarn add @sammy-labs/sammy-three
    ```
  </Step>

  <Step title="Set up authentication">
    ```tsx theme={null}
    // Create your authentication hook
    const useAuth = () => {
      return {
        token: 'your-jwt-token',
        baseUrl: 'https://your-api-url.com',
        onTokenExpired: async () => {
          // Handle token refresh
          await refreshYourToken();
        },
      };
    };
    ```
  </Step>

  <Step title="Wrap your application">
    ```tsx theme={null}
    import { SammyAgentProvider } from '@sammy-labs/sammy-three';
    import '@sammy-labs/sammy-three/styles.css';

    function App() {
      const auth = useAuth();

      if (!auth.token) {
        return <div>Loading authentication...</div>;
      }

      return (
        <SammyAgentProvider
          config={{
            auth: auth,
            captureMethod: 'render', // or 'video'
            debugLogs: true,
            model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
          }}
          onError={(error) => console.error('Agent error:', error)}
          onTokenExpired={auth.onTokenExpired}
        >
          <YourApp />
        </SammyAgentProvider>
      );
    }
    ```
  </Step>

  <Step title="Use in your components">
    ```tsx theme={null}
    import { useSammyAgentContext } from '@sammy-labs/sammy-three';

    function ChatComponent() {
      const {
        startAgent,
        stopAgent,
        sendMessage,
        toggleMuted,
        agentStatus,
        agentVolume,
        userVolume,
      } = useSammyAgentContext();

      const handleStart = async () => {
        const success = await startAgent({
          agentMode: 'user', // 'admin', 'user', or 'sammy'
          sammyThreeOrganisationFeatureId: 'your-org-id', // optional
          guideId: 'guide-123', // optional - for guided experiences
        });
        
        if (success) {
          console.log('Agent started successfully');
        }
      };

      return (
        <div>
          <button onClick={handleStart} disabled={agentStatus === 'connecting'}>
            {agentStatus === 'connecting' ? 'Starting...' : 'Start Agent'}
          </button>
          
          <button onClick={stopAgent} disabled={agentStatus === 'disconnected'}>
            Stop Agent
          </button>
          
          <button onClick={() => sendMessage('Hello!')}>
            Send Message
          </button>
          
          <button onClick={toggleMuted}>
            Mute/Unmute
          </button>

          <div>Status: {agentStatus}</div>
          <div>Agent Volume: {Math.round(agentVolume * 100)}%</div>
          <div>User Volume: {Math.round(userVolume * 100)}%</div>
        </div>
      );
    }
    ```
  </Step>
</Steps>

## Core Configuration

Configure SAMMY to match your application's needs.

<CodeGroup>
  ```tsx Basic Configuration theme={null}
  const config = {
    // REQUIRED: Authentication
    auth: {
      token: 'your-jwt-token',
      baseUrl: 'https://your-api.com',
      onTokenExpired: () => refreshToken(),
    },
    
    // Screen capture method
    captureMethod: 'render', // or 'video'
    
    // AI model
    model: 'models/gemini-2.5-flash-preview-native-audio-dialog',
    
    // Debug logging
    debugLogs: false,
    
    // Voice settings
    defaultVoice: 'aoede',
  };
  ```

  ```tsx Advanced Configuration theme={null}
  const config = {
    auth: { /* ... */ },
    
    // Capture configuration
    captureConfig: {
      frameRate: 30,
      quality: 0.8,
      enableAudioAdaptation: true,
    },
    
    // Audio processing
    audioConfig: {
      noiseSuppression: {
        enabled: true,
        enhancementLevel: 'medium',
      },
      noiseGate: {
        enabled: true,
        threshold: 0.04,
      },
      environmentPreset: 'office',
    },
    
    // Observability
    observability: {
      enabled: true,
      useWorker: true,
      workerConfig: {
        batchSize: 50,
        batchIntervalMs: 5000,
      },
    },
    
    // MCP Integration
    mcp: {
      enabled: true,
      servers: [/* ... */],
    },
    
    // Guides
    guides: {
      enabled: true,
      autoStartFromURL: true,
    },
  };
  ```
</CodeGroup>

## Feature Documentation

Explore the full capabilities of SAMMY.

<Columns cols={2}>
  <Card title="Audio Processing" icon="waveform" href="/features/audio-processing">
    Advanced noise suppression, noise gate, and environment presets
  </Card>

  <Card title="Screen Capture" icon="camera" href="/features/render">
    Render-based and video-based capture with optimization strategies
  </Card>

  <Card title="Interactive Guides" icon="map" href="/features/guides">
    Built-in walkthrough system with URL activation and progress tracking
  </Card>

  <Card title="Custom Tools" icon="wrench" href="/features/tools">
    Extend agent capabilities with custom tools and handlers
  </Card>

  <Card title="MCP Integration" icon="plug" href="/features/mcp">
    Model Context Protocol support for dynamic tool discovery
  </Card>

  <Card title="Observability" icon="chart-line" href="/features/observability">
    Comprehensive event tracking, analytics, and debugging
  </Card>

  <Card title="Context Management" icon="layer-group" href="/features/context-management">
    Automatic context tracking and memory management
  </Card>

  <Card title="Performance" icon="gauge" href="/features/performance">
    Optimization strategies and worker architecture
  </Card>
</Columns>

## Environment Variables

Configure SAMMY using environment variables.

```bash theme={null}
# API Configuration
NEXT_PUBLIC_SAMMY_API_BASE_URL=https://your-api.com
NEXT_PUBLIC_APP_VERSION=1.0.0

# Feature Flags
NEXT_PUBLIC_DISABLE_WORKER_MODE=false

# Audio Processing (optional)
NEXT_PUBLIC_KOALA_ACCESS_KEY=your-key
NEXT_PUBLIC_KOALA_MODEL_PATH=/models/koala.pv

# MCP Integration (optional)
NEXT_PUBLIC_MCP_API_KEY=your-mcp-key

# Debug Settings (development only)
NEXT_PUBLIC_DEBUG_AUDIO=true
NEXT_PUBLIC_DEBUG_OBSERVABILITY=true
```

## Common Issues & Solutions

<Tabs>
  <Tab title="Authentication Errors">
    ### Token Expired

    ```tsx theme={null}
    // Ensure onTokenExpired is implemented
    const config = {
      auth: {
        token: jwtToken,
        onTokenExpired: async () => {
          await refreshToken();
        },
      },
    };
    ```
  </Tab>

  <Tab title="Microphone Issues">
    ### Permission Denied

    ```tsx theme={null}
    import { useMicrophonePermission } from '@sammy-labs/sammy-three';

    const { permission, requestPermission } = useMicrophonePermission();
    if (permission === 'denied') {
      // Show instructions to enable in browser settings
    }
    ```
  </Tab>

  <Tab title="Audio Stuttering">
    ### Debug & Optimize

    ```tsx theme={null}
    // Enable audio debugging
    audioStutterAnalyzer.setDebugMode(true);
    audioStutterAnalyzer.getAnalysis();

    // Reduce capture quality
    const config = {
      captureConfig: {
        quality: 0.7,
        enableAudioAdaptation: true,
      },
    };
    ```
  </Tab>

  <Tab title="Noise Issues">
    ### Environment Presets

    ```tsx theme={null}
    const config = {
      audioConfig: {
        environmentPreset: 'noisy', // More aggressive filtering
      },
    };
    ```
  </Tab>
</Tabs>

## Best Practices

<Check>
  **Authentication**: Always implement token refresh logic and handle expiration gracefully
</Check>

<Check>
  **Audio Configuration**: Use environment presets for quick setup and test noise gate threshold
</Check>

<Check>
  **Production**: Enable worker mode for better performance and prevent main thread blocking
</Check>

<Check>
  **Capture Methods**: Use `render` for web apps (recommended) and `video` for full screen needs
</Check>

<Check>
  **Error Handling**: Implement both provider-level and component-level error boundaries
</Check>

<Check>
  **Performance**: Enable audio-aware capture and use appropriate quality settings
</Check>

<Check>
  **Monitoring**: Use observability for production monitoring and debug logs in development
</Check>

## Additional Resources

<Columns cols={3}>
  <Card title="API Reference" icon="book" href="/features/api-reference">
    Complete API documentation
  </Card>

  <Card title="Error Handling" icon="shield-exclamation" href="/features/error-handling">
    Comprehensive error management
  </Card>

  <Card title="Migration Guide" icon="arrow-up-right" href="/features/migration">
    Upgrade from legacy versions
  </Card>
</Columns>

## Support

Need help? We're here for you.

<Columns cols={2}>
  <Card title="GitHub Repository" icon="github" href="https://github.com/SammyClub/package">
    View source code and contribute
  </Card>

  <Card title="Contact Support" icon="envelope" href="mailto:founders@sammylabs.com">
    Email our founders directly
  </Card>
</Columns>