> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sammylabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio Processing

> Advanced audio processing capabilities for cleaner voice input and better agent interactions

# Audio Processing

SAMMY Three includes advanced audio processing capabilities to ensure high-quality voice interactions in any environment.

## Overview

The audio processing system provides multiple layers of enhancement to deliver crystal-clear voice communication.

<Columns cols={3}>
  <Card title="Noise Suppression" icon="volume-slash">
    Remove background noise with browser-native or AI-powered suppression
  </Card>

  <Card title="Noise Gate" icon="filter">
    Software-based filtering to eliminate ambient sounds
  </Card>

  <Card title="Environment Presets" icon="sliders">
    Pre-configured settings optimized for different environments
  </Card>
</Columns>

## Configuration

Enable audio processing in your SAMMY Three configuration.

```tsx theme={null}
const config = {
  audioConfig: {
    noiseSuppression: {
      enabled: true,
      enhancementLevel: 'medium',
    },
    noiseGate: {
      enabled: true,
      threshold: 0.04,
    },
    environmentPreset: 'office',
  },
};
```

## Noise Suppression

Two powerful noise suppression modes are available to match your needs.

<Tabs>
  <Tab title="Browser Native">
    ### Default Browser Suppression

    Uses the browser's built-in echo cancellation and noise suppression capabilities.

    ```tsx theme={null}
    const config = {
      audioConfig: {
        noiseSuppression: {
          enabled: true,
          enhancementLevel: 'light', // 'light', 'medium', 'aggressive'
        },
      },
    };
    ```

    **Pros:**

    * No additional configuration required
    * Works on all modern browsers
    * Zero latency
    * Free to use

    **Cons:**

    * Limited effectiveness in very noisy environments
    * Quality varies by browser and device
  </Tab>

  <Tab title="Koala AI">
    ### Advanced AI Suppression

    Professional-grade noise removal using Koala AI's neural networks.

    ```tsx theme={null}
    const config = {
      audioConfig: {
        noiseSuppression: {
          enabled: true,
          enhancementLevel: 'medium',
          // Koala-specific configuration
          accessKey: process.env.KOALA_ACCESS_KEY,
          modelPath: '/models/koala_model.pv',
        },
      },
    };
    ```

    **Pros:**

    * Superior noise removal quality
    * Consistent across all platforms
    * Handles extreme noise conditions
    * Professional broadcast quality

    **Cons:**

    * Requires Koala access key
    * Additional model download
    * Small processing latency
  </Tab>
</Tabs>

### Enhancement Levels

Choose the right suppression level for your environment:

<Steps>
  <Step title="Light" icon="feather">
    **Best for:** Quiet offices, home offices

    Minimal processing that preserves natural voice quality while removing light background noise.

    ```tsx theme={null}
    enhancementLevel: 'light'
    ```
  </Step>

  <Step title="Medium" icon="balance-scale">
    **Best for:** Open offices, cafes

    Balanced approach that removes moderate noise while maintaining voice clarity.

    ```tsx theme={null}
    enhancementLevel: 'medium'
    ```
  </Step>

  <Step title="Aggressive" icon="shield">
    **Best for:** Noisy environments, public spaces

    Maximum noise removal that prioritizes voice isolation over natural sound.

    ```tsx theme={null}
    enhancementLevel: 'aggressive'
    ```
  </Step>
</Steps>

## Noise Gate

Software-based noise gate filters out background noise between speech segments.

### How It Works

The noise gate acts like an automatic mute button, opening only when you speak and closing during silence to eliminate ambient noise.

```tsx theme={null}
const config = {
  audioConfig: {
    noiseGate: {
      enabled: true,
      threshold: 0.04,      // Volume threshold (0-1)
      attackTime: 30,       // How fast gate opens (ms)
      holdTime: 400,        // Hold open during pauses (ms)
      releaseTime: 150,     // How fast gate closes (ms)
    },
  },
};
```

### Configuration Parameters

<ResponseField name="threshold" type="number" default={0.04}>
  Volume level (0-1) required to open the gate. Lower values are more sensitive.

  * **0.02-0.03**: Very sensitive, good for quiet environments
  * **0.04-0.06**: Standard setting for most environments
  * **0.07-0.10**: Less sensitive, for noisy environments
</ResponseField>

<ResponseField name="attackTime" type="number" default={30}>
  Time in milliseconds for the gate to fully open when speech is detected.

  * **10-20ms**: Very fast, may clip beginning of words
  * **30-50ms**: Standard, natural sounding
  * **60-100ms**: Slower, smoother transitions
</ResponseField>

<ResponseField name="holdTime" type="number" default={400}>
  Time in milliseconds to keep gate open after speech stops, preventing choppy audio.

  * **200-300ms**: Quick release, good for fast conversations
  * **400-500ms**: Standard, handles normal pauses
  * **600-800ms**: Longer hold, better for thoughtful speech
</ResponseField>

<ResponseField name="releaseTime" type="number" default={150}>
  Time in milliseconds for the gate to fully close after hold time expires.

  * **50-100ms**: Fast close, may sound abrupt
  * **150-200ms**: Standard, natural fade
  * **250-400ms**: Slow close, very smooth
</ResponseField>

## Environment Presets

Pre-configured audio settings optimized for common environments.

<CodeGroup>
  ```tsx Office theme={null}
  // Quiet office environment
  {
    audioConfig: {
      environmentPreset: 'office',
    },
  }

  // Automatically configures:
  // - Light noise suppression
  // - Low noise gate threshold (0.03)
  // - Standard timing parameters
  ```

  ```tsx Home theme={null}
  // Home environment with moderate noise
  {
    audioConfig: {
      environmentPreset: 'home',
    },
  }

  // Automatically configures:
  // - Medium noise suppression
  // - Medium threshold (0.04)
  // - Balanced parameters
  ```

  ```tsx Noisy theme={null}
  // Cafe, public space, or noisy environment
  {
    audioConfig: {
      environmentPreset: 'noisy',
    },
  }

  // Automatically configures:
  // - Aggressive noise suppression
  // - Higher threshold (0.06)
  // - Longer hold times
  ```

  ```tsx Studio theme={null}
  // Professional recording environment
  {
    audioConfig: {
      environmentPreset: 'studio',
    },
  }

  // Automatically configures:
  // - Minimal processing
  // - Very low threshold (0.02)
  // - Fast response times
  ```

  ```tsx Custom theme={null}
  // Manual configuration
  {
    audioConfig: {
      environmentPreset: 'custom',
      // Specify all parameters manually
    },
  }
  ```
</CodeGroup>

## Audio Debugging

Built-in tools to diagnose and fix audio issues.

### Stutter Analyzer

Debug audio stuttering and performance issues:

```tsx theme={null}
// Enable performance monitoring
const config = {
  debugAudioPerformance: true,
};

// In browser console:
audioStutterAnalyzer.setDebugMode(true);  // Start collection
audioStutterAnalyzer.getAnalysis();       // View analysis
audioStutterAnalyzer.clear();             // Clear buffer

// The analyzer tracks:
// - Audio underruns (stuttering)
// - Long DOM captures during audio
// - Buffer statistics
// - Correlation between renders and stutters
```

### Common Issues and Solutions

<Tabs>
  <Tab title="Echo Problems">
    **Issue:** Hearing echo or feedback

    **Solutions:**

    ```tsx theme={null}
    // Enable echo cancellation
    {
      audioConfig: {
        noiseSuppression: {
          enabled: true,
          // Echo cancellation is automatic
        },
      },
    }
    ```

    **Additional Tips:**

    * Use headphones when possible
    * Reduce speaker volume
    * Increase distance between mic and speakers
  </Tab>

  <Tab title="Choppy Audio">
    **Issue:** Words getting cut off

    **Solutions:**

    ```tsx theme={null}
    // Adjust noise gate timing
    {
      audioConfig: {
        noiseGate: {
          enabled: true,
          threshold: 0.03,     // Lower threshold
          attackTime: 20,      // Faster opening
          holdTime: 500,       // Longer hold
        },
      },
    }
    ```
  </Tab>

  <Tab title="Background Noise">
    **Issue:** Too much background noise

    **Solutions:**

    ```tsx theme={null}
    // Use aggressive filtering
    {
      audioConfig: {
        environmentPreset: 'noisy',
        // or
        noiseSuppression: {
          enabled: true,
          enhancementLevel: 'aggressive',
        },
      },
    }
    ```
  </Tab>

  <Tab title="Muffled Voice">
    **Issue:** Voice sounds unnatural or muffled

    **Solutions:**

    ```tsx theme={null}
    // Reduce processing
    {
      audioConfig: {
        noiseSuppression: {
          enabled: true,
          enhancementLevel: 'light',
        },
        noiseGate: {
          enabled: false, // Try disabling gate
        },
      },
    }
    ```
  </Tab>
</Tabs>

## Best Practices

<Check>
  **Test in Target Environment**: Always test audio settings in the actual environment where the agent will be used
</Check>

<Check>
  **Start with Presets**: Use environment presets as a starting point, then fine-tune if needed
</Check>

<Check>
  **Monitor Performance**: Enable debug mode during development to catch audio issues early
</Check>

<Check>
  **User Feedback**: Provide visual feedback for volume levels and mute status
</Check>

<Check>
  **Fallback Options**: Have a text input fallback for environments where audio isn't suitable
</Check>

## Advanced Configuration

### Combining Multiple Techniques

Layer different audio processing techniques for optimal results:

```tsx theme={null}
const config = {
  audioConfig: {
    // Start with a preset
    environmentPreset: 'office',
    
    // Override specific settings
    noiseSuppression: {
      enabled: true,
      enhancementLevel: 'medium',
    },
    
    // Fine-tune noise gate
    noiseGate: {
      enabled: true,
      threshold: 0.045,
      holdTime: 450,
    },
  },
};
```

### Dynamic Adjustment

Adjust audio settings based on user feedback or environment detection:

```tsx theme={null}
function adjustAudioForEnvironment(noiseLevel: 'low' | 'medium' | 'high') {
  const presets = {
    low: { environmentPreset: 'office' },
    medium: { environmentPreset: 'home' },
    high: { environmentPreset: 'noisy' },
  };
  
  // Update configuration dynamically
  updateAgentConfig({
    audioConfig: presets[noiseLevel],
  });
}
```

## Platform Considerations

<Info>
  Audio processing behavior may vary across different platforms and browsers. Always test on your target platforms.
</Info>

### Browser Support

| Feature            | Chrome | Firefox | Safari | Edge |
| ------------------ | ------ | ------- | ------ | ---- |
| Native Suppression | ✅      | ✅       | ✅      | ✅    |
| Echo Cancellation  | ✅      | ✅       | ✅      | ✅    |
| Noise Gate         | ✅      | ✅       | ✅      | ✅    |
| Koala AI           | ✅      | ✅       | ⚠️     | ✅    |

<Note>
  Safari may require additional permissions for advanced audio processing features.
</Note>

## Related Features

<Columns cols={2}>
  <Card title="VAD Configuration" icon="microphone-lines" href="/features/vad">
    Configure voice activity detection settings
  </Card>

  <Card title="Performance Optimization" icon="gauge" href="/features/performance">
    Optimize audio processing for better performance
  </Card>
</Columns>