> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sammylabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Screen Capture System

> Simple and reliable screen capture implementation for Sammy Agent with DOM-to-image conversion

<Note>
  The **Simple Render Capture System** is a streamlined screen capture implementation that captures DOM content every second and sends it to the Gemini AI service. This is the current active implementation used in the sammy-three package.
</Note>

## Performance Optimization

<Info>
  **Optimize for Your Use Case**: Screen capture performance can be tuned based on your specific needs. The system offers flexible configuration options to balance quality, frequency, and performance.
</Info>

## Performance Tuning Options

<CodeGroup>
  ```tsx Performance-Optimized Configuration theme={null}
  import { SammyAgentProvider } from '@sammy-labs/sammy-three';

  const config = {
    auth: { /* your auth config */ },
    
    // Screen capture performance optimization
    screenCapture: {
      method: 'render',
      
      // Optimize capture frequency for your needs
      checkInterval: 5000,  // 5 seconds for less frequent updates
      
      // Balance quality and performance
      jpegQuality: 0.3,     // Lower quality for faster processing
      
      // Optimize dimensions for your use case
      maxWidth: 1280,       // Smaller dimensions for better performance
      maxHeight: 720,       // Adjust based on your UI complexity
      
      // Fine-tune DOM observation
      domChangeDetection: {
        debounceMs: 500,    // Optimize mutation debouncing
        observerConfig: {
          subtree: false,   // Focus on direct changes only
          childList: true,  // Monitor content changes
          attributes: false,// Skip attribute changes if not needed
          characterData: false
        }
      }
    }
  };

  <SammyAgentProvider config={config}>
    <App />
  </SammyAgentProvider>
  ```

  ```tsx High-Performance Alternative theme={null}
  // For maximum performance - uses native browser capture
  const config = {
    screenCapture: {
      method: 'video',  // Native getDisplayMedia() API
      maxWidth: 1280,
      maxHeight: 720,
      jpegQuality: 0.5
    }
  };
  ```
</CodeGroup>

## Architecture Overview

<Card title="System Features" icon="camera">
  The system uses a simplified approach for reliable screen capture:

  * **Fixed 1-second intervals** for capturing (configurable)
  * **domToPng** from modern-screenshot library for DOM-to-image conversion
  * **Fallback mechanism** when capture times out
  * **Configurable DOM change detection** and performance settings
  * **Support for targetElement** to capture specific DOM elements
</Card>

## Component Flow

```mermaid theme={null}
graph TD
    A[SammyAgentProvider] -->|config props| B[useScreenCapture Hook]
    B -->|render method| C[useSimpleRenderCapture Hook]
    
    C --> D{Capture Element Resolution}
    D -->|1. targetElement| E[Resolve Target Element]
    D -->|2. scope: context| F[Context Element Ref]
    D -->|3. scope: document| G[Find App Container]
    
    E --> H[Capture Element]
    F --> H
    G --> H
    
    H --> I[Start Capture Interval]
    I -->|Every 1000ms| J[captureAndSend Function]
    
    J --> K{Capture Attempt}
    K -->|Success| L[domToPng Conversion]
    K -->|Timeout 2s| M[Fallback Canvas]
    
    L --> N[Base64 Encoding]
    M --> N
    
    N --> O[Send to GenAI Client]
    O --> P[client.sendRealtimeInput]
    
    P --> Q[Gemini Live API]
```

## Configuration Flow

<Tabs>
  <Tab title="Provider Config">
    ```typescript theme={null}
    // SammyAgentProvider
    config: {
      captureMethod: 'render',     // Use render method (not video)
      debugLogs: true,              // Enable debug logging
      targetElement: '#my-div',     // Optional: specific element to capture
    }
    ```
  </Tab>

  <Tab title="Hook Flow">
    ```typescript theme={null}
    // useScreenCapture
    {
      method: config?.captureMethod ?? 'render',
      debugLogs: config?.debugLogs ?? false,
      targetElement: config?.targetElement,
    }
    ↓
    // useSimpleRenderCapture
    {
      ...config,
      scope: config.scope || 'document',
      debugLogs: config.debugLogs ?? true,
    }
    ```
  </Tab>
</Tabs>

## Capture Element Resolution

<Info>
  The system determines which element to capture using a **priority-based resolution** system.
</Info>

### Priority Order

<Steps>
  <Step title="targetElement (Highest Priority)">
    Can be: CSS selector string, `HTMLElement`, or `RefObject<HTMLElement>`

    Examples: `'#my-capture-area'`, `document.getElementById('app')`, `useRef()`
  </Step>

  <Step title="scope: 'context'">
    Uses the `contextElementRef` from the hook

    Internal wrapper element managed by sammy-three
  </Step>

  <Step title="scope: 'document' (Default)">
    Searches for common app containers in order:

    * `#root`
    * `#app`
    * `[data-testid="app"]`
    * `<main>`
    * `document.body` (last resort)
  </Step>
</Steps>

### Resolution Logic

```typescript theme={null}
// Resolution logic
if (stableConfig.targetElement) {
  captureElement = resolveTargetElement(stableConfig.targetElement);
} else if (effectiveScope === 'context' && contextElementRef.current) {
  captureElement = contextElementRef.current;
} else {
  // Find first available app container
  const candidates = ['#root', '#app', '[data-testid="app"]', 'main', 'body'];
  for (const candidate of candidates) {
    if (element = document.querySelector(candidate)) {
      captureElement = element;
      break;
    }
  }
}
```

## Capture Process

### Interval Setup

```typescript theme={null}
// When capturing starts
if (isCapturing && !intervalRef.current && clientRef.current) {
  // Set up 1-second interval
  intervalRef.current = window.setInterval(() => {
    captureAndSend();
  }, 1000);
  
  // Perform initial capture immediately
  captureAndSend();
}
```

### Capture Function Steps

<Steps>
  <Step title="Pre-flight Checks">
    * Verify `isCapturing` is true
    * Verify `clientRef.current` exists
    * Verify `captureElement` exists
  </Step>

  <Step title="DOM to PNG Conversion">
    ```typescript theme={null}
    // Attempt capture with 2-second timeout
    const capturePromise = domToPng(captureElement, {
      width: Math.min(captureElement.scrollWidth, 1920),
      height: Math.min(captureElement.scrollHeight, 1080),
    });

    const timeoutPromise = new Promise((_, reject) =>
      setTimeout(() => reject(new Error('domToPng timeout after 2 seconds')), 2000)
    );

    dataUrl = await Promise.race([capturePromise, timeoutPromise]);
    ```
  </Step>

  <Step title="Fallback Mechanism">
    When timeout occurs:

    ```typescript theme={null}
    // Create fallback canvas
    const canvas = document.createElement('canvas');
    canvas.width = 800;
    canvas.height = 600;
    const ctx = canvas.getContext('2d');

    // Draw placeholder content
    ctx.fillStyle = '#f8f9fa';
    ctx.fillRect(0, 0, 800, 600);
    ctx.fillText('Screen capture in progress...', 400, 280);
    ctx.fillText(`Element: ${captureElement.tagName}`, 400, 310);
    ctx.fillText(`Size: ${width}x${height}`, 400, 330);
    ctx.fillText(`Timestamp: ${time}`, 400, 350);

    dataUrl = canvas.toDataURL('image/png');
    ```
  </Step>

  <Step title="Data Transmission">
    ```typescript theme={null}
    // Extract base64 data (remove data:image/png;base64, prefix)
    const base64Data = dataUrl.split(',')[1];

    // Send to Gemini
    clientRef.current.sendRealtimeInput([
      { mimeType: 'image/png', data: base64Data }
    ]);
    ```
  </Step>
</Steps>

## Fallback Canvas

<Warning>
  The fallback canvas with "Screen capture in progress..." message appears under these conditions:
</Warning>

<Columns cols={3}>
  <Card title="domToPng Timeout" icon="clock">
    * Large or complex DOM structure
    * Heavy CSS animations or transforms
    * Many external resources
    * Browser performance issues
  </Card>

  <Card title="domToPng Errors" icon="exclamation-triangle">
    * CORS issues with external resources
    * Invalid DOM structure
    * Memory constraints
    * Browser security restrictions
  </Card>

  <Card title="Element Issues" icon="eye-slash">
    * Element has zero dimensions
    * Element is hidden or off-screen
    * Element contains problematic content (iframes, canvas, video)
  </Card>
</Columns>

### Fallback Canvas Details

<Card title="Canvas Specifications" icon="palette">
  ```
  Size: 800x600 pixels
  Background: #f8f9fa (light gray)
  Text Color: #333 (dark gray)
  Font: 16px/12px Arial
  Content:
    - "Screen capture in progress..."
    - Element type and ID
    - Original element dimensions
    - Current timestamp
  ```
</Card>

## Modern Screenshot Process

```mermaid theme={null}
graph LR
    A[DOM Node] --> B[Clone & Process]
    B --> C[Foreign Object SVG]
    C --> D[SVG Data URL]
    D --> E[Image Element]
    E --> F[Canvas]
    F --> G[PNG Data URL]
```

### Detailed Steps

<Steps>
  <Step title="DOM Cloning">
    Deep clone with computed styles
  </Step>

  <Step title="Resource Embedding">
    Inline external resources
  </Step>

  <Step title="SVG Creation">
    Wrap in ForeignObject SVG
  </Step>

  <Step title="Data URL Conversion">
    Convert SVG to data URL
  </Step>

  <Step title="Image Loading">
    Create image from data URL
  </Step>

  <Step title="Canvas Rendering">
    Draw image to canvas
  </Step>

  <Step title="PNG Export">
    Export canvas as PNG data URL
  </Step>
</Steps>

## Usage Examples

### Basic Usage

```tsx theme={null}
<SammyAgentProvider
  config={{
    auth: authConfig,
    screenCapture: {
      method: 'render',
      // Default settings - captures every 1 second
    },
    debugLogs: true,
  }}
>
  <App />
</SammyAgentProvider>
```

### With Target Element

<CodeGroup>
  ```tsx CSS Selector theme={null}
  <SammyAgentProvider
    config={{
      auth: authConfig,
      captureMethod: 'render',
      targetElement: '#dashboard-content',
      debugLogs: true,
    }}
  >
    <div id="dashboard-content">
      {/* This will be captured */}
    </div>
    <div id="sidebar">
      {/* This won't be captured */}
    </div>
  </SammyAgentProvider>
  ```

  ```tsx React Ref theme={null}
  function App() {
    const captureRef = useRef<HTMLDivElement>(null);
    
    return (
      <SammyAgentProvider
        config={{
          auth: authConfig,
          captureMethod: 'render',
          targetElement: captureRef,
        }}
      >
        <div ref={captureRef}>
          {/* Captured content */}
        </div>
      </SammyAgentProvider>
    );
  }
  ```

  ```tsx CSS Class theme={null}
  <SammyAgentProvider
    config={{
      auth: authConfig,
      captureMethod: 'render',
      targetElement: '.main-dashboard',
      debugLogs: true,
    }}
  >
    <div className="sidebar">Sidebar (not captured)</div>
    <div className="main-dashboard">
      {/* This content will be captured */}
      <h1>Dashboard</h1>
      <div>Charts and data...</div>
    </div>
  </SammyAgentProvider>
  ```
</CodeGroup>

## Configuration Options

### Complete Configuration Reference

<ParamField path="method" type="'render' | 'video'" default="'render'">
  Capture method. Use `'video'` to avoid DOM cloning entirely.
</ParamField>

<ParamField path="checkInterval" type="number" default="1000">
  How often to capture in milliseconds. Adjust based on your update frequency needs.
</ParamField>

<ParamField path="jpegQuality" type="number" default="0.5">
  JPEG quality from 0.0 to 1.0. Balance quality and file size for your use case.
</ParamField>

<ParamField path="maxWidth" type="number" default="1920">
  Maximum width for captured images. Optimize based on your UI requirements.
</ParamField>

<ParamField path="maxHeight" type="number" default="1080">
  Maximum height for captured images. Adjust for your display needs.
</ParamField>

<ParamField path="minInterval" type="number" default="100">
  Minimum time between captures in milliseconds when changes are detected.
</ParamField>

<ParamField path="maxInterval" type="number" default="5000">
  Maximum time between captures in milliseconds when no changes are detected.
</ParamField>

<ParamField path="useHashing" type="boolean" default="true">
  Enable smart deduplication using image hashing to skip identical frames.
</ParamField>

<ParamField path="scope" type="'context' | 'document'" default="'document'">
  Scope of capture. Use `'context'` to capture only wrapped content.
</ParamField>

<ParamField path="domChangeDetection.debounceMs" type="number" default="300">
  Debounce time after DOM mutations stop before capturing.
</ParamField>

<ParamField path="domChangeDetection.observerConfig" type="MutationObserverInit" default="{ subtree: true, childList: true, attributes: true, characterData: false }">
  MutationObserver configuration for detecting DOM changes.
</ParamField>

### Target Element Options

<Info>
  The `targetElement` option accepts three types of values:
</Info>

| Type                     | Example                             | Description              |
| ------------------------ | ----------------------------------- | ------------------------ |
| `string`                 | `'#my-div'`, `'.capture-area'`      | CSS selector string      |
| `HTMLElement`            | `document.getElementById('my-div')` | Direct element reference |
| `RefObject<HTMLElement>` | `useRef<HTMLDivElement>()`          | React ref object         |

### Configuration Presets

<Tabs>
  <Tab title="Performance Focused">
    **Optimized for maximum performance and efficiency**

    ```tsx theme={null}
    const config = {
      auth: authConfig,
      screenCapture: {
        method: 'video',        // Native browser capture
        checkInterval: 10000,   // 10 seconds for efficiency
        jpegQuality: 0.3,
        maxWidth: 854,
        maxHeight: 480,
        useHashing: true,
        domChangeDetection: {
          debounceMs: 1000,
          observerConfig: {
            subtree: false,
            attributes: false,
            characterData: false
          }
        }
      }
    };
    ```
  </Tab>

  <Tab title="Balanced">
    **Good balance of quality and performance**

    ```tsx theme={null}
    const config = {
      auth: authConfig,
      screenCapture: {
        method: 'render',
        checkInterval: 3000,    // 3 seconds
        jpegQuality: 0.5,
        maxWidth: 1280,
        maxHeight: 720,
        useHashing: true,
        domChangeDetection: {
          debounceMs: 500,
          observerConfig: {
            subtree: false,
            childList: true,
            attributes: false
          }
        }
      }
    };
    ```
  </Tab>

  <Tab title="Quality Focused">
    **Optimized for maximum visual quality and detail**

    ```tsx theme={null}
    const config = {
      auth: authConfig,
      screenCapture: {
        method: 'render',
        checkInterval: 1000,    // 1 second for responsiveness
        jpegQuality: 0.8,
        maxWidth: 1920,
        maxHeight: 1080,
        useHashing: true,
        domChangeDetection: {
          debounceMs: 300,
          observerConfig: {
            subtree: true,
            childList: true,
            attributes: true
          }
        }
      }
    };
    ```
  </Tab>
</Tabs>

## Key Differences from Complex Render

| Feature              | Simple Render | Complex Render             |
| -------------------- | ------------- | -------------------------- |
| Capture Interval     | Fixed 1s      | Dynamic (100ms-2s)         |
| DOM Change Detection | No            | Yes (MutationObserver)     |
| Audio Adaptation     | No            | Yes (throttling)           |
| Frame Hashing        | No            | Yes (deduplication)        |
| Critical Renders     | No            | Yes (conversation events)  |
| Worker Threads       | No            | Yes (optional)             |
| Performance Mode     | Single        | Multiple (worker/main)     |
| Capture Method       | domToPng only | domToCanvas with fallbacks |

## Debug Logging

<Note>
  When `debugLogs: true`, the system provides extensive logging:
</Note>

```
🔧 [EXPLICIT-RENDER-abc123] Hook initialized with config
🎯 [EXPLICIT-RENDER-abc123] Resolving targetElement
🎯 [EXPLICIT-RENDER-abc123] ✅ Successfully resolved targetElement
🎬 [EXPLICIT-RENDER] captureAndSend called
🎨 [EXPLICIT-RENDER] Starting domToPng capture...
✅ [EXPLICIT-RENDER] domToPng succeeded
📤 [EXPLICIT-RENDER] Sending to client...
✅ [EXPLICIT-RENDER] Frame sent successfully
📸 [EXPLICIT-RENDER-abc123] Currently capturing: #my-div
```

## Performance Characteristics

### Timing Analysis

<Columns cols={2}>
  <Card title="Capture Frequency" icon="clock">
    Every 1000ms (1 FPS)
  </Card>

  <Card title="domToPng Timeout" icon="hourglass">
    2000ms maximum
  </Card>

  <Card title="Fallback Canvas" icon="palette">
    \~5-10ms creation
  </Card>

  <Card title="Base64 Encoding" icon="code">
    \~10-20ms
  </Card>
</Columns>

### Resource Usage

<Columns cols={2}>
  <Card title="CPU" icon="microchip">
    Medium (spikes during capture)
  </Card>

  <Card title="Memory" icon="memory">
    Low-Medium (temporary canvas/image)
  </Card>

  <Card title="Network" icon="network-wired">
    \~50-200KB per frame (compressed PNG)
  </Card>

  <Card title="UI Impact" icon="gauge">
    Minimal (no worker threads)
  </Card>
</Columns>

## Error Handling

### Timeout Recovery

```typescript theme={null}
if (error.message.includes('timeout')) {
  // Clear and restart interval
  clearInterval(intervalRef.current);
  intervalRef.current = null;
  // Will be restarted by useEffect
}
```

### Client Loss Protection

```typescript theme={null}
// Check client exists during interval tick
if (!clientRef.current) {
  console.error('Client lost during interval');
  clearInterval(intervalRef.current);
  return;
}
```

### Capture Element Validation

```typescript theme={null}
if (!captureElement || captureElement.offsetHeight === 0) {
  console.log('Capture skipped - invalid element');
  return;
}
```

## Benefits of Target Element Capture

<Columns cols={2}>
  <Card title="Reduced Size" icon="compress">
    Capture only the relevant part of your UI
  </Card>

  <Card title="Better Performance" icon="gauge">
    Smaller capture area means faster processing
  </Card>

  <Card title="Focused Context" icon="crosshairs">
    AI agent sees only the important content
  </Card>

  <Card title="Flexible Integration" icon="puzzle-piece">
    Works with any existing DOM structure
  </Card>
</Columns>

## Important Notes

<Warning>
  Keep these points in mind when using screen capture:
</Warning>

* Only works with `captureMethod: 'render'` (explicit render capture)
* The target element must exist in the DOM when capture starts
* If the target element is not found, the system falls back to the default scope behavior
* Enable `debugLogs: true` to see which element is being captured in the console

## Optimization Guide

### Performance Tuning

<AccordionGroup>
  <Accordion title="Optimize for slower devices">
    **When to use**: Targeting older devices or complex UIs

    **Optimizations**:

    1. Increase `checkInterval` to 5000ms or higher
    2. Switch to `method: 'video'` for native browser capture
    3. Reduce `maxWidth` and `maxHeight` to match your UI needs
    4. Lower `jpegQuality` to 0.3 for smaller files
    5. Set `domChangeDetection.observerConfig.subtree: false`
  </Accordion>

  <Accordion title="Optimize capture responsiveness">
    **When to use**: Need faster updates for dynamic content

    **Optimizations**:

    1. Reduce `domChangeDetection.debounceMs` to 200ms
    2. Enable more mutation observer options
    3. Decrease `checkInterval` for more frequent captures
  </Accordion>

  <Accordion title="Optimize bandwidth usage">
    **When to use**: Limited bandwidth or high traffic applications

    **Optimizations**:

    1. Enable `useHashing: true` for smart deduplication
    2. Increase `minInterval` to reduce capture frequency
    3. Use restrictive `domChangeDetection.observerConfig`
    4. Lower `jpegQuality` to reduce file sizes
  </Accordion>
</AccordionGroup>

### Performance Monitoring

```tsx theme={null}
// Monitor capture performance in browser DevTools
// Use Performance tab to analyze capture timing

// Enable debug logs to track performance metrics
const config = {
  screenCapture: {
    debugLogs: true
  }
};

// Console output shows capture timing:
// ✅ [EXPLICIT-RENDER] Capture in 45.2ms  <- Good performance
// 📊 [EXPLICIT-RENDER] Capture in 150.3ms <- Consider optimization
```

## Best Practices

<Check>
  **Choose the right method** - Use `video` for simplicity, `render` for DOM-specific features
</Check>

<Check>
  **Profile your specific use case** - Use Chrome DevTools to understand your app's needs
</Check>

<Check>
  **Test across devices** - Validate performance on your target device range
</Check>

<Check>
  **Monitor performance metrics** - Enable `debugLogs` to track capture timing
</Check>

<Check>
  **Enable smart optimizations** - Use `useHashing: true` for automatic deduplication
</Check>

<Check>
  **Tune DOM observation** - Configure `observerConfig` based on your UI update patterns
</Check>

## Summary

<Note>
  The Screen Capture system provides a flexible, configurable approach to screen capture that can be tuned for your specific performance and quality requirements. Choose from multiple presets or create custom configurations to match your application's needs.
</Note>

Key features:

* **Flexible intervals** from 1-10+ seconds based on your needs
* **High-quality captures** using modern DOM-to-image conversion
* **Reliable fallback mechanism** ensures continuous operation
* **Targeted capture support** for focused UI regions
* **Performance tuning options** for various device capabilities
* **Multiple configuration presets** for common use cases