> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sammylabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Highlighting System

> Enable AI-powered visual highlighting of interactive elements on web pages for enhanced user guidance

> Visual highlighting system that automatically detects and highlights interactive elements while the AI provides guidance

The highlighting system is a sophisticated feature that enables the AI agent to visually highlight interactive elements on web pages while providing guidance. It operates through a multi-layered architecture that seamlessly integrates visual guidance with AI conversation.

## Overview

The highlighting system provides:

* **Automatic Element Detection**: Continuously scans the DOM for interactive elements
* **Proactive Highlighting**: Agent highlights elements automatically when providing guidance
* **Context-Aware Injection**: Injects element information into the AI's context at strategic moments
* **Performance Optimized**: Uses caching and debouncing to minimize performance impact
* **SPA Support**: Handles single-page application navigation seamlessly

<Note>
  Highlighting is **disabled by default** (`enableHighlighting: false`) and must be explicitly enabled for performance reasons.
</Note>

## Quick Start

### Enable Highlighting

```tsx theme={null}
import { SammyAgentProvider } from '@sammyjs/sammy-three';

function App() {
  return (
    <SammyAgentProvider
      enableHighlighting={true}  // Enable highlighting
      config={{
        // Your other configuration
      }}
    >
      {/* Your app content */}
    </SammyAgentProvider>
  );
}
```

### How It Works

Once enabled, the system automatically:

1. **Detects interactive elements** on page load and navigation
2. **Injects element context** into the AI's knowledge
3. **Highlights elements** when the AI references them in responses
4. **Updates dynamically** as the page changes

## Interactive Element Detection

### What Gets Detected

The system identifies elements as interactive if they match **any** of these criteria:

<Tabs>
  <Tab title="HTML Tags">
    ```html theme={null}
    <!-- Interactive HTML elements -->
    <button>Submit</button>
    <a href="/page">Link</a>
    <input type="text" placeholder="Enter text">
    <select>
      <option>Choose option</option>
    </select>
    <textarea placeholder="Enter message"></textarea>
    ```
  </Tab>

  <Tab title="ARIA Roles">
    ```html theme={null}
    <!-- Elements with interactive ARIA roles -->
    <div role="button">Custom Button</div>
    <div role="link">Custom Link</div>
    <div role="checkbox">Custom Checkbox</div>
    <div role="menuitem">Menu Item</div>
    ```
  </Tab>

  <Tab title="Event Handlers">
    ```html theme={null}
    <!-- Elements with click handlers -->
    <div onclick="handleClick()">Clickable Div</div>
    <div ng-click="action()">Angular Click</div>
    <div @click="method">Vue Click</div>
    ```
  </Tab>

  <Tab title="Accessibility">
    ```html theme={null}
    <!-- Elements with interactive ARIA properties -->
    <div aria-expanded="false">Expandable</div>
    <div aria-pressed="false">Toggle Button</div>
    <div tabindex="0">Focusable Element</div>
    ```
  </Tab>
</Tabs>

### Element Descriptions

The system extracts descriptions in this priority order:

1. **Text content** (first 50 characters)
2. **`aria-label`** attribute
3. **`placeholder`** attribute (for inputs)
4. **`title`** attribute
5. **Fallback**: "No text"

## Visual Highlighting

### Highlight Behavior

When the AI highlights an element, it:

* **Applies visual styling**: Orange border with glow effect
* **Scrolls element into view**: Ensures visibility
* **Auto-removes after 10 seconds**: Prevents visual clutter
* **Removes on click**: Interactive cleanup

<CodeGroup>
  ```css Visual Styling theme={null}
  .sammy-element-highlight {
    border: 3px solid #ff6b35 !important;
    box-shadow: 0 0 10px rgba(255, 107, 53, 0.5) !important;
    border-radius: 4px !important;
    position: relative !important;
    z-index: 9999 !important;
  }
  ```

  ```javascript Programmatic Access theme={null}
  // Access the highlighting service (if needed)
  const highlightingService = sammyAgent.getHighlightingService();

  // Highlight element by index
  await highlightingService.highlightElement(elementIndex);

  // Clear all highlights
  highlightingService.clearHighlights();
  ```
</CodeGroup>

## AI Integration

### Context Injection Format

Interactive elements are automatically injected into the AI's context in this format:

```xml theme={null}
<interactive-elements>
1: Submit Application
2: Dashboard  
3: People
4: Add Employee
5: Settings
</interactive-elements>
```

### Agent Behavior

The AI is automatically instructed to:

* **Never announce** highlighting ("Let me highlight...")
* **Highlight proactively** when mentioning UI elements
* **Match descriptions** to the injected element list
* **Continue naturally** after highlighting

<Tip>
  The AI will automatically highlight relevant elements as it provides guidance, creating a seamless user experience without explicit highlighting requests.
</Tip>

## Configuration

### Provider Configuration

```tsx theme={null}
<SammyAgentProvider
  enableHighlighting={true}
  config={{
    highlighting: {
      enabled: true,           // Master switch
      debounceMs: 1000,       // Injection debounce (default: 1000ms)
      refreshInterval: 6000,   // Refresh interval (default: 6000ms)
    }
  }}
>
```

### Configuration Options

<ResponseField name="enabled" type="boolean" default={false}>
  Master switch to enable/disable the highlighting system
</ResponseField>

<ResponseField name="debounceMs" type="number" default={1000}>
  Debounce time in milliseconds for element detection to prevent excessive DOM scanning
</ResponseField>

<ResponseField name="refreshInterval" type="number" default={6000}>
  Interval in milliseconds for periodic element refresh to catch dynamic content changes
</ResponseField>

## Detection Timing

### When Elements Are Detected

<Steps>
  <Step title="Initial Load" icon="bolt">
    **Immediate detection** when WebSocket connection opens

    * No debounce delay
    * Ensures elements are available for first AI response
  </Step>

  <Step title="Follow-up Detection" icon="clock">
    **1 second after initial** to catch late-loading elements

    * Captures dynamically loaded content
    * Handles async component mounting
  </Step>

  <Step title="Periodic Refresh" icon="refresh">
    **Every 6 seconds** (configurable) for ongoing changes

    * Balances freshness with performance
    * Catches new interactive elements
  </Step>

  <Step title="User Interactions" icon="mouse-pointer">
    **500ms after user clicks** to capture DOM changes

    * Allows DOM to settle after interaction
    * Detects newly appeared elements
  </Step>

  <Step title="Navigation Changes" icon="route">
    **Immediate on URL change** for SPA navigation

    * Clears cache for new page
    * Fresh detection for new content
  </Step>
</Steps>

## Performance Considerations

### Optimization Features

<Tabs>
  <Tab title="Caching Strategy" icon="database">
    ```javascript theme={null}
    // Elements are cached by page
    const cacheKey = document.body; // Root element as key

    // Cache invalidation triggers:
    - URL changes
    - Manual cache clear
    - Navigation events

    // Typical cache hit rate: ~90%
    ```
  </Tab>

  <Tab title="Lazy Initialization" icon="zap">
    ```javascript theme={null}
    // Services only created when first needed
    function getServices() {
      if (!domAnalyzer) {
        domAnalyzer = new DOMAnalyzerImpl();
        elementCache = new ElementCacheImpl();
        highlightingService = new HighlightingServiceImpl();
      }
      return { domAnalyzer, elementCache, highlightingService };
    }
    ```
  </Tab>

  <Tab title="Debouncing" icon="timer">
    ```javascript theme={null}
    // Prevents excessive DOM scanning
    const config = {
      debounceMs: 1000,        // Injection debounce
      refreshInterval: 6000,   // Periodic refresh
    };

    // Skip if too recent
    if (now - lastInjectionTime < debounceMs) {
      return; // Skip this detection cycle
    }
    ```
  </Tab>
</Tabs>

### Performance Metrics

* **Initial Detection**: \~10-50ms for typical page
* **Cached Retrieval**: \< 1ms
* **Context Injection**: \~5-10ms
* **Visual Highlighting**: \< 5ms
* **Memory Usage**: \~50-200KB total

## Advanced Usage

### Shadow DOM Support

The system automatically traverses Shadow DOM boundaries:

```javascript theme={null}
// Automatic Shadow DOM detection
function traverse(element) {
  // Handle Shadow DOM
  if (element.shadowRoot) {
    Array.from(element.shadowRoot.children).forEach(traverse);
  }
  
  // Continue with regular children
  Array.from(element.children).forEach(traverse);
}
```

### Custom Element Integration

Works seamlessly with custom elements and web components:

```html theme={null}
<!-- Custom elements are automatically detected -->
<my-custom-button onclick="handleClick()">
  Custom Button
</my-custom-button>

<web-component role="button" aria-label="Action">
  Web Component Button  
</web-component>
```

## Troubleshooting

### Common Issues

<Warning>
  **Elements Not Being Detected**

  Check these common causes:

  * Is highlighting enabled in configuration?
  * Are elements visible on screen?
  * Do elements meet interactive criteria?
  * Check browser console for errors
</Warning>

<Warning>
  **Highlighting Not Working**

  Verify these conditions:

  * Is the highlight tool registered with the AI?
  * Is the element index valid and current?
  * Is the element still present in the DOM?
  * Check for CSS conflicts with highlight styles
</Warning>

<Warning>
  **Performance Issues**

  Try these optimizations:

  * Increase `debounceMs` for stable pages
  * Reduce `refreshInterval` if not needed
  * Check for DOM mutation loops
  * Monitor console logs for excessive scanning
</Warning>

### Debug Logging

Enable verbose logging to troubleshoot issues:

```javascript theme={null}
// Look for these log prefixes in browser console:
'🎯 [SammyAgentCore]'           // Initialization
'🚀 [InteractiveElementsManager]' // Manager operations  
'🔍 [DOMAnalyzer]'               // Element detection
'💉 [CONTEXT-INJECTOR]'          // Context injection
'🖱️ [INTERACTIVE-ELEMENTS]'      // Click handling
```

## Best Practices

### When to Enable Highlighting

<Check>
  **Enable for guided experiences** where users need visual assistance navigating interfaces
</Check>

<Check>
  **Enable for complex applications** with many interactive elements that benefit from AI guidance
</Check>

<Check>
  **Enable for onboarding flows** where highlighting enhances user understanding
</Check>

### Performance Best Practices

1. **Enable only when needed** - Highlighting has computational overhead
2. **Configure appropriately** - Adjust debounce and refresh intervals based on your application
3. **Monitor performance** - Watch console logs for detection timing
4. **Test thoroughly** - Verify highlighting works across different page states

### Integration Tips

* **Test with dynamic content** - Ensure highlighting works with async-loaded elements
* **Verify SPA compatibility** - Test navigation between different routes/pages
* **Check mobile responsiveness** - Ensure highlights are visible on mobile devices
* **Validate accessibility** - Confirm highlighting doesn't interfere with screen readers

## Summary

The highlighting system transforms static AI conversations into interactive, visually-guided experiences. By automatically detecting interactive elements and highlighting them contextually, it creates intuitive user guidance without requiring explicit user requests.

Key benefits:

* **Seamless integration** with existing applications
* **Performance optimized** with extensive caching
* **Flexible configuration** for different use cases
* **Robust architecture** that handles edge cases gracefully
* **Enhanced user experience** through visual guidance

<Tip>
  Start with the default configuration and adjust `debounceMs` and `refreshInterval` based on your application's specific needs and performance requirements.
</Tip>