Skip to main content
Visual highlighting system that automatically detects and highlights interactive elements while the AI provides guidance
The highlighting system is a sophisticated feature that enables the AI agent to visually highlight interactive elements on web pages while providing guidance. It operates through a multi-layered architecture that seamlessly integrates visual guidance with AI conversation.

​
Overview

The highlighting system provides:
  • Automatic Element Detection: Continuously scans the DOM for interactive elements
  • Proactive Highlighting: Agent highlights elements automatically when providing guidance
  • Context-Aware Injection: Injects element information into the AI’s context at strategic moments
  • Performance Optimized: Uses caching and debouncing to minimize performance impact
  • SPA Support: Handles single-page application navigation seamlessly
Highlighting is disabled by default (enableHighlighting: false) and must be explicitly enabled for performance reasons.

​
Quick Start

​
Enable Highlighting

import { SammyAgentProvider } from '@sammyjs/sammy-three';

function App() {
  return (
    <SammyAgentProvider
      enableHighlighting={true}  // Enable highlighting
      config={{
        // Your other configuration
      }}
    >
      {/* Your app content */}
    </SammyAgentProvider>
  );
}

​
How It Works

Once enabled, the system automatically:
  1. Detects interactive elements on page load and navigation
  2. Injects element context into the AI’s knowledge
  3. Highlights elements when the AI references them in responses
  4. Updates dynamically as the page changes

​
Interactive Element Detection

​
What Gets Detected

The system identifies elements as interactive if they match any of these criteria:
  • HTML Tags
  • ARIA Roles
  • Event Handlers
  • Accessibility
<!-- Interactive HTML elements -->
<button>Submit</button>
<a href="/page">Link</a>
<input type="text" placeholder="Enter text">
<select>
  <option>Choose option</option>
</select>
<textarea placeholder="Enter message"></textarea>

​
Element Descriptions

The system extracts descriptions in this priority order:
  1. Text content (first 50 characters)
  2. aria-label attribute
  3. placeholder attribute (for inputs)
  4. title attribute
  5. Fallback: “No text”

​
Visual Highlighting

​
Highlight Behavior

When the AI highlights an element, it:
  • Applies visual styling: Orange border with glow effect
  • Scrolls element into view: Ensures visibility
  • Auto-removes after 10 seconds: Prevents visual clutter
  • Removes on click: Interactive cleanup
.sammy-element-highlight {
  border: 3px solid #ff6b35 !important;
  box-shadow: 0 0 10px rgba(255, 107, 53, 0.5) !important;
  border-radius: 4px !important;
  position: relative !important;
  z-index: 9999 !important;
}

​
AI Integration

​
Context Injection Format

Interactive elements are automatically injected into the AI’s context in this format: 
<interactive-elements>
1: Submit Application
2: Dashboard  
3: People
4: Add Employee
5: Settings
</interactive-elements>

​
Agent Behavior

The AI is automatically instructed to:
  • Never announce highlighting (“Let me highlight…”)
  • Highlight proactively when mentioning UI elements
  • Match descriptions to the injected element list
  • Continue naturally after highlighting
The AI will automatically highlight relevant elements as it provides guidance, creating a seamless user experience without explicit highlighting requests.

​
Configuration

​
Provider Configuration

<SammyAgentProvider
  enableHighlighting={true}
  config={{
    highlighting: {
      enabled: true,           // Master switch
      debounceMs: 1000,       // Injection debounce (default: 1000ms)
      refreshInterval: 6000,   // Refresh interval (default: 6000ms)
    }
  }}
>

​
Configuration Options

​
enabled
boolean
default:false
Master switch to enable/disable the highlighting system
​
debounceMs
number
default:1000
Debounce time in milliseconds for element detection to prevent excessive DOM scanning
​
refreshInterval
number
default:6000
Interval in milliseconds for periodic element refresh to catch dynamic content changes

​
Detection Timing

​
When Elements Are Detected

Initial Load

Immediate detection when WebSocket connection opens
  • No debounce delay
  • Ensures elements are available for first AI response

Follow-up Detection

1 second after initial to catch late-loading elements
  • Captures dynamically loaded content
  • Handles async component mounting

Periodic Refresh

Every 6 seconds (configurable) for ongoing changes
  • Balances freshness with performance
  • Catches new interactive elements

User Interactions

500ms after user clicks to capture DOM changes
  • Allows DOM to settle after interaction
  • Detects newly appeared elements

Navigation Changes

Immediate on URL change for SPA navigation
  • Clears cache for new page
  • Fresh detection for new content

​
Performance Considerations

​
Optimization Features

  • Caching Strategy
  • Lazy Initialization
  • Debouncing
// Elements are cached by page
const cacheKey = document.body; // Root element as key

// Cache invalidation triggers:
- URL changes
- Manual cache clear
- Navigation events

// Typical cache hit rate: ~90%

​
Performance Metrics

  • Initial Detection: ~10-50ms for typical page
  • Cached Retrieval: < 1ms
  • Context Injection: ~5-10ms
  • Visual Highlighting: < 5ms
  • Memory Usage: ~50-200KB total

​
Advanced Usage

​
Shadow DOM Support

The system automatically traverses Shadow DOM boundaries: 
// Automatic Shadow DOM detection
function traverse(element) {
  // Handle Shadow DOM
  if (element.shadowRoot) {
    Array.from(element.shadowRoot.children).forEach(traverse);
  }
  
  // Continue with regular children
  Array.from(element.children).forEach(traverse);
}

​
Custom Element Integration

Works seamlessly with custom elements and web components: 
<!-- Custom elements are automatically detected -->
<my-custom-button onclick="handleClick()">
  Custom Button
</my-custom-button>

<web-component role="button" aria-label="Action">
  Web Component Button  
</web-component>

​
Troubleshooting

​
Common Issues

Elements Not Being DetectedCheck these common causes:
  • Is highlighting enabled in configuration?
  • Are elements visible on screen?
  • Do elements meet interactive criteria?
  • Check browser console for errors
Highlighting Not WorkingVerify these conditions:
  • Is the highlight tool registered with the AI?
  • Is the element index valid and current?
  • Is the element still present in the DOM?
  • Check for CSS conflicts with highlight styles
Performance IssuesTry these optimizations:
  • Increase debounceMs for stable pages
  • Reduce refreshInterval if not needed
  • Check for DOM mutation loops
  • Monitor console logs for excessive scanning

​
Debug Logging

Enable verbose logging to troubleshoot issues: 
// Look for these log prefixes in browser console:
'🎯 [SammyAgentCore]'           // Initialization
'🚀 [InteractiveElementsManager]' // Manager operations  
'🔍 [DOMAnalyzer]'               // Element detection
'💉 [CONTEXT-INJECTOR]'          // Context injection
'🖱️ [INTERACTIVE-ELEMENTS]'      // Click handling

​
Best Practices

​
When to Enable Highlighting

Enable for guided experiences where users need visual assistance navigating interfaces
Enable for complex applications with many interactive elements that benefit from AI guidance
Enable for onboarding flows where highlighting enhances user understanding

​
Performance Best Practices

  1. Enable only when needed - Highlighting has computational overhead
  2. Configure appropriately - Adjust debounce and refresh intervals based on your application
  3. Monitor performance - Watch console logs for detection timing
  4. Test thoroughly - Verify highlighting works across different page states

​
Integration Tips

  • Test with dynamic content - Ensure highlighting works with async-loaded elements
  • Verify SPA compatibility - Test navigation between different routes/pages
  • Check mobile responsiveness - Ensure highlights are visible on mobile devices
  • Validate accessibility - Confirm highlighting doesn’t interfere with screen readers

​
Summary

The highlighting system transforms static AI conversations into interactive, visually-guided experiences. By automatically detecting interactive elements and highlighting them contextually, it creates intuitive user guidance without requiring explicit user requests. Key benefits:
  • Seamless integration with existing applications
  • Performance optimized with extensive caching
  • Flexible configuration for different use cases
  • Robust architecture that handles edge cases gracefully
  • Enhanced user experience through visual guidance
Start with the default configuration and adjust debounceMs and refreshInterval based on your application’s specific needs and performance requirements.
⌘I