Tools System - SAMMY Labs Docs

The SAMMY Three package implements a sophisticated tool system that allows the Live API agent to perform actions beyond conversation. Tools are type-safe, event-driven, and seamlessly integrated with Google’s Gemini Live API.

Architecture Overview

System Components

The tools system consists of several integrated components:

Core Components

ToolManager

Central registry for managing tool registration, execution, and events

Tool Definitions

Type-safe tool declarations with handlers and metadata

Event System

Type-safe event emission and subscription for tool interactions

Live API Integration

Seamless integration with Google’s Gemini Live API

Tool Definition Structure

Every tool in the system follows a consistent structure defined by the ToolDefinition interface.

interface ToolDefinition<TEventMap> {
  handler: (fc: FunctionCall, context: ToolContext<TEventMap>) => Promise<FunctionResponse>;
  category: ToolCategory;
  declaration: TypedFunctionDeclaration;
}

Components Breakdown

Declaration
Handler Function
Tool Context

The declaration defines the tool’s schema for the Live API:

{
  name: ToolName.EXAMPLE_TOOL,
  description: "Clear description of what the tool does",
  behavior: Behavior.NON_BLOCKING, // or Behavior.BLOCKING
  parameters: {
    type: Type.OBJECT,
    properties: {
      paramName: {
        type: Type.STRING,
        description: "Parameter description"
      }
    },
    required: ['paramName']
  }
}

The handler processes the actual tool execution:

handler: async (fc: FunctionCall, context: ToolContext) => {
  // Parse arguments
  const args = typeof fc.args === 'string' 
    ? JSON.parse(fc.args) 
    : fc.args;
  
  // Perform tool logic
  const result = await performToolAction(args);
  
  // Emit events if needed
  context.emit('customEvent', result);
  
  // Return response
  return {
    id: fc.id,
    name: fc.name,
    response: {
      output: result,
      scheduling: FunctionResponseScheduling.WHEN_IDLE
    }
  };
}

The context provides access to services, state, and event emission:

interface ToolContext<TEventMap> {
  emit: (event: keyof TEventMap, ...args: any[]) => void;
  getState: () => any; // Access to agent state
  services: SammyAgentServices; // Access to service layer
}

Tool Lifecycle

Registration Phase

Tool Registration

Tools are registered during agent initialization:

// In SammyAgentCore constructor
this.toolManager = new ToolManager(
  this.services, 
  this.tools
);

Automatic Setup

The ToolManager automatically:

Registers default tools from DefaultToolDefinitions
Registers custom tools passed via constructor
Creates type-safe event emitters

Configuration Phase

During agent startup, tools are integrated with the Live API:

// In agent start() method
const agentConnectConfig: LiveConnectConfig = getAgentConfig(
  this.toolManager.getAllDeclarations(), // Tools passed here
  systemPrompt
);

// The getAgentConfig function structures tools for the Live API:
tools: [
  {
    functionDeclarations, // Array of tool declarations
  },
],

Execution Phase

Message Reception

Live API sends tool call message

Event Emission

GenAI client emits ‘toolcall’ event

Handler Routing

ToolManager routes to appropriate handler

Execution

Tool handler executes with context

Response

Function response sent back to Live API

// In SammyAgentCore.handleToolCall()
private async handleToolCall(toolCall: LiveServerToolCall): Promise<void> {
  console.log('[Agent] Tool call:', toolCall);
  
  // Notify callback
  this.callbacks.onToolCall?.(toolCall);
  
  // Handle tool calls
  const functionResponses = await this.toolManager.handleToolCall(toolCall);
  
  // Send responses back
  if (functionResponses && functionResponses.length > 0) {
    this.client.sendToolResponse({ functionResponses });
  }
}

Built-in Tools

SAMMY Three includes several pre-configured tools that handle common scenarios:

End Session Tool
Get Context Tool
Escalate Tool

Session Management

Purpose: Manages agent session termination gracefullyCategory: ActionBehavior: Non-blocking

// Usage: Agent calls when user wants to end session
{
  name: "endSession",
  parameters: {
    reason: "user requested",
    confirmed: true
  }
}

Automatic Triggers:

User says goodbye or wants to end chat
Task has been completed successfully
User explicitly requests to stop

Handler Logic:

Validates confirmation parameter
Emits session end event if confirmed
Saves conversation state
Triggers cleanup processes

Context Retrieval

Purpose: Retrieves current page and session contextCategory: ContextBehavior: Non-blocking

// Usage: Agent calls to understand current environment
{
  name: "getCurrentContext",
  parameters: {
    reason: "Getting context for better assistance"
  }
}

Returns:

Current page URL and title
User preferences and settings
Session metadata
Custom context data
Timestamp information

Use Cases:

Understanding user’s current location in app
Providing contextual help
Personalizing responses

Human Escalation

Purpose: Escalates to human support when neededCategory: SystemBehavior: Blocking

// Automatically triggered by AI when needed
{
  name: "escalate",
  parameters: {
    reason: "Unable to resolve user issue",
    category: "technical_support",
    urgency: "medium"
  }
}

The escalate tool is called automatically by the AI when it cannot help. You don’t need to manually trigger it.

Automatic Triggers:

AI cannot understand the request
User explicitly asks for human help
Sensitive topics are detected
Complex issues beyond AI scope

Process:

Marks conversation for review
Notifies support team
Provides escalation reason
Maintains full conversation context
Optionally transfers to live chat

Creating Custom Tools

Step 1: Define Tool Types

Add your tool to the enum and event map:

// In types.ts
export enum ToolName {
  CUSTOM_TOOL = 'customTool',
}

export interface CustomToolEventMap extends BaseToolEventMap {
  [ToolName.CUSTOM_TOOL]: (data: any) => void;
}

Step 2: Create Tool Definition

import { Behavior, Type } from '@google/genai';
import { ToolDefinition, ToolCategories } from '../types';

export const customTool: ToolDefinition<CustomToolEventMap> = {
  declaration: {
    name: ToolName.CUSTOM_TOOL,
    description: "Description of your custom tool",
    behavior: Behavior.NON_BLOCKING,
    parameters: {
      type: Type.OBJECT,
      properties: {
        input: {
          type: Type.STRING,
          description: "Input parameter"
        }
      },
      required: ['input']
    }
  },
  category: ToolCategories.ACTION,
  handler: async (fc, context) => {
    const args = typeof fc.args === 'string' 
      ? JSON.parse(fc.args) 
      : fc.args;
    
    // Your custom logic here
    const result = await processCustomLogic(args.input);
    
    // Emit custom event
    context.emit(ToolName.CUSTOM_TOOL, result);
    
    return {
      id: fc.id,
      name: fc.name,
      response: {
        output: result,
        scheduling: FunctionResponseScheduling.WHEN_IDLE
      }
    };
  }
};

// Pass your tool when creating the agent
const customTools = [customTool];

const agentCore = new SammyAgentCore({
  services,
  config,
  tools: customTools,
  // ... other options
});

Tool Execution Flow

Parallel Processing

The ToolManager processes multiple function calls in parallel for optimal performance.

async handleToolCall(toolCall: LiveServerToolCall): Promise<FunctionResponse[]> {
  const responses: FunctionResponse[] = [];
  
  if (!toolCall.functionCalls) return responses;
  
  // Process all function calls in parallel
  const functionResponses = await Promise.all(
    toolCall.functionCalls.map(async (fc) => {
      return await this.handleFunctionCall(fc);
    })
  );
  
  responses.push(...functionResponses);
  return responses;
}

Error Handling

Robust error handling is implemented at multiple levels:

Missing Tool Name

Returns structured error response

Missing Handler

Returns “tool not found” error

Handler Errors

Catches and formats exceptions

async handleFunctionCall(fc: FunctionCall, agentState?: any): Promise<FunctionResponse> {
  if (!fc.name) return missingToolNameError(fc);
  
  const tool = this.tools.get(fc.name);
  if (!tool) return missingToolHandlerError(fc);
  
  try {
    return await tool.handler(fc, context);
  } catch (error) {
    return toolError(fc, error);
  }
}

Event System

Type-Safe Events

Compile-Time Safety

The event system provides compile-time type safety:

// Tool handler can emit events
context.emit(ToolName.END_SESSION); // ✅ Type-safe
context.emit('invalidEvent'); // ❌ Compile error

// External listeners
toolManager.on(ToolName.END_SESSION, () => {
  console.log('Session ending...');
});

Event Flow

Error Response Structure

All errors return consistent FunctionResponse objects:

{
  id: fc.id,
  name: fc.name,
  response: {
    output: {
      error: "Error message",
      success: false
    },
    scheduling: FunctionResponseScheduling.SILENT
  }
}

Error Types

Missing Tool Name

Tool call without name

Missing Handler

Tool not registered

Handler Exceptions

Runtime errors in tool logic

Errors use SILENT scheduling to avoid interrupting conversation flow.

Best Practices

Tool Design

Single Responsibility

Each tool should have one clear purpose

Clear Descriptions

Provide comprehensive descriptions for the LLM

Parameter Validation

Always validate input parameters

Error Handling

Implement robust error handling

Async Operations

Use Behavior.NON_BLOCKING for async operations
Handle timing with appropriate FunctionResponseScheduling
Consider user experience when choosing scheduling

Event Usage

Emit events for significant state changes
Use type-safe event definitions
Document event contracts

Performance

Process multiple calls in parallel when possible
Avoid blocking operations in tool handlers
Use appropriate scheduling for response timing

Advanced Features

Asynchronous Function Calling

Scheduling Options
Custom Event Maps

For non-blocking operations, tools can use different scheduling options:

// Interrupt current conversation
scheduling: FunctionResponseScheduling.INTERRUPT

// Wait for natural pause
scheduling: FunctionResponseScheduling.WHEN_IDLE

// Silent execution
scheduling: FunctionResponseScheduling.SILENT

Extend the base event map for custom tool events:

interface CustomEventMap extends BaseToolEventMap {
  'custom:action': (data: ActionData) => void;
  'custom:error': (error: Error) => void;
}

Service Integration

Tools have full access to the service layer:

handler: async (fc, context) => {
  // Access to all services
  const { sammyApi, coreServices, memoryServices } = context.services;
  
  // Perform service operations
  const result = await sammyApi.processRequest(data);
  
  return response;
}

State Access

Tools can access current agent state:

handler: async (fc, context) => {
  const currentState = context.getState();
  
  // Use state in tool logic
  if (currentState.userVolume > 0.8) {
    // Handle high volume scenario
  }
  
  return response;
}

Integration with Live API

The tools system integrates seamlessly with Live API capabilities:

Function Calling

Direct integration with Gemini’s function calling

Code Execution

Can be combined with code execution tools

Google Search

Compatible with search grounding

Multi-modal

Supports audio, text, and visual contexts

Summary

The comprehensive tool system enables powerful agent capabilities while maintaining:

Type Safety: Full TypeScript support throughout
Performance: Parallel processing and optimal scheduling
Developer Experience: Clean API and clear patterns
Extensibility: Easy to add custom tools without breaking existing ones

​Architecture Overview

System Components

​Core Components

ToolManager

Tool Definitions

Event System

Live API Integration

​Tool Definition Structure

​Components Breakdown

​Tool Lifecycle

​Registration Phase

​Configuration Phase

​Execution Phase

​Built-in Tools

Session Management

Context Retrieval

Human Escalation

​Creating Custom Tools

​Step 1: Define Tool Types

​Step 2: Create Tool Definition

​Tool Execution Flow

​Parallel Processing

​Error Handling

​Event System

​Type-Safe Events

Compile-Time Safety

​Event Flow

​Error Response Structure

​Error Types

Missing Tool Name

Missing Handler

Handler Exceptions

​Best Practices

​Tool Design

​Async Operations

​Event Usage

​Performance

​Advanced Features

​Asynchronous Function Calling

​Service Integration

​State Access

​Integration with Live API

Function Calling

Code Execution

Google Search

Multi-modal

​Summary

Architecture Overview

Core Components

Tool Definition Structure

Components Breakdown

Tool Lifecycle

Registration Phase

Configuration Phase

Execution Phase

Built-in Tools

Creating Custom Tools

Step 1: Define Tool Types

Step 2: Create Tool Definition

Tool Execution Flow

Parallel Processing

Error Handling

Event System

Type-Safe Events

Event Flow

Error Response Structure

Error Types

Best Practices

Tool Design

Async Operations

Event Usage

Performance

Advanced Features

Asynchronous Function Calling

Service Integration

State Access

Integration with Live API

Summary