AgentRace

中文版

eBPF-based observability tool for AI Agents on Linux, providing zero-intrusion monitoring of LLM API calls, token consumption, process behavior, and SSL/TLS traffic. Its eBPF observation approach is inspired by agentsight, the observability component of ANOLISA.

Features

• Zero-Intrusion Monitoring — eBPF kernel probes capture events without modifying agent code or configurations.
• SSL/TLS Traffic Decryption — uprobe-based interception of OpenSSL/GnuTLS library calls to capture plaintext HTTP traffic.
• LLM Token Accounting — Precise token counting with Hugging Face tokenizer support (Qwen series and more).
• AI Agent Auto-Discovery — Scans /proc and monitors execve events to dynamically detect running AI agent processes.
• Streaming Response Support — Parses Server-Sent Events (SSE) for tracking streamed LLM responses.
• Audit Logging — Complete audit trail of LLM calls and process operations with structured records.
• ATIF v1.7 Trajectory Export — Export observed LLM calls, tool usage, and agent interactions to the Agent Trajectory Interchange Format (ATIF v1.7) for replay and analysis.
• GenAI Semantic Events — Builds structured semantic events for LLM calls, tool usage, and agent interactions.
• Build-ID Offset Cache — Caches SSL function offsets for statically-linked agent binaries (e.g. Codex), so probes attach instantly on restart with no first-call loss.

Architecture

AgentRace operates a unified data pipeline:

┌──────────┐    ┌────────┐    ┌────────────┐    ┌──────────┐    ┌───────┐    ┌─────────┐
│  Probes  │───▶│ Parser │───▶│ Aggregator │───▶│ Analyzer │───▶│ GenAI │───▶│ Storage │
└──────────┘    └────────┘    └────────────┘    └──────────┘    └───────┘    └─────────┘
  eBPF events    HTTP/SSE      Req-Resp          Token/Audit     Semantic     SQLite
  (kernel)       extraction    correlation       extraction      events       (local)

Stage	Description
Probes	eBPF programs (sslsniff, tcpsniff, proctrace, procmon) capture kernel events via ring buffer
Parser	Extracts structured HTTP messages, SSE events, and process exec data
Aggregator	Correlates request-response pairs; tracks process lifecycle via LRU cache
Analyzer	Produces audit records, token usage stats, and LLM API messages
GenAI	Transforms results into semantic events (LLM calls, tool use, agent interactions)
Storage	Persists to a local SQLite database

eBPF Probes

Probe	Source	Description
sslsniff	src/bpf/sslsniff.bpf.c	uprobe on SSL_read/SSL_write to capture plaintext from encrypted connections
tcpsniff	src/bpf/tcpsniff.bpf.c	fentry/fexit on TCP functions for connection metadata (IP/port)
proctrace	src/bpf/proctrace.bpf.c	Traces execve syscalls, captures command-line args, builds process tree
procmon	src/bpf/procmon.bpf.c	Lightweight process monitor for creation/exit events (agent discovery)

Project Structure

agentrace/
├── src/
│   ├── bpf/            # eBPF C programs (sslsniff, tcpsniff, proctrace, procmon)
│   ├── probes/         # eBPF probe management and event polling
│   ├── parser/         # HTTP, HTTP/2, SSE, and process event parsers
│   ├── aggregator/     # Request-response correlation and process aggregation
│   ├── analyzer/       # Message routing, token extraction, audit records
│   ├── genai/          # GenAI semantic event builder
│   ├── atif/           # ATIF v1.7 trajectory converter
│   ├── storage/        # SQLite-backed stores (genai, audit, token, http, interruption)
│   ├── discovery/      # AI agent process scanner (/proc + cmdline glob matching)
│   ├── tokenizer/      # HuggingFace tokenizer integration for token counting
│   ├── token_breakdown/# Token analysis
│   ├── health/         # Agent health checker
│   ├── interruption/   # SSE truncation / crash / context-overflow detection
│   ├── server/         # Actix-web HTTP API and auth
│   ├── bin/            # CLI entry points (agentrace and subcommands)
│   ├── utils/          # Decompression, process, and thread helpers
│   ├── unified.rs      # Main pipeline orchestrator
│   ├── config.rs       # Unified configuration management
│   └── event.rs        # Unified event type definitions
├── dashboard/          # React dashboard (embedded into serve at build time)
├── Cargo.toml
├── build.rs            # eBPF skeleton generation for four probes
└── agentrace.spec      # RPM packaging spec

CLI Commands

Commands token, audit, discover, metrics, interruption, and summary require Linux eBPF. trace and serve run the full eBPF pipeline on Linux.

agentrace trace

Start eBPF-based tracing of AI agent activity (probes → parser → aggregator → storage).

# Foreground mode
sudo agentrace trace

# Daemon mode
sudo agentrace trace --daemon

agentrace token

Query token consumption data.

# Today's token usage
agentrace token

# This week, compared to last week
agentrace token --period week --compare

# Detailed breakdown by role and type
agentrace token --detail

# JSON output
agentrace token --json

agentrace audit

Query audit events (LLM calls, process operations).

# Recent audit events
agentrace audit

# Filter by PID and event type
agentrace audit --pid 12345 --type llm

# Summary statistics
agentrace audit --summary

agentrace serve

Start the HTTP API server and serve the embedded Dashboard UI.

# Start with default settings (binds to 127.0.0.1:7396)
agentrace serve

# Bind to all interfaces on a custom port
agentrace serve --host 0.0.0.0 --port 8080

# Point to a specific database file
agentrace serve --db /path/to/genai_events.db

agentrace discover

Discover AI agents running on the system.

# Scan for running agents
agentrace discover

# List all known agent types
agentrace discover --list-known

# Verbose output with executable paths
agentrace discover --verbose

agentrace export atif

Export observed agent activity as ATIF (Agent Trajectory Interchange Format) v1.7 JSON.

# Export a single trace
agentrace export atif --trace <trace-id>

# Export a conversation
agentrace export atif --conversation <conversation-id>

# Export a full session
agentrace export atif --session <session-id>

Dashboard

The Dashboard is a React-based web UI for visualizing conversation history, trace details, and token statistics. It is embedded into the agentrace serve binary at compile time.

Build the Dashboard

# From the repo root: build frontend and embed into frontend-dist/
# (required before cargo build)
make build-frontend

# Then build the Rust binary with the embedded UI
make build

# Or do both in one step
make build-all

Scenario 1 — Collect data and view the Dashboard simultaneously

Run the tracer and the API server in two separate terminals:

# Terminal 1: start eBPF tracing (writes to SQLite)
sudo agentrace trace

# Terminal 2: start the API server (reads from the same SQLite)
agentrace serve

Open http://127.0.0.1:7396 in your browser. The Dashboard auto-refreshes as new data arrives.

Running on a remote server? Bind to all interfaces and access via the server's public IP:

agentrace serve --host 0.0.0.0 --port 7396

Then open http://<server-public-ip>:7396 in your local browser. Make sure port 7396 is allowed in the server's firewall / security group rules.

Scenario 2 — Browse historical data only

No tracing needed. Just start the server pointing at an existing database:

agentrace serve --db /path/to/genai_events.db

Open http://127.0.0.1:7396 to explore recorded conversations and traces.

Dashboard Development

To iterate on the frontend without rebuilding the Rust binary:

cd dashboard
npm install
npm run dev          # starts webpack-dev-server on http://localhost:3004

When finished, run make build-frontend && cargo build --release to embed the updated UI.

Quick Start

Prerequisites

System Packages

Before building, install the required system packages:

Anolis OS / CentOS / RHEL:

sudo yum install -y openssl-devel elfutils-libelf-devel perl-IPC-Cmd libbpf-devel clang llvm bpftool

Ubuntu / Debian:

sudo apt install -y pkg-config libssl-dev libelf-dev libbpf-dev clang llvm linux-tools-common

Package	Required for
openssl-devel	OpenSSL vendored build (used via openssl = { features = ["vendored"] })
elfutils-libelf-devel	libbpf-sys crate (provides gelf.h, libelf.h)
perl-IPC-Cmd	OpenSSL source build (Perl IPC::Cmd module)
libbpf-devel	eBPF program compilation and loading
clang / llvm	eBPF C program compilation to BPF bytecode
bpftool	eBPF skeleton generation

You can verify all dependencies with the included check script:

./scripts/check-deps.sh

Version Requirements

Component	Version
Linux kernel	>= 5.8 (BTF support)
Rust	>= 1.80
clang / llvm	>= 11 (for eBPF compilation)
libbpf	>= 0.8

Install via RPM

sudo yum install agentrace

Installs:

• /usr/local/bin/agentrace — CLI binary
• /usr/lib/systemd/system/agentrace.service — AgentRace system unit

The RPM is a Linux system package. Its unit is installed but not enabled by default.

Build from Source

# From the repo root

# Verify dependencies (optional but recommended)
./scripts/check-deps.sh

# Build frontend and Rust binary with embedded Dashboard UI
make build-all

The binary is output to target/release/agentrace.

cargo build --release only compiles Rust. It does not rebuild the embedded Dashboard UI, so use make build-all for user-facing builds.

Start Tracing

# Requires root for eBPF
sudo agentrace trace

Configuration

AgentRace is configured via agentrace.json (default path /etc/agentrace/config.json; falls back to embedded defaults if absent).

Basic Options

Category	Option	Description
Storage	db_path	SQLite database file path
Storage	data_retention_days	Data retention period
Probes	target_uid	Filter events by UID
Probes	poll_timeout_ms	Ring buffer poll timeout
HTTP	connection_cache_capacity	LRU cache size for connection tracking
Tokenizer	tokenizer_file	Path or URL to HuggingFace tokenizer

Feature Flags (features)

All optional features are enabled by default. Disable them individually via the features block in agentrace.json to reduce memory and I/O overhead:

Feature	JSON Path	Default	Description
Token Stats	features.token_stats	true	Core functionality, not recommended to disable
Local Tokenizer	features.tokenizer.enabled	false	HuggingFace model fallback (50–100 MB per model)
Session Mapping	features.session_mapping.enabled	true	responseId → sessionId correlation (LRU 10,000)
SQLite Storage	features.sqlite_storage.enabled	true	Persist to disk SQLite; disabled uses noop store
Interruption Detection	features.interruption_detection.enabled	true	Dead loop / crash / context overflow detection
Audit	features.audit	true	LLM call audit event persistence
Token Consumption	features.token_consumption	false	Aggregated token consumption records

Runtime Resource Limits (runtime_limits)

Configure buffer caps to prevent unbounded memory growth:

Option	Default	Description
event_channel_capacity	10,000	Bounded channel capacity for probe events
event_channel_policy	"backpressure"	Full-channel policy: backpressure / drop_newest / sample
pending_genai_max_count	1,000	Max pending events awaiting session_id
pending_genai_max_bytes_mb	64	Max bytes for pending events
pid_cache_size	1,024	PID → agent_name LRU cache size
max_connection_body_mb	8	Per-connection HTTP body buffer cap
connection_idle_timeout_secs	60	HTTP connection idle timeout (seconds)
ring_buffer_mb	32	eBPF Ring Buffer size (must be power of 2)

Minimal Memory Configuration

For resource-constrained environments, disable non-essential features and reduce ring buffer:

{
  "features": {
    "token_stats": true,
    "tokenizer": { "enabled": false },
    "session_mapping": { "enabled": false },
    "sqlite_storage": { "enabled": false },
    "interruption_detection": { "enabled": false },
    "audit": false,
    "token_consumption": false
  },
  "runtime_limits": {
    "ring_buffer_mb": 8,
    "event_channel_capacity": 5000,
    "pending_genai_max_count": 500,
    "pending_genai_max_bytes_mb": 32
  }
}

With this config: idle RSS ~24–30 MB, with event traffic ~35–40 MB.

Supported LLM Providers

Token parsing supports multiple LLM API formats:

• OpenAI / OpenAI-compatible APIs
• Anthropic (Claude, including cache token handling)
• Google Gemini
• Qwen (with native chat template support)

Origins

This project is derived from https://github.com/eunomia-bpf/agentrace.git.

License

Apache License 2.0 — see LICENSE for details.