A production-oriented distributed AI agent runtime with semantic model routing, asynchronous task execution, isolated tool execution, and highly available shared state.
Agenyx is designed as a distributed runtime for building and executing AI agents that can reason, call tools, execute long-running tasks, and dynamically route LLM requests to the most appropriate model.
The architecture separates agent reasoning, semantic routing, inference, asynchronous execution, tool isolation, and infrastructure into independently deployable services.
┌──────────────────────┐
│ Client │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Gateway │
│ Nginx │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Agent │
│ FastAPI │
│ Agent Runtime │
└──────────┬───────────┘
│
│ LLM Request
▼
┌──────────────────────┐
│ Semantic Router │
│ │
│ Intent Classification│
│ Model Selection │
│ Session State │
│ Model Affinity │
└──────────┬───────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Model A Model B Model C
│ │ │
└──────────────┼──────────────┘
│
▼
┌──────────────────────┐
│ Inference │
│ OpenAI Compatible │
│ FastAPI │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Ollama │
│ Local Models │
└──────────────────────┘
Asynchronous Execution Pipeline
────────────────────────────────
┌──────────────────────┐
│ Agent │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Orchestrator │
│ Go │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Valkey Streams │
│ Consumer Groups │
└──────────┬───────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Worker 1 Worker 2 Worker 3
│ │ │
└──────────────┼──────────────┘
│
▼
┌──────────────────────┐
│ Task Execution │
└──────────────────────┘
Tool Execution Pipeline
───────────────────────
┌──────────────────────┐
│ Agent │
└──────────┬───────────┘
│
│ Tool Call
▼
┌──────────────────────┐
│ Sandbox │
│ FastAPI │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ gVisor / runsc │
│ Isolated Execution │
└──────────────────────┘
High Availability State
────────────────────────
┌─────────────────────┐
│ Valkey Sentinel │
│ │
│ Sentinel 0 │
│ Sentinel 1 │
│ Sentinel 2 │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Valkey Primary │
│ StatefulSet │
└─────────────────────┘
| Component | Responsibility | Technology |
|---|---|---|
| Gateway | External API entry point and routing | Nginx |
| Agent | Agent reasoning and tool orchestration | Python / FastAPI |
| Semantic Router | Semantic model selection and routing sessions | Go |
| Inference | OpenAI-compatible LLM interface | Python / FastAPI |
| Model Backend | Local LLM execution | Ollama |
| Orchestrator | Asynchronous task coordination | Go |
| Worker | Background task processing | Python |
| Sandbox | Isolated tool/code execution | Python / FastAPI |
| Isolation Runtime | Secure execution boundary | gVisor / runsc |
| Task Queue | Distributed task delivery | Valkey Streams |
| Shared State | Execution and routing state | Valkey |
| HA | Primary discovery and failover | Valkey Sentinel |
| Containerization | Service packaging | Docker |
| Deployment | Production orchestration | Kubernetes |
| Configuration | Kubernetes resource management | Kustomize |
| Observability | Metrics, logs and tracing | Prometheus / Grafana / Loki / OpenTelemetry |
Traditional AI agent implementations often look like:
Request
│
▼
Single Python Process
│
├── LLM
├── Tools
├── State
└── Background Tasks
This works for prototypes but becomes difficult to operate when:
- LLM requests become expensive
- Different requests require different models
- Agent tasks become long-running
- Workers need horizontal scaling
- A worker crashes during execution
- Generated code needs isolation
- Multiple router instances need shared state
- Model routing needs conversation awareness
- The application needs Kubernetes deployment
Agenyx separates these responsibilities:
Gateway
│
▼
Agent
│
├──────────────► Semantic Router ───► Inference
│
├──────────────► Sandbox
│
▼
Orchestrator
│
▼
Valkey Streams
│
▼
Worker Pool
The Agent focuses on the agent execution loop.
Infrastructure concerns such as:
- Model selection
- Asynchronous task processing
- Tool isolation
- Shared state
- Failover
are handled by dedicated components.
Agenyx does not assume that every LLM request should use the same model.
The Semantic Router evaluates the request and selects an appropriate model according to the routing policy.
Agent
│
▼
Semantic Router
│
├──► Fast model
├──► General model
├──► Reasoning model
└──► Specialized model
This makes model selection an explicit part of the runtime rather than hardcoding a single model inside the Agent.
The semantic routing pipeline is one of the core components of Agenyx.
At a high level:
User Request
│
▼
Agent
│
▼
Semantic Router
│
├── Analyze request
│
├── Determine semantic intent
│
├── Evaluate routing policy
│
├── Consider session state
│
├── Consider model affinity
│
└── Select model
│
▼
Inference
│
▼
LLM Model
The router therefore acts as a decision layer between the Agent and the inference infrastructure.
Agenyx supports routing decisions that are aware of the current conversation session.
A routing session can maintain telemetry such as:
CurrentModel
TurnCount
SwitchCount
ModelTurns
LastDecisionName
This allows routing decisions to take previous turns into account.
For example:
Turn 1
│
▼
Router
│
└──► Model A
Turn 2
│
▼
Router
│
└──► Model A
Turn 3
│
▼
Router
│
└──► Model A
Turn 4
│
▼
Router
│
└──► Model B
The router can maintain model affinity while still switching models when the routing policy determines that another model is more appropriate.
Model affinity reduces unnecessary model switching within a session.
For example:
Session
│
├── Request 1 ──► Model A
│
├── Request 2 ──► Model A
│
├── Request 3 ──► Model A
│
└── Request 4 ──► Model B
The router can continue using the current model when appropriate while allowing the semantic routing policy to switch models when the request characteristics change.
This can improve:
- Routing stability
- Latency
- Context continuity
- Model utilization
The Semantic Router is designed to support multiple router instances.
Load Balancer
│
┌────────────┼────────────┐
▼ ▼ ▼
Router 1 Router 2 Router 3
│ │ │
└────────────┼────────────┘
│
▼
Shared Session State
│
▼
Valkey
Router instances do not need to own permanent local session state.
Any router instance can retrieve the shared routing session and continue the decision process.
Concurrent router instances can potentially attempt to update the same routing session.
Agenyx uses versioned session state to prevent stale router decisions from silently overwriting newer state.
Conceptually:
Session:
version = 10
current_model = model-a
turn_count = 5
A router reads the current state and attempts a conditional update:
Read version 10
│
▼
Make routing decision
│
▼
Update only if version == 10
│
├── Success
│ │
│ └── version = 11
│
└── Conflict
│
▼
Re-read state
This provides compare-and-swap style semantics for distributed routing updates.
The Inference service provides a stable OpenAI-compatible interface between the routing layer and model backends.
Current endpoints include:
GET /health
GET /ready
GET /v1/models
POST /v1/chat/completions
The architecture is:
Agent
│
▼
Semantic Router
│
▼
Inference Service
│
▼
Model Backend
│
▼
Ollama
The Agent and Router therefore do not need to directly depend on Ollama's API.
This makes it possible to replace the model backend without redesigning the Agent runtime.
Potential future inference backends include:
- Ollama
- vLLM
- TGI
- Other OpenAI-compatible servers
Local development currently uses Ollama for model inference.
Ollama runs the actual local model while Agenyx provides the surrounding distributed runtime.
Agenyx
│
▼
Inference
│
▼
Ollama
│
▼
Local LLM
This allows Agenyx to be developed and tested locally without requiring a remote commercial LLM API.
The Agent service implements the core agent execution loop.
Its responsibilities include:
- Receiving agent requests
- Building execution context
- Calling the semantic router
- Calling the inference layer
- Processing model responses
- Handling tool calls
- Sending tool requests to the sandbox
- Continuing the agent loop
- Returning the final response
The basic execution loop is:
Request
│
▼
Agent Runtime
│
▼
Semantic Router
│
▼
Inference
│
▼
LLM
│
├──────────────► Final response
│
└──────────────► Tool call
│
▼
Sandbox
│
▼
Tool result
│
▼
LLM
Agenyx isolates tool execution from the main Agent process.
Instead of allowing generated code or potentially untrusted tools to execute directly inside the Agent container:
Agent
│
▼
Sandbox
│
▼
gVisor / runsc
│
▼
Isolated execution
The Sandbox is responsible for executing tools inside an isolated environment.
This creates a security boundary between:
Agent Runtime
│
│ untrusted execution
▼
Sandbox
Agenyx uses gVisor / runsc for additional container-level isolation of sandbox workloads.
The sandbox can be configured with:
- Restricted Linux capabilities
- No privilege escalation
- Restricted networking
- Non-root execution
- Seccomp
- gVisor isolation
The goal is to avoid executing potentially untrusted generated code directly inside the Agent runtime.
Long-running work is separated from synchronous request processing.
The asynchronous pipeline is:
Agent
│
▼
Orchestrator
│
▼
Valkey Stream
│
▼
Worker Pool
│
▼
Task Execution
This allows the Worker layer to scale independently.
For example:
Valkey Stream
│
┌───────────┼───────────┐
▼ ▼ ▼
Worker 1 Worker 2 Worker 3
│ │ │
└───────────┼───────────┘
│
▼
Task Execution
The Orchestrator coordinates asynchronous task execution.
It is responsible for:
- Receiving asynchronous work
- Creating execution records
- Publishing tasks
- Coordinating task lifecycle
- Communicating through Valkey
- Tracking task state
The Orchestrator does not need to execute every long-running task itself.
Instead, it publishes work to the distributed Worker pool.
Workers are long-running task consumers.
Each Worker:
- Connects to Valkey
- Ensures the consumer group exists
- Reads tasks from the stream
- Executes tasks
- Updates execution state
- Acknowledges successfully handled tasks
- Recovers abandoned pending tasks
- Applies retry behavior
- Sends permanently failed tasks to the dead-letter stream
- Handles graceful shutdown
Workers use Valkey Streams consumer groups:
Stream:
agenyx:tasks
Consumer Group:
agenyx-workers
Each Kubernetes worker pod receives its own consumer name.
Valkey Streams maintain pending entries for messages that have been delivered but not acknowledged.
If a Worker crashes:
Worker A
│
▼
Task received
│
X
Worker crashes
│
▼
Pending Entry
│
▼
Worker B
│
▼
Task reclaimed
Agenyx workers periodically reclaim abandoned pending tasks.
The behavior is configurable through:
PENDING_IDLE_MS
PENDING_BATCH_SIZE
This provides crash recovery without requiring a separate distributed recovery system.
Agenyx supports configurable task retries.
Important configuration values include:
MAX_ATTEMPTS
RETRY_BASE_DELAY_SECONDS
RETRY_MAX_DELAY_SECONDS
A task can move through:
pending
│
▼
processing
│
├──────────────► completed
│
├──────────────► retrying
│ │
│ ▼
│ processing
│
└──────────────► failed
│
▼
DLQ
Retry timing is stored as execution state rather than relying only on local Worker memory.
Tasks that cannot be successfully processed after the configured retry limit are moved to a dead-letter stream.
Default stream:
agenyx:tasks:dead-letter
Dead-letter records contain information such as:
execution_id
intent
attempt
original_message_id
error
consumer
This provides a durable location for permanently failed tasks.
Valkey is a central infrastructure component in Agenyx.
It is used for:
- Streams
- Consumer groups
- Pending task tracking
- Execution state
- Shared routing session state
- Model affinity state
- Distributed coordination
The task stream is:
agenyx:tasks
The Worker consumer group is:
agenyx-workers
Agenyx uses Valkey Sentinel for high availability and primary discovery.
The Kubernetes deployment contains:
Valkey
└── StatefulSet
└── Primary
Sentinel
├── Sentinel 0
├── Sentinel 1
└── Sentinel 2
Sentinel monitors the Valkey primary and provides failover coordination.
Applications should discover the current primary through Sentinel rather than relying on a permanently fixed primary address.
Configuration is supplied through environment variables such as:
VALKEY_SENTINEL_HOSTS
VALKEY_MASTER_NAME
VALKEY_PASSWORD
Example:
VALKEY_MASTER_NAME=mymaster
The actual Kubernetes service and Sentinel addresses are supplied through Kubernetes configuration.
Agenyx avoids hardcoding environment-specific infrastructure addresses in application logic.
Configuration is provided through environment variables and Kubernetes ConfigMaps/Secrets.
Examples include:
VALKEY_SENTINEL_HOSTS
VALKEY_MASTER_NAME
VALKEY_PASSWORD
TASK_STREAM
CONSUMER_GROUP
CONSUMER_NAME
AGENT_URL
AGENT_TIMEOUT_SECONDS
MAX_ATTEMPTS
RETRY_BASE_DELAY_SECONDS
RETRY_MAX_DELAY_SECONDS
PENDING_IDLE_MS
PENDING_BATCH_SIZE
Secrets such as Valkey passwords are stored in Kubernetes Secrets rather than ConfigMaps.
Agenyx is designed to run as independently deployable Kubernetes workloads.
Current deployment structure:
deploy/
└── kubernetes/
├── agent/
├── worker/
├── orchestrator/
├── inference/
├── gateway/
├── sandbox/
└── valkey/
The Kubernetes deployment uses Kustomize.
The Valkey deployment includes:
deploy/kubernetes/valkey/
├── kustomization.yaml
├── secret.yaml
├── service.yaml
├── statefulset.yaml
├── networkpolicy.yaml
├── sentinel-configmap.yaml
├── sentinel-statefulset.yaml
└── sentinel-networkpolicy.yaml
The Valkey primary runs as a StatefulSet.
Sentinel runs as a separate StatefulSet with three replicas.
Workers run as Kubernetes Deployment replicas.
Example:
agenyx-worker
├── Worker Pod
├── Worker Pod
└── Worker Pod
Workers are configured to:
- Run as non-root
- Drop Linux capabilities
- Disable privilege escalation
- Use RuntimeDefault seccomp
- Use resource requests and limits
- Gracefully terminate
- Spread replicas across nodes when possible
The Worker receives its Kubernetes pod name as its consumer identity:
CONSUMER_NAME = metadata.name
This ensures that each Worker has a unique consumer name.
Agenyx applies security controls at multiple layers.
Services use:
runAsNonRoot
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
Where appropriate, containers use:
readOnlyRootFilesystem: true
Worker pods use dedicated Kubernetes service accounts.
Where Kubernetes API access is not required:
automountServiceAccountToken: false
This reduces unnecessary access to Kubernetes credentials.
Kubernetes NetworkPolicies are used to control communication between services.
The goal is to avoid exposing every service to every other service.
Each Agenyx service can be packaged as a Docker image.
Example Worker image:
FROM python:3.14-slim
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-m", "app.main"]Images are then deployed to the Kubernetes container runtime.
Install:
- Docker
- Docker Compose
- Python 3.12+
- Go
- Ollama
- kubectl
- A local or remote Kubernetes cluster
Agenyx does not require Kind specifically.
Any Kubernetes cluster capable of running the required workloads can be used.
Build and start the services:
docker compose up -d --buildCheck services:
docker compose psFollow logs:
docker compose logs -fApply the Kubernetes configuration using Kustomize:
kubectl apply -k deploy/kubernetesCheck the namespace:
kubectl get pods -n agenyxCheck services:
kubectl get svc -n agenyxCheck deployments:
kubectl get deployments -n agenyxCheck StatefulSets:
kubectl get statefulsets -n agenyxBuild a service image:
docker build -t agenyx-worker:0.1.0 ./workerFor Kubernetes clusters using the local container runtime, ensure the image is available to the runtime used by the cluster.
Verify the image through the container runtime.
Then restart the deployment:
kubectl -n agenyx rollout restart deployment/agenyx-workerCheck rollout:
kubectl -n agenyx rollout status deployment/agenyx-workerCheck pods:
kubectl -n agenyx get pods -l app.kubernetes.io/name=agenyx-worker -o wideCheck pod details:
kubectl -n agenyx describe pod <pod-name>View logs:
kubectl -n agenyx logs <pod-name>Follow logs:
kubectl -n agenyx logs -f <pod-name>View logs from the previous crashed container:
kubectl -n agenyx logs <pod-name> --previousCheck recent events:
kubectl -n agenyx get events --sort-by=.lastTimestampServices expose health/readiness endpoints where appropriate.
Typical checks include:
/health
/ready
Kubernetes readiness and liveness probes are used to prevent unhealthy instances from receiving traffic.
Agenyx is designed to integrate with a production observability stack.
The planned/used observability components include:
OpenTelemetry
│
├──► Traces
│
▼
Prometheus
│
▼
Grafana
Application Logs
│
▼
Loki
│
▼
Grafana
Important observability signals include:
- Request latency
- Task processing latency
- Task success/failure counts
- Retry counts
- Pending tasks
- Worker health
- Model routing decisions
- Model switching
- Inference latency
- Sandbox execution failures
Agenyx is organized around independent services:
.
├── agent/
│ ├── app/
│ ├── tests/
│ └── Dockerfile
│
├── orchestrator/
│ ├── ...
│ └── Dockerfile
│
├── worker/
│ ├── app/
│ ├── tests/
│ └── Dockerfile
│
├── inference/
│ ├── app/
│ │ ├── backend.py
│ │ ├── config.py
│ │ └── main.py
│ └── Dockerfile
│
├── sandbox/
│ ├── ...
│ └── Dockerfile
│
├── gateway/
│ └── ...
│
├── deploy/
│ └── kubernetes/
│ ├── agent/
│ ├── worker/
│ ├── orchestrator/
│ ├── inference/
│ ├── gateway/
│ ├── sandbox/
│ └── valkey/
│
├── scripts/
│ └── setup-gvisor.sh
│
├── docs/
│ └── ...
│
└── docker-compose.yml
A normal agent request follows this path:
Client
│
▼
Gateway
│
▼
Agent
│
▼
Semantic Router
│
├── Session state
├── Model affinity
├── Semantic decision
└── Model selection
│
▼
Inference
│
▼
Ollama
│
▼
Model Response
│
├──────────────► Final Answer
│
└──────────────► Tool Call
│
▼
Sandbox
│
▼
Tool Execution
│
▼
Tool Result
│
▼
Agent
Long-running work follows a separate pipeline:
Client
│
▼
Gateway
│
▼
Agent / Orchestrator
│
▼
Execution Record
│
▼
Valkey Stream
│
▼
Consumer Group
│
├──────────────┐
▼ ▼
Worker 1 Worker 2
│ │
└───────┬──────┘
│
▼
Task Execution
│
├── Success
│
├── Retry
│
└── Dead Letter
Agenyx is designed around failure as a normal operating condition.
Examples:
Worker A
│
X crash
│
▼
Pending task
│
▼
Worker B
│
▼
Task recovery
Inference failure
│
▼
Agent / Worker
│
▼
Retry policy
│
├── Retry
│
└── Permanent failure
Valkey Primary
│
X
│
▼
Sentinel detects failure
│
▼
Failover
│
▼
New Primary
Router 1
│
X
│
▼
Router 2
│
▼
Shared routing session
│
▼
Continue routing
- Python
- FastAPI
- Go
- Ollama
- OpenAI-compatible inference API
- Semantic model routing
- Session-aware routing
- Valkey
- Valkey Streams
- Consumer Groups
- Valkey Sentinel
- Kubernetes
- gVisor
- runsc
- Kubernetes NetworkPolicies
- Linux capabilities
- Seccomp
- Docker
- Docker Compose
- Kubernetes
- Kustomize
- OpenTelemetry
- Prometheus
- Grafana
- Loki
The architecture is designed to evolve toward a larger distributed AI platform.
Potential future improvements include:
- More advanced semantic routing policies
- Additional model providers
- GPU-backed inference with vLLM
- Dynamic model health scoring
- Routing based on latency and model availability
- More sophisticated model affinity policies
- Multi-tenant isolation
- Authentication and authorization
- API rate limiting
- Distributed tracing across all services
- Advanced task scheduling
- Task prioritization
- DLQ replay tooling
- Persistent database for execution history
- Vector-based retrieval and long-term memory
- Automated CI/CD
- Production model registry
- Kubernetes autoscaling based on workload metrics
- GPU inference cluster
Agenyx intentionally separates infrastructure responsibilities instead of placing everything into one application.
The core architecture is:
┌─────────────────────┐
│ Gateway │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Agent │
└──────┬──────┬───────┘
│ │
│ └───────────────┐
▼ ▼
Semantic Router Sandbox
│
▼
Inference
│
▼
Ollama
Agent
│
▼
Orchestrator
│
▼
Valkey Streams
│
┌───────┼───────┐
▼ ▼ ▼
Worker Worker Worker
The goal is not to add infrastructure simply for the sake of complexity.
Each component exists because it solves a specific distributed-system problem:
| Problem | Agenyx Component |
|---|---|
| External API access | Gateway |
| Agent reasoning | Agent |
| Model selection | Semantic Router |
| LLM execution | Inference |
| Local model backend | Ollama |
| Long-running tasks | Orchestrator |
| Distributed execution | Worker Pool |
| Tool isolation | Sandbox |
| Secure execution | gVisor |
| Task distribution | Valkey Streams |
| Shared routing state | Valkey |
| High availability | Valkey Sentinel |
| Container orchestration | Kubernetes |
| Observability | OpenTelemetry / Prometheus / Grafana / Loki |
Agenyx is under active development.
The core architecture currently focuses on:
- Distributed Agent execution
- Semantic model routing
- Session-aware routing
- Model affinity
- Versioned routing state
- OpenAI-compatible inference
- Ollama integration
- Asynchronous task execution
- Valkey Streams
- Consumer groups
- Pending task recovery
- Retry handling
- Dead-letter processing
- Valkey Sentinel
- gVisor sandbox isolation
- Kubernetes deployment
APIs, internal protocols, and deployment configuration may continue to evolve as the system moves toward production readiness.
Linux developers can configure gVisor using:
./scripts/setup-gvisor.shVerify that runsc is available:
docker info | grep -A10 Runtimesrunsc should appear in the available Docker runtimes.
Then start Agenyx:
docker compose up -d --buildThis project is currently under active development.
Add the appropriate license here when the project is ready for public distribution.