GitHub - NoahMoscovici/bottest: Automated testing platform for AI chatbots. Record conversations with your chatbot and easily replay them to benchmark behavior over time.

Automated testing platform for AI chatbots. Record conversations with your chatbot and easily replay them to benchmark behavior over time.

Notice: This project is no longer actively maintained. However, contributions are welcome and will be reviewed. Feel free to open issues for questions or support.

Quick Start

git clone https://github.com/your-org/bottest.git
cd bottest
cp .env.example .env
# Edit .env and add at least one LLM API key (OpenAI, Anthropic, DeepSeek, or Gemini)
docker-compose up

Once running:

Dashboard: http://localhost:3000
API docs: http://localhost:8000/docs

To record tests, install the Chrome extension:

Run cd extension && npm install && npm run build
Go to chrome://extensions/ and enable Developer mode (top-right)
Click Load unpacked and select the nested extension/ folder (it may be bottest/extension/extension/)

Architecture

Component Overview

Directory	Description
`frontend/`	Next.js dashboard for test management
`backend/`	FastAPI REST API for test orchestration
`extension/`	Chrome extension for recording and replaying tests
`puppeteer/`	BullMQ worker with headless Chromium for cloud test execution

Core Concepts

Understanding the domain model is essential for working with this codebase. Here are the key entities and what they represent:

Entity	Description
Bot	A chatbot application being tested (e.g., "Customer Support Bot", "Sales Assistant"). Acts as a container for related suites and environments.
Environment	A deployment target where tests run. Contains the chatbot URL and optional authentication configuration. A bot can have multiple environments (e.g., staging, production).
Suite	A collection of related tests that run together. Suites group tests logically (e.g., "Onboarding Flow", "FAQ Responses").
Test	A single test case representing one conversation scenario. Contains the recorded interaction steps.
Baseline	The expected conversation outcome for a test. Defines what the chatbot "should" respond with. A test can have multiple baselines for different acceptable responses.
Variant	An alternative phrasing of a test (e.g., same question in a different language, or rephrased). Variants allow testing the same intent with different inputs.
Suite Run	An execution of a suite. When you "run a suite," a suite run is created containing multiple test runs.
Test Run	An execution of a single test within a suite run. Contains one or more variant runs.
Variant Run	An execution of a specific variant. Produces evaluations comparing actual vs expected responses.
Evaluation	The result of comparing an actual chatbot response against baselines. Uses LLM-based assessment to determine Pass/Fail.
Sub-Evaluation	A comparison against a single baseline within an evaluation. An evaluation may compare against multiple baselines.
Evaluation Framework	Step-by-step instructions that guide the LLM on how to evaluate responses for a suite. Customizable per-suite.

Entity Relationships

flowchart TD
    Bot --> Environment
    Bot --> Suite
    Suite --> Test
    Suite --> SuiteRun
    Suite --> EvaluationFramework
    Test --> Baseline
    Test --> Variant
    Test --> TestRun
    SuiteRun --> TestRun
    TestRun --> VariantRun
    VariantRun --> Evaluation
    Evaluation --> SubEvaluation
    Baseline --> SubEvaluation

Codebase Structure

Backend (`backend/`)

The backend is a FastAPI application with a layered architecture separating API routes, business logic, and data models.

backend/
├── src/
│   ├── app.py              # FastAPI application entry point, router registration
│   ├── db.py               # Database engine, session management, connection pooling
│   ├── settings.py         # Configuration loaded from environment variables
│   ├── middleware.py       # Request middleware (logging, CORS)
│   ├── utils.py            # LLM prompt templates and utility functions
│   │
│   ├── api/v1/             # REST API endpoint handlers
│   │   ├── bots.py             # Bot CRUD endpoints
│   │   ├── environments.py     # Environment CRUD endpoints
│   │   ├── suites.py           # Suite CRUD, CSV import/export, preset generation
│   │   ├── tests.py            # Test CRUD endpoints
│   │   ├── baselines.py        # Baseline CRUD endpoints
│   │   ├── variants.py         # Variant CRUD and bulk operations
│   │   ├── suite_runs.py       # Suite execution, status tracking, rebaselining
│   │   ├── test_runs.py        # Test execution endpoints
│   │   ├── variant_runs.py     # Variant run tracking
│   │   ├── evaluations.py      # Evaluation creation and management
│   │   └── analytics.py        # Analytics, reporting, and usage endpoints
│   │
│   ├── core/               # Business logic layer
│   │   ├── bots.py             # Bot operations and management
│   │   ├── environments.py     # Environment operations
│   │   ├── suites.py           # Suite operations, CSV handling
│   │   ├── tests.py            # Test operations
│   │   ├── baselines.py        # Baseline operations
│   │   ├── variants.py         # Variant generation and management
│   │   ├── suite_runs.py       # Suite run execution logic
│   │   ├── test_runs.py        # Test run execution logic
│   │   ├── variant_runs.py     # Variant run tracking
│   │   ├── evaluations.py      # LLM-based evaluation logic
│   │   ├── framework.py        # Evaluation framework generation
│   │   ├── analytics.py        # Analytics computation with caching
│   │   ├── schedules.py        # Scheduled suite run management
│   │   ├── copy_item.py        # Deep copy operations for bots/suites/tests
│   │   ├── redis_client.py     # Redis operations for caching and concurrency
│   │   └── utils.py            # Shared utility functions
│   │
│   └── models/             # Data models
│       ├── db_schema.py        # SQLModel database table definitions
│       ├── api_schema.py       # Pydantic request/response models
│       ├── enums.py            # Status enums (Pass, Fail, Running, etc.)
│       ├── llm_evals.py        # LLM evaluation response models
│       └── base.py             # Base model classes
│
├── alembic/                # Database migrations
│   ├── env.py                  # Alembic environment configuration
│   └── versions/               # Migration scripts
│
├── Dockerfile
├── alembic.ini             # Alembic configuration
└── requirements.txt

Key files to understand:

src/app.py - Application bootstrap, see how routers and middleware are registered
src/models/db_schema.py - All database models, understand the data structure
src/core/evaluations.py - LLM evaluation logic, the heart of the testing system

Frontend (`frontend/`)

The frontend is a Next.js 14 application using the App Router pattern with React Context for state management.

frontend/src/
├── app/                    # Next.js App Router
│   ├── app/                    # Main application routes (dashboard pages)
│   │   ├── dashboard/              # Overview dashboard
│   │   ├── tests/                  # Test management page
│   │   ├── suite-runs/             # Suite run history and details
│   │   ├── analytics/              # Analytics charts and trends
│   │   ├── reports/                # Comparison reports
│   │   ├── usage/                  # Usage statistics
│   │   └── layout.tsx              # App layout with sidenav
│   ├── api/                    # API routes (server-side)
│   │   └── download/               # File download endpoints
│   ├── layout.tsx              # Root layout
│   └── page.tsx                # Home page (redirects to dashboard)
│
├── hooks/                  # Custom React hooks for API calls
│   ├── useBots.ts              # Bot CRUD operations
│   ├── useTests.ts             # Test operations
│   ├── useSuites.ts            # Suite operations
│   ├── useSuiteRuns.ts         # Suite run operations
│   ├── useEnvironments.ts      # Environment operations
│   ├── useAnalyticsReport.ts   # Analytics data fetching
│   ├── useOverview.ts          # Dashboard overview data
│   └── useApi.ts               # Base API hook with Axios
│
├── components/             # React components (organized by feature)
│   ├── modals/                 # Modal dialog components
│   ├── analyticsCharts/        # Chart components (ApexCharts)
│   ├── sidenav/                # Navigation sidebar
│   ├── statusBar/              # Status indicators
│   └── testItem/               # Test list item components
│
├── elements/               # Reusable UI primitives
│   ├── button/                 # Button component
│   ├── input/                  # Input component
│   ├── select/                 # Select dropdown
│   ├── modal/                  # Modal wrapper
│   └── chip/                   # Status chip component
│
├── utils/                  # Utility functions and types
│   └── typesInterface.ts       # TypeScript type definitions
│
├── globalState.tsx         # React Context definition
├── globalStateProvider.tsx # Global state provider
└── themeProvider.tsx       # Theme provider (dark mode support)

Key files to understand:

hooks/useApi.ts - Base API hook, all API calls go through here
globalStateProvider.tsx - Global state management, selected bot/suite/environment
app/app/layout.tsx - Main app layout with navigation

Chrome Extension (`extension/`)

The extension is a Chrome Manifest V3 extension with a React popup UI and content scripts for recording/replay.

extension/src/
├── manifest.ts             # Extension manifest configuration (MV3)
│
├── background/             # Service worker (background scripts)
│   ├── index.ts                # Main entry, message handling, test coordination
│   ├── utils.ts                # Tab management, play history tracking
│   └── lib/
│       ├── api.ts              # Backend API communication
│       ├── playing.ts          # Test execution queue management
│       └── evaluation.ts       # Evaluation result handling
│
├── contents/               # Content scripts (injected into web pages)
│   └── all/
│       ├── index.ts            # Main content script coordinator
│       ├── recorder.ts         # Records user interactions (clicks, keypresses)
│       ├── player.ts           # Replays recorded interactions
│       ├── mutation.ts         # DOM mutation detection and waiting
│       ├── selector.ts         # Element selector generation
│       └── utils.ts            # Conversation HTML extraction
│
├── popup/                  # Extension popup UI (React application)
│   ├── App.tsx                 # Main React app entry
│   ├── Routes.tsx              # React Router configuration
│   ├── screens/                # Screen components (Dashboard, Recording, etc.)
│   ├── components/             # Reusable UI components
│   ├── lib/                    # API client libraries (mirrors backend resources)
│   ├── hooks/                  # React hooks (useRunTest, useTheme, etc.)
│   └── context/                # React context providers
│
└── utils/                  # Shared utilities
    └── index.tsx

Key files to understand:

contents/all/recorder.ts - How user interactions are captured
contents/all/player.ts - How recorded tests are replayed
background/lib/playing.ts - Test execution queue and coordination

Puppeteer Service (`puppeteer/`)

The Puppeteer service is a BullMQ worker that processes evaluation jobs from a Redis queue. It runs headless Chromium to execute tests.

puppeteer/src/
├── worker.ts               # BullMQ worker entry point
│                           #   - Connects to Redis queue
│                           #   - Processes evaluation jobs
│                           #   - Manages browser sessions
├── local.ts                # Local development runner (CLI tool)
│
├── types/                  # TypeScript type definitions
│   └── index.ts
│
└── utils/
    ├── browser.ts              # Puppeteer browser management
    │                           #   - Browser initialization
    │                           #   - Page creation and navigation
    │                           #   - Screenshot capture
    ├── utils.ts                # Replay execution and HTML extraction
    │                           #   - playReplayJson() - main replay orchestrator
    │                           #   - waitForMutations() - DOM change detection
    │                           #   - getConversationHTML() - response extraction
    └── storage.ts              # Storage and state management
                                #   - Redis for concurrency tracking
                                #   - Local filesystem for screenshots
                                #   - Backend API communication

Key files to understand:

worker.ts - BullMQ worker that consumes jobs from the queue
utils/utils.ts - Core replay logic, same approach as the extension
utils/storage.ts - Backend API calls and file storage

How It Works

Recording Flow

User opens the extension popup and clicks "Record"
Extension opens a new tab and injects the content script
Content script starts listening for DOM events:
- Active events: clicks, keypresses, form inputs (user actions)
- Passive events: DOM mutations (chatbot responses)
Each event is timestamped and stored with element selectors
Recording is saved as JSON with events and timing deltas
User saves the test to the backend via the extension API

Replay Flow (Extension)

User triggers a test run from the extension or dashboard
Extension opens a new browser tab
Navigates to the environment URL
Content script replays events from the JSON recording:
- Executes user actions (clicks, typing)
- Waits for DOM mutations to settle after each action
- Uses the same timing as the original recording
Captures the final conversation HTML
Sends HTML to backend for evaluation

Replay Flow (Puppeteer/Cloud)

Suite run is triggered from dashboard or scheduled
Backend creates evaluation jobs and pushes them to Redis queue (BullMQ)
Puppeteer worker consumes jobs from the queue
Worker checks concurrency limits in Redis before processing
For each evaluation job:
- Launches/reuses headless Chromium browser
- Handles authentication if configured
- Navigates to environment URL
- Replays events (same logic as extension)
- Captures screenshots at key points
- Extracts conversation HTML
- Submits results to backend API
Failed jobs are automatically retried with exponential backoff
Results aggregated into test run and suite run status

Evaluation Flow

Backend receives conversation HTML and test configuration
Loads baselines (expected responses) for the test
Loads evaluation framework for the suite
Calls LLM API (OpenAI, Anthropic, DeepSeek, or Gemini):
- Sends actual conversation HTML
- Sends expected baselines
- Sends evaluation framework instructions
LLM compares actual vs expected using framework steps
Returns Pass/Fail with reasoning for each baseline
Results stored as sub-evaluations
Overall status bubbles up: Evaluation → Variant Run → Test Run → Suite Run

Key Patterns

Backend Patterns

Layered Architecture: API routes (api/) handle HTTP, business logic lives in core/, data structures in models/
SQLModel ORM: Combines SQLAlchemy and Pydantic for type-safe database models
Decorator-based Sessions: @with_db_session decorator handles database session lifecycle
Single-tenant Mode: Simplified auth - all requests use a system actor

Frontend Patterns

App Router: Next.js 14 file-based routing in app/ directory
React Context: Global state via GlobalStateProvider for selected bot/suite/environment
Custom Hooks: Each resource has a hook (useBots, useTests) encapsulating API calls
LocalStorage Sync: Selected items persist across sessions

Extension Patterns

Manifest V3: Modern Chrome extension architecture with service worker
Content Script Architecture: Single content script handles both recording and replay
Chrome Messaging: Background ↔ Content communication via chrome.runtime API
Mutation Observers: Detect DOM changes to know when chatbot has responded

Puppeteer Patterns

BullMQ Worker: Consumes evaluation jobs from Redis queue
Automatic Retries: Failed jobs retry with exponential backoff (up to 3 attempts)
Two-Level Concurrency Control:
- Worker Concurrency (WORKER_CONCURRENCY): How many jobs/browsers this worker runs simultaneously. Each worker has its own browser pool. Default: 10.
- Suite Run Concurrency (set in frontend): Global limit per suite run, tracked in Redis across all workers. Prevents overloading the target website.
Browser Pool: Each worker maintains a pool of browser instances (one per concurrent job). Separate browser instances provide process-level isolation -- no shared memory pressure, no risk of one evaluation's crash affecting another, and simpler sequential code per job. Idle browsers are recycled between jobs for efficiency.

Prerequisites

Docker and Docker Compose (for containerized deployment)
Python 3.10+ (for backend development)
Node.js 18+ (for frontend/extension development)
PostgreSQL 15+ (if running locally)
Redis 7+ (if running locally)
At least one LLM API key (OpenAI, Anthropic, DeepSeek, or Gemini)

Configuration

There is a single .env.example file at the root of the project. It contains all variables needed for both Docker and local development.

cp .env.example .env
# Edit .env with your values

How Environment Variables Are Loaded

Each component loads environment variables differently. The root .env file is the single source of truth, but components access it in different ways:

Component	Method	Location	Notes
Backend	`python-dotenv`	Copies root `.env` to `backend/`	Loaded in `settings.py` via `load_dotenv()`
Frontend	Next.js built-in	`frontend/.env.local`	Only `NEXT_PUBLIC_*` vars exposed to browser
Extension	`dotenv-webpack`	Root `.env` (via webpack)	Variables injected at build time
Puppeteer	`process.env`	Runtime environment	Set via Docker or shell exports

Backend (Python/FastAPI)

The backend uses python-dotenv to load a .env file from its working directory. For local development, copy the root .env file:

cd backend
cp ../.env .env

Variables are loaded in src/settings.py:

from dotenv import load_dotenv
load_dotenv()  # Loads .env from current directory

Frontend (Next.js)

Next.js automatically loads .env.local from the frontend folder. Only variables prefixed with NEXT_PUBLIC_ are exposed to the browser.

cd frontend

Create a .env.local file in the frontend folder with the following content:

NEXT_PUBLIC_ROOT_API_URL=http://localhost:8000

Extension (Chrome/Webpack)

The extension uses dotenv-webpack which reads from the root .env file at build time. Variables are injected into the bundle via webpack's DefinePlugin.

The path is configured in extension/server/configs/webpack.common.ts:

new Dotenv({
  path: resolve(PROJECT_ROOT, "../.env"),  // Points to root .env
})

Variables like BACKEND_URL become available as process.env.BACKEND_URL in the extension code.

Puppeteer (Node.js Worker)

The Puppeteer worker uses dotenv to load environment variables from a .env file in the puppeteer folder. In Docker, variables are passed through docker-compose.yml.

Create a .env file in the puppeteer folder:

REDIS_URL=redis://localhost:6379
BACKEND_URL=http://localhost:8000
CHROMIUM_PATH=./node_modules/localChromium/chromium/[...]

Note: The path to your Chromium installation will depend on your OS. Follow the Local Setup Instructions for more information on what to set it as.

Environment Variables by Purpose

The .env.example contains variables organized by purpose:

Section	Variables	Used By
Docker Compose	`DB_PASSWORD`	`docker-compose.yml` constructs database connection strings from this
Local Development	`DATABASE_URI`, `REDIS_URL`, `ENVIRONMENT`	Backend when running directly (not in Docker)
LLM APIs	`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, `GEMINI_API_KEY`	Backend for evaluating chatbot responses
Storage	`STORAGE_PATH`	Puppeteer worker for storing screenshots
Security	`ENCRYPTION_KEY`	Backend for encrypting auto-login credentials

Docker vs Local Development

Docker Compose reads the root .env file automatically. It uses DB_PASSWORD to construct the database connection string, so you don't need to set DATABASE_URI.

Local Development requires you to set DATABASE_URI and REDIS_URL manually because you're connecting to databases directly. The .env.example includes both sets of variables so one file works for either setup.

Component-Specific Variables

The frontend and extension each need one environment variable pointing to the backend:

Component	File	Variable
Frontend	`frontend/.env.local`	`NEXT_PUBLIC_ROOT_API_URL=http://localhost:8000`
Extension	`extension/.env`	`BACKEND_URL=http://localhost:8000`

All Variables Reference

Variable	Description	Required
`DB_PASSWORD`	PostgreSQL password (Docker only)	Docker
`DATABASE_URI`	PostgreSQL connection string	Local dev
`REDIS_URL`	Redis connection string	Local dev
`ENVIRONMENT`	Set to `local` for local development	Local dev
`OPENAI_API_KEY`	OpenAI API key	At least one LLM
`ANTHROPIC_API_KEY`	Anthropic API key	At least one LLM
`DEEPSEEK_API_KEY`	DeepSeek API key	At least one LLM
`GEMINI_API_KEY`	Google Gemini API key	At least one LLM
`ENCRYPTION_KEY`	32-character encryption key	Yes
`STORAGE_PATH`	Local storage path for screenshots	Optional
`CHROMIUM_PATH`	Path to Chromium (Puppeteer only)	Puppeteer
`WORKER_CONCURRENCY`	Concurrent jobs and browsers per worker (default: 10)	Optional
`HEADLESS`	Set to `false` to show browser window for debugging	Optional

LLM Model Configuration

The LLM models used for evaluations are configured in backend/src/core/utils.py. The models were set as of January 2026 and should be updated periodically to use the most recent frontier models from each provider.

Current model configuration:

Provider	Default (Fast)	Available Models
OpenAI	`gpt-5-nano`	`gpt-5-nano`, `gpt-5-mini`, `gpt-5.2`
Anthropic	`claude-haiku-4-5`	`claude-haiku-4-5`, `claude-sonnet-4-5`, `claude-opus-4-5`
DeepSeek	`deepseek-chat`	`deepseek-chat` (V3.2), `deepseek-reasoner`
Gemini	`gemini-3-flash-preview`	`gemini-3-flash-preview`, `gemini-3-pro-preview`

Recommendation: Check each provider's documentation for their latest models and update the constants in backend/src/core/utils.py accordingly. The codebase uses a tiered approach:

Fast/cheap models (defaults) for simple parsing and extraction tasks
Reasoning models (gpt-5.2, deepseek-reasoner) for complex evaluation and analysis tasks

Running with Docker

# Start all services
docker-compose up

# Start in background
docker-compose up -d

# View logs
docker-compose logs -f

# View puppeteer worker logs specifically
docker-compose logs -f puppeteer-worker

# Stop services
docker-compose down

This starts:

PostgreSQL - Database
Redis - Job queue and caching
Backend - FastAPI REST API
Frontend - Next.js dashboard
Puppeteer Worker - Processes cloud test executions

Data Persistence

Docker Compose uses named volumes (postgres_data, redis_data, screenshot_data), so your data persists across container restarts and docker-compose down.

# Stop services but KEEP data
docker-compose down

# Stop services and DELETE all data
docker-compose down -v

Development Setup

To run the backend and frontend locally with live reloading (using Docker only for the PostgreSQL and Redis databases).

Local development requires these processes (each in a separate terminal):

Process	Required	Command	Description
PostgreSQL	Yes	`docker start postgres`	Database
Redis	Yes	`docker start redis`	Queue & caching
Backend	Yes	`uvicorn src.app:app --reload`	API server
Frontend	Yes	`npm run dev`	Dashboard
Extension	For recording	`npm run start`	Chrome extension
Puppeteer Worker	For cloud execution	`npm run worker:dev`	Runs tests headlessly
Scheduler	For scheduled runs	`python -m src.scheduler`	Executes scheduled suite runs

Note: When using docker-compose up, all services start automatically. For local development, you need to start each process manually.

1. Start PostgreSQL and Redis

# Using Docker for just the databases (data lost when container is removed)
docker run -d --name postgres -p 5432:5432 -e POSTGRES_USER=bottest -e POSTGRES_PASSWORD=bottest -e POSTGRES_DB=bottest postgres:15-alpine
docker run -d --name redis -p 6379:6379 redis:7-alpine

For persistent data (recommended), add volume mounts:

# Linux/MacOS - data persists across container restarts
docker run -d --name postgres -p 5432:5432 \
  -e POSTGRES_USER=bottest -e POSTGRES_PASSWORD=bottest -e POSTGRES_DB=bottest \
  -v bottest_postgres:/var/lib/postgresql/data \
  postgres:15-alpine

docker run -d --name redis -p 6379:6379 \
  -v bottest_redis:/data \
  redis:7-alpine

# Windows PowerShell - data persists across container restarts
docker run -d --name postgres -p 5432:5432 -e POSTGRES_USER=bottest -e POSTGRES_PASSWORD=bottest -e POSTGRES_DB=bottest -v bottest_postgres:/var/lib/postgresql/data postgres:15-alpine

docker run -d --name redis -p 6379:6379 -v bottest_redis:/data redis:7-alpine

Note: Named volumes (like bottest_postgres) persist until explicitly deleted with docker volume rm.

Managing containers:

# Stop containers (keeps data)
docker stop postgres redis

# Start them again
docker start postgres redis

# Remove containers (volumes persist separately)
docker rm postgres redis

# Delete volumes to reset data
docker volume rm bottest_postgres bottest_redis

2. Backend

cd backend
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or: venv\Scripts\activate  # Windows

pip install -r requirements.txt
cp ../.env .env  # Copy from root .env
alembic upgrade head
uvicorn src.app:app --reload --port 8000

API docs available at http://localhost:8000/docs (Swagger) or http://localhost:8000/redoc

Scheduler (Optional - for Scheduled Runs)

If you want to use the Scheduled Runs feature, start the scheduler in a separate terminal:

cd backend
python -m src.scheduler

Note: When using Docker Compose, the scheduler runs automatically as a separate container. For local development, you need to start it manually. The scheduler syncs with the database every 30 seconds and executes suite runs based on their cron schedules.

3. Frontend

cd frontend
npm install
npm run dev

Create a .env.local file in the frontend folder:

NEXT_PUBLIC_ROOT_API_URL=http://localhost:8000

Dashboard available at http://localhost:3000

4. Chrome Extension

cd extension
npm install
npm run start

Create a .env file in the extension folder:

BACKEND_URL=http://localhost:8000

Then load the extension in Chrome:

Go to chrome://extensions/
Enable Developer mode using the toggle in the top-right corner
Click Load unpacked (appears after enabling Developer mode)
Select the extension/ folder and click "Select Folder"
The extension should now appear in your list—click the reload icon (↻) after making code changes

5. Puppeteer (Optional)

Only needed for cloud/headless test execution:

cd puppeteer
npm install

Install Chromium:

# MacOS/Linux:
npx @puppeteer/browsers install chromium@latest --path $(pwd)/node_modules/localChromium

# Windows (PowerShell):
npx @puppeteer/browsers install chromium@latest --path $PWD\node_modules\localChromium

Configure environment: Create a .env file in the puppeteer folder with your Chromium path (adjust version number based on what was installed):

Tip: MacOS path typically looks like: /Users/you/project/puppeteer/node_modules/localChromium/chromium/mac_arm-XXXXXX/chrome-mac/Chromium.app/Contents/MacOS/Chromium

Tip: Windows path typically looks like: C:/Users/you/project/node_modules/localChromium/chromium/win64-XXXXXX/chrome-win/chrome.exe

# Local Chromium Installation Path
CHROMIUM_PATH=full_path_to_chromium

# Backend API (optional, defaults to http://localhost:8000)
BACKEND_URL=http://localhost:8000

# Redis (optional, defaults to localhost:6379)
REDIS_URL=redis://localhost:6379

# Concurrency - how many jobs/browsers to run simultaneously
# Each job gets its own browser instance for process-level isolation.
# Higher = more parallelism but more memory (~150MB per browser).
# Recommended: 5 for 8GB RAM, 10 for 16GB RAM
WORKER_CONCURRENCY=5

# Set to false to see browser windows (useful for debugging)
# HEADLESS=false

Important: Use an absolute path with forward slashes for CHROMIUM_PATH, even on Windows.

Concurrency:

There are two concurrency controls:

Worker Concurrency (WORKER_CONCURRENCY): How many jobs run simultaneously on this worker. Each job gets its own browser instance for process-level isolation, ensuring one evaluation can't affect another.
Suite Run Concurrency (set in frontend): Max concurrent evaluations for a suite run, tracked globally in Redis via atomic Lua scripts. Prevents overloading the target website.

The suite run limit is the effective cap since it's enforced globally. Set WORKER_CONCURRENCY to match or exceed your typical suite run limit.

Visual Debugging:

To watch the browser in action:

Set HEADLESS=false in .env
Set WORKER_CONCURRENCY=1 (easier to watch one browser)
Restart the worker

Running the Puppeteer Worker (Full Cloud Execution)

To enable cloud/headless test execution with the full application:

Make sure Redis is running (see Backend setup above)
Start the worker:

npm run worker:dev

The worker connects to Redis and processes evaluation jobs from the queue. Keep this running alongside the backend and frontend for full cloud execution support.

Testing a Single Replay File (Debugging)

To test a single replay.json file locally without the full system:

npm run start -- <url> <replay_json_path>
# Example: npm run start -- https://example.com/chatbot ./test_replay.json

This opens a browser window and runs the replay visually, useful for debugging tests.

Storage Configuration

By default, screenshots are stored on the local filesystem at /app/storage (or ./storage for local development).

STORAGE_PATH=/app/storage

Using Cloud Storage (S3, GCS, Azure Blob, etc.):

To use a cloud storage provider instead of local filesystem:

Install your provider's SDK in puppeteer/package.json:

cd puppeteer && npm install @aws-sdk/client-s3  # or equivalent

Modify puppeteer/src/utils/storage.ts - Update the uploadScreenshot function to use your provider's SDK instead of local filesystem.
Update the download endpoint in frontend/src/app/api/download/screenshot/route.ts to fetch from your storage.
Add environment variables for your provider (e.g., bucket name, credentials)

Self-Hosting on a Custom Domain

To deploy on your own domain, update the following:

What	Change	File
Frontend → Backend	Set `NEXT_PUBLIC_ROOT_API_URL` to your backend URL	`frontend/.env.local`
Extension → Backend	Set `BACKEND_URL` to your backend URL, then rebuild	Root `.env`
Extension → Frontend	Add your frontend origin to `externally_connectable.matches`	`extension/src/manifest.ts`
Puppeteer → Backend	Set `BACKEND_URL` to your backend URL	Puppeteer `.env` or Docker env
Dashboard links (optional)	Replace `bottest.ai` URLs with your frontend domain	`extension/src/popup/screens/Dashboard/index.tsx`, `StartRecording.tsx`

Note: Chrome does not allow wildcards in externally_connectable — you must specify your exact frontend domain (e.g. https://app.yourdomain.com/*). After changes, rebuild the extension and reload it in chrome://extensions/.

Database Migrations

This project uses Alembic for database migrations. Alembic is a database migration tool for SQLAlchemy (which SQLModel is built on). It tracks changes to your database schema over time, allowing you to:

Version control your database schema alongside your code
Apply incremental changes to the database structure
Roll back changes if something goes wrong
Keep development, staging, and production databases in sync

When to Run Migrations

After pulling new code that includes new migration files in backend/alembic/versions/
After modifying models in backend/src/models/db_schema.py
When setting up a fresh database for the first time

Migration Commands

All commands run from the backend/ directory:

cd backend

# Apply all pending migrations (run this after pulling code)
alembic upgrade head

# Create a new migration after modifying models
alembic revision --autogenerate -m "description of change"

# Roll back one migration
alembic downgrade -1

# View current migration status
alembic current

# View migration history
alembic history

Migration Files

Migration scripts are located in backend/alembic/versions/. Each file represents a schema change and includes:

upgrade() - Apply the change
downgrade() - Reverse the change

Testing

This project includes tests for all four components. Each component uses its own test framework appropriate for its technology stack.

Running Tests

Backend (Python/pytest):

cd backend
pip install -r requirements.txt  # includes test dependencies
pytest                           # run all tests
pytest tests/unit/               # run unit tests only
pytest tests/integration/        # run integration tests only
pytest -v                        # verbose output

Frontend (Vitest):

cd frontend
npm install
npm test                         # run all tests
npm run test:watch               # watch mode

Extension (Vitest):

cd extension
npm install
npm test                         # run all tests

Puppeteer Worker (Vitest):

cd puppeteer
npm install
npm test                         # run all tests

Test Architecture

Backend: Uses pytest with SQLite in-memory database for fast, isolated tests
Frontend/Extension/Puppeteer: Use Vitest with React Testing Library for component and hook testing
API calls are mocked using MSW (Mock Service Worker) in frontend tests
Chrome extension APIs are mocked for extension tests

Optional: CI/CD Setup

To run tests automatically on every push/PR, you can add a GitHub Actions workflow. Create .github/workflows/test.yml:

name: Tests
on: [push, pull_request]
jobs:
  backend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r backend/requirements.txt
      - run: pytest backend/

  frontend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
        working-directory: frontend
      - run: npm test
        working-directory: frontend

  extension:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
        working-directory: extension
      - run: npm test
        working-directory: extension

  puppeteer:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
        working-directory: puppeteer
      - run: npm test
        working-directory: puppeteer

License

This project is licensed under the MIT License - see the LICENSE file for details.

Name		Name	Last commit message	Last commit date
backend		backend
branding		branding
extension		extension
frontend		frontend
migration_scripts		migration_scripts
puppeteer		puppeteer
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation

Table of Contents

Quick Start

Architecture

Component Overview

Core Concepts

Entity Relationships

Codebase Structure

Backend (backend/)

Frontend (frontend/)

Chrome Extension (extension/)

Puppeteer Service (puppeteer/)

How It Works

Recording Flow

Replay Flow (Extension)

Replay Flow (Puppeteer/Cloud)

Evaluation Flow

Key Patterns

Backend Patterns

Frontend Patterns

Extension Patterns

Puppeteer Patterns

Prerequisites

Configuration

How Environment Variables Are Loaded

Backend (Python/FastAPI)

Frontend (Next.js)

Extension (Chrome/Webpack)

Puppeteer (Node.js Worker)

Environment Variables by Purpose

Docker vs Local Development

Component-Specific Variables

All Variables Reference

LLM Model Configuration

Running with Docker

Data Persistence

Development Setup

1. Start PostgreSQL and Redis

2. Backend

Scheduler (Optional - for Scheduled Runs)

3. Frontend

4. Chrome Extension

5. Puppeteer (Optional)

Running the Puppeteer Worker (Full Cloud Execution)

Testing a Single Replay File (Debugging)

Storage Configuration

Self-Hosting on a Custom Domain

Database Migrations

When to Run Migrations

Migration Commands

Migration Files

Testing

Running Tests

Test Architecture

Optional: CI/CD Setup

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Backend (`backend/`)

Frontend (`frontend/`)

Chrome Extension (`extension/`)

Puppeteer Service (`puppeteer/`)