Expanded LLM/STT/TTS usage tracking across pre-prod flows and some fixes

[MSami625]

What Changed?

• Expanded LLM/STT/TTS usage tracking across pre-prod flows: evaluator results, Metrics Studio runs, persona generation, playground voice calls, and test-agent LLM-to-LLM simulation
• Added external_agent_usage.py to extract and record provider usage from Vapi, Retell, ElevenLabs, and Smallest call payloads
• Wired usage recording into playground polling, evaluator result processing, and test-agent bridge/simulation paths (with dedup via external_usage_recorded)
• Added LLM-to-LLM evaluator simulation for voice-bundle-only agents (no external voice provider)
• Fixed Celery run_evaluator asyncio event loop handling for voice-bridge runs
• Improved playground polling (skip duplicate processing, Vapi polls on refresh only)
• Relaxed CSP connect-src / worker-src for voice provider WebRTC connections
• Added tests for pre-prod usage, simulation usage, LLM-to-LLM simulation, and CSP voice provider rules

Why?

Pre-prod testing (evaluators, playground, personas, Metrics Studio) was not consistently attributing LLM/STT/TTS usage, and external voice provider token usage from call payloads was not being captured. This gives accurate per-workspace usage/cost tracking across synthetic testing and live provider calls.

How to Test?

1. pytest tests/test_services/test_usage/test_pre_prod_usage.py tests/test_services/test_usage/test_test_agent_simulation_usage.py tests/test_services/test_testing/test_llm_to_llm_evaluator_simulation.py tests/test_core/test_security_headers_middleware.py -v
2. Run an evaluator with voice bundle only → confirm LLM-to-LLM simulation completes and usage is recorded under TEST_AGENT
3. Run an evaluator with voice bridge → confirm bridge call works (no asyncio event loop error) and usage is tracked
4. Make a playground web call (Vapi/Retell/ElevenLabs) → confirm provider LLM/STT/TTS usage is recorded once
5. Run a Metrics Studio batch → confirm usage is attributed to the metric studio run
6. Verify playground loads and connects to voice providers without CSP errors

Release Label

• major - breaking change, next release bumps major version
• minor - backward-compatible feature, next release bumps minor version
• fix - backward-compatible bug fix, next release bumps patch version
• No label (defaults to patch release)

Checklist

• [ x]] I have read the CONTRIBUTING.md guide.
• [ x]] My code follows the project's style guidelines.
• I have added tests that prove my fix is effective or my feature works.
• [ x]] I have updated documentation where needed.

[greptile-apps]

Greptile Summary

The PR expands local and external billing attribution across evaluator, playground, Metrics Studio, persona, and test-agent workflows, and adds internal LLM-to-LLM evaluator simulation.

• Adds provider payload extraction and contextual LLM/STT/TTS recording.
• Adds atomic row locking around playground result creation and a persisted usage marker.
• Updates evaluator simulation, Celery event-loop handling, frontend voice flows, CSP policy, Flexprice events, and focused tests.

Confidence Score: 3/5

The PR does not appear safe to merge until playground usage persistence is made retry-safe and evaluator simulation retries stop replaying already billed turns.

Provider usage is still applied before its durable dedup marker commits, allowing later polls to double-count after a database failure, while LLM simulation failures retry from the beginning after earlier turns have already added usage.

Files Needing Attention: app/services/playground/post_call_processing.py, app/services/usage/external_agent_usage.py, app/workers/tasks/run_evaluator.py, app/services/testing/llm_to_llm_evaluator_simulation.py

Important Files Changed

Filename	Overview
app/services/playground/post_call_processing.py	Adds row-locked playground post-call deduplication, but the previously reported gap between additive usage application and dedup-flag commit remains.
app/api/v1/routes/playground.py	Integrates the new post-call claims and prevents duplicate evaluator creation through a locked create-and-link transaction.
app/services/usage/external_agent_usage.py	Extracts provider-specific usage and records contextual LLM, STT, and TTS counters while propagating incomplete recording outcomes.
app/workers/tasks/run_evaluator.py	Adds voice-bundle LLM simulation and safer event-loop handling, but whole-task retries still replay already billed simulation turns.
app/services/testing/llm_to_llm_evaluator_simulation.py	Implements turn-based evaluator simulation and records usage per call, with durable simulation state written only after all turns finish.

_{Reviews (12): Last reviewed commit: "feat(playground): enhance call data proc..." | Re-trigger Greptile}