[nissrin2020ali-ux]

Motivation

VLMEvalKit evaluates large vision-language models against a large registry of named API providers — Together AI, SiliconFlow, MiniMax, and others each have a first-class wrapper in vlmeval/api/ plus preset entries in vlmeval/config.py. When benchmarking with an external model gateway, users currently have no way to pick OrcaRouter without treating it as an anonymous custom --base-url. This PR adds OrcaRouter as a first-class provider in the same style as the existing Together AI integration, so Cosmos Reasoner benchmarks can run against OrcaRouter's model namespace directly.

OrcaRouter is an OpenAI-compatible AI gateway built for both models and agents. Like OpenRouter, it exposes a provider/model namespace across many models — but it also combines adaptive routing, automatic failover, zero-markup inference, observability, guardrails, and agent-tool governance behind the same endpoint. Adding orcarouter as a first-class provider means this project's users can use that stack directly, without treating OrcaRouter as an anonymous custom base URL.

It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.

Changes

• vlmeval/api/orcarouter.py — new OrcaRouterAPI wrapper, modeled directly on vlmeval/api/together.py: OpenAI-compatible chat completions with image_url (base64 or URL) support, ORCAROUTER_API_KEY env key resolution, retries, and a configurable api_base defaulting to https://api.orcarouter.ai/v1/chat/completions.
• vlmeval/api/__init__.py — import and export OrcaRouterAPI.
• vlmeval/config.py — four preset api_models entries: OrcaRouter_Gemini-3.1-Flash-Lite, OrcaRouter_Gemini-3.5-Flash, OrcaRouter_Claude-Sonnet-5, and OrcaRouter_DeepSeek-V4-Flash-Vision.
• docs/en/Quickstart.md — add ORCAROUTER_API_KEY to the documented .env API-key list.

Verification

• python -m py_compile and flake8 --max-line-length=99 pass on the new files (config.py is pre-commit-excluded).
• L3 live round-trips through the new OrcaRouterAPI class against the live https://api.orcarouter.ai/v1 endpoint succeeded for all four preset models (google/gemini-3.1-flash-lite, google/gemini-3.5-flash, anthropic/claude-sonnet-5, deepseek/deepseek-v4-flash-vision-exp), each returning ret_code=0 with a valid completion.

Discord: discord.gg/YEubt8enRA · X: https://x.com/OrcaRouter

I'm an engineer on the OrcaRouter team.

[yaoxu-crypto]

Thank you for the contribution and for checking all four presets against the live endpoint, @nissrin2020ali-ux.

We're going to close this one, for two reasons:

1. The code has moved. The evaluation suites, including this VLMEvalKit copy, moved to NVIDIA/cosmos-framework under .agents/skills/evaluation/ and were removed from this repo in Sync repository contents with nvidia-cosmos/cosmos-private #384. This PR can no longer merge here.
2. Provider integrations belong upstream. Our vlmevalkit/ directory is a vendored copy of open-compass/VLMEvalKit, kept close to upstream plus Cosmos-specific benchmarks. We don't add third-party API providers to our copy, so we stay neutral between gateways and keep the fork easy to sync. Upstream VLMEvalKit is the right place for an OrcaRouter wrapper, and Cosmos users would pick it up from there. Meanwhile, any OpenAI-compatible endpoint can already be used through the custom base-URL path.

Thanks again for taking the time.