Skip to content

feat(evals): add local VQA dataset editor - #3605

Draft
ruthwikdasyam wants to merge 3 commits into
ruthwik/feat/vqa-pointcloudfrom
ruthwik/feat/vqa-visualizer
Draft

feat(evals): add local VQA dataset editor#3605
ruthwikdasyam wants to merge 3 commits into
ruthwik/feat/vqa-pointcloudfrom
ruthwik/feat/vqa-visualizer

Conversation

@ruthwikdasyam

Copy link
Copy Markdown
Contributor

Contribution path

Problem

VQA datasets need a local workflow for frame-level generation, review, and correction.

Solution

Add a browser-based dataset editor with atomic submission, generation feedback, and editor-startup model preloading.

How to Test

uv run dimos evals vqa edit /path/to/recording.db /path/to/vqa-dataset

AI assistance

OpenCode with GPT-5.6-sol provided substantial implementation and validation assistance.

Checklist

  • I have read and approved the CLA.

@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.21400% with 95 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/evals/vqa/editor.py 82.96% 34 Missing and 28 partials ⚠️
dimos/evals/vqa/visualizer.py 72.94% 18 Missing and 5 partials ⚠️
dimos/evals/vqa/preprocessing.py 28.57% 5 Missing ⚠️
dimos/evals/vqa/test_visualizer.py 93.65% 3 Missing and 1 partial ⚠️
dimos/evals/vqa/test_editor.py 99.44% 1 Missing ⚠️
@@                       Coverage Diff                       @@
##           ruthwik/feat/vqa-pointcloud    #3605      +/-   ##
===============================================================
+ Coverage                        76.61%   76.67%   +0.05%     
===============================================================
  Files                             1260     1265       +5     
  Lines                           122285   123061     +776     
  Branches                         10874    10949      +75     
===============================================================
+ Hits                             93691    94355     +664     
- Misses                           25444    25521      +77     
- Partials                          3150     3185      +35     
Flag Coverage Δ
OS-ubuntu-24.04-arm 71.30% <87.21%> (+0.09%) ⬆️
OS-ubuntu-latest 73.01% <87.21%> (+0.08%) ⬆️
Py-3.10 73.01% <87.21%> (+0.07%) ⬆️
Py-3.11 73.01% <87.21%> (+0.07%) ⬆️
Py-3.12 73.01% <87.21%> (+0.07%) ⬆️
Py-3.13 73.01% <87.21%> (+0.08%) ⬆️
Py-3.14 73.01% <87.21%> (+0.07%) ⬆️
Py-3.14t 73.01% <87.21%> (+0.07%) ⬆️
SelfHosted-Large 29.76% <21.93%> (-0.05%) ⬇️
SelfHosted-Linux 35.59% <21.93%> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/cli/test_vqa.py 100.00% <100.00%> (ø)
dimos/cli/vqa.py 96.66% <100.00%> (+0.66%) ⬆️
dimos/evals/vqa/author.py 91.48% <100.00%> (+3.98%) ⬆️
dimos/evals/vqa/generate.py 81.31% <100.00%> (ø)
dimos/evals/vqa/test_contracts.py 98.92% <100.00%> (+<0.01%) ⬆️
dimos/evals/vqa/test_preprocessing.py 100.00% <100.00%> (ø)
dimos/evals/vqa/test_editor.py 99.44% <99.44%> (ø)
dimos/evals/vqa/test_visualizer.py 93.65% <93.65%> (ø)
dimos/evals/vqa/preprocessing.py 60.33% <28.57%> (+1.81%) ⬆️
dimos/evals/vqa/visualizer.py 72.94% <72.94%> (ø)
... and 1 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant