Pipeline for building a vision-language fine-tuning dataset that targets localizing UI elements on web-page screenshots from natural-language instructions (e.g., "Locate the button labeled 'Submit'" → bbox + center point).
The published dataset: Khabner/moondream-data.
A Russian-language version of this README is available at README_RU.md.
| revision | records | role |
|---|---|---|
ec637966 |
18 694 | Recommended for production fine-tuning (default of build_dataset.py). Earlier, simpler cut: 15 base section types, no paraphrase pipeline yet, no calendar / modal / forum sections. |
ba10ae2 |
21 832 | Built through subsequent benchmark-targeted iteration: paraphrase pipeline (~800 pairs), 5 additional synthetic section types (calendar_widget, product_card, dense_action_panel, modal_dialog, forum_thread), icon_controls weight reduced 16→8. Kept available for reproducibility. |
End-to-end functional testing on Florence-2 showed that the v22 cut regresses by ~12 pp relative to v12 on a 50-test Magnitude suite — a documented case of distribution drift. The smaller, earlier cut is the recommended production target.
1. Real-site scraping scrape_site.py + parsers/
2. Hand-annotated examples hand_parsed/
3. Synthetic page generation generate_dataset.py
↓
build_dataset.py → HuggingFace Dataset
↓
push.py → Khabner/moondream-data
Schema: image | instruction | point | bbox | element_type | element_label | site_name | url | viewport. point is the normalised centre of bbox; both in [0, 1]. Train / validation / test split: 80 / 10 / 10, seed 42.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m playwright install chromiumScrape a single site:
python scrape_site.py https://crates.io --out scrape_out/cratesBuild the dataset:
# Recommended (v12 cut — better on end-to-end functional testing)
python build_dataset.py --num-pages 900 --paraphrase
# Reproducing the v22 cut
python build_dataset.py --num-pages 900 --paraphrase --profile v22| flag | meaning |
|---|---|
--num-pages N |
number of synthetic pages to add to real-data records |
--paraphrase |
apply paraphrases from paraphrases/done.json to instructions |
--paraphrase-prob FLOAT |
probability of replacing a templated instruction with a paraphrase. Default 0.5 |
--profile {v12,v22} |
synthetic section-pool composition. v12 (default, recommended) — 15 sections matching ec637966 distribution. v22 — 20 sections matching ba10ae2 |
The dataset is written to dataset/hf_dataset/ in HF DatasetDict format.
Push to Hugging Face:
export HF_TOKEN=hf_… # write-scoped token
export HF_DATASET_REPO=user/repo # optional, defaults to Khabner/moondream-data
python push.pyThe build pipeline is deterministic given the seed (42), but section bodies have evolved over time and several pipeline features (paraphrase pipeline, calendar / modal / forum sections, weight rebalances) post-date the ec637966 push — so --profile v12 produces a profile-equivalent dataset, not a bit-exact match. For bit-exact reproduction, pull the immutable HF revision directly:
from huggingface_hub import snapshot_download
snapshot_download("Khabner/moondream-data", revision="ec637966", repo_type="dataset")- Scraper (
scrape_site.py+parsers/): Playwright-based; for each URL, scrolls through the page, injects a JS extractor to enumerate interactive elements (button / link / input / textarea / select / checkbox / radio / menu item), filters by visibility / size / occlusion, and emits a tight text bounding box per element. Per-site parsers are subclasses ofBaseParserregistered via@register("domain.com"). - Synthetic generator (
generate_dataset.py): composes 2–5 sections per page from a profile-switched weighted pool (15 sections inv12, 20 inv22). Each section emits HTML with stable element IDs; Playwright renders the page; bounding boxes are read back from the DOM. - Paraphrase pipeline (
paraphrase_instructions.py): manual-loop tool for adding natural-language paraphrases of"Locate the X labeled 'Y'"instructions. Current pool ≈ 800 pairs, applied with 50 % probability under--paraphrase. Added in commitfe7e943(post-ec637966). - Bbox convention — tight text bounds: text-bearing element types (heading / link / button / label / menu_item) get a TreeWalker-derived tight bbox rather than
getBoundingClientRect(). This favours functional-testing semantics (correct click hits the visible element) over centre-precise pointing on padded buttons. Training median bbox width 3.21 % vs WebClick 7.96 % vs ScreenSpot-V2 12.41 %.
The dataset has been used to fine-tune Moondream-2 (1.86 B) and Florence-2 (270 M / 770 M). End-to-end pass rate on a 50-scenario Magnitude testing suite is the headline production metric:
| variant | dataset cut | Khabner ClickAcc | Magnitude e2e (50 tests) |
|---|---|---|---|
| Moondream-2 + LoRA v12 | ec637966 |
81.6 % | 96 % (48/50, best overall) |
| Moondream-2 base | — | 67.7 % | 88 % (44/50) |
| Florence-2-base + LoRA v1 | ec637966 |
80.8 % | 84 % (42/50) |
| Florence-2-base + LoRA v22 | ba10ae2 |
74.9 % | 70 % (35/50, regression) |
| Florence-2-large + LoRA v1 | ec637966 |
81.7 % | 78 % (39/50) |
| Florence-2-large + LoRA v22 | ba10ae2 |
79.6 % | 91 % (10/11, partial) |
| Florence-2-base (no LoRA) | — | ≤ 2.6 % | 0 % |
The cross-architecture / cross-cut comparison shows that dataset construction philosophy translates into measurable end-to-end regression: Florence-2 trained on the smaller ec637966 cut outperforms Florence-2 trained on ba10ae2 by 14 pp on the 50-test Magnitude suite, despite ba10ae2 having ~17 % more training records. The fine-tuned Florence-2 is integrated into the Magnitude testing framework as the executor (UI-grounding) component.
A complementary ablation (warming Florence-2 on a 5K subset of Wave-UI before fine-tuning on this dataset) regressed WebClick by 5 pp, indicating that a sufficiently domain-specific dataset can be hurt by general-grounding warmup.
For the full per-experiment lab notebook (28 LoRA experiments, ablations, broken-pattern analysis, geometric-precision trade-off), see the bachelor's thesis.
build_dataset.py Assemble real + synthetic records into a HF DatasetDict
generate_dataset.py Synthetic page generator
scrape_site.py Playwright-based scraper driver
parsers/ Per-site parser registry
paraphrase_instructions.py Manual-loop tool for adding instruction paraphrases
push.py Pushes the built dataset to Hugging Face
review.py Visual review helper for scraped pages
scrape_out/ Output of scrape_site.py runs (gitignored)
hand_parsed/ Hand-annotated example pages (gitignored)
paraphrases/done.json Curated instruction paraphrases (~800 pairs)
dataset/ Local build output (gitignored)
README_RU.md Russian-language version
CITATION.cff Citation metadata (GitHub-native)
LICENSE MIT
requirements.txt Pinned dependencies
See CITATION.cff — GitHub renders a "Cite this repository" button automatically.
Built as part of a bachelor's thesis at ITMO University on improving the quality of automated functional testing for web applications via parameter-efficient fine-tuning of compact VLMs.
Models fine-tuned: Moondream-2, Florence-2. Benchmarks: WebClick, ScreenSpot / ScreenSpot-V2. Deployment target: Magnitude.
MIT — see LICENSE.