Repository navigation

BQ to SquashFS download pipeline with test suite - #79

Open

mohamedelabbas1996 wants to merge 26 commits into

mainfrom

feature/bq-squashfs-pipeline

Commits on May 1, 2026
- Add BQ/SquashFS pipeline scripts and documentation
  
  mohamedelabbas1996
  and
  claude
  committed
  98dc4e2
Commits on Jun 3, 2026
- chore: ignore image data dirs, logs, stats, and scratch outputs
  mohamedelabbas1996
  committed
  df9824c
- feat(download): retry backoff, session pooling, inline MERGE, mksquashfs error handling
  
  mohamedelabbas1996
  and
  claude
  committed
  bb68b8a
- test: add BQ test table creation script for download_images.py
  
  mohamedelabbas1996
  and
  claude
  committed
  f60f004
- test(download): failure scenario tests for download_images.py
  
  mohamedelabbas1996
  and
  claude
  committed
  06751d7
- test(download): add MOD split correctness tests for --num-jobs/--task-id
  
  mohamedelabbas1996
  and
  claude
  committed
  45329ca
Commits on Jun 4, 2026
- test: add shared conftest.py with fixtures for all pipeline stage tests
  
  mohamedelabbas1996
  and
  claude
  committed
  386f3c4
- docs(download): improve --help descriptions for all arguments
  
  mohamedelabbas1996
  and
  claude
  committed
  f99c58e
- test(download): add multi-task distribution and merge tests
  
  mohamedelabbas1996
  and
  claude
  committed
  c029785
- feat(pack): add stream_chunks_to_tar.py with 18 tests
  
  mohamedelabbas1996
  and
  claude
  committed
  ce26ff3
- fix(pack): handle BrokenPipeError cleanly when sqfstar is OOM killed
  
  mohamedelabbas1996
  and
  claude
  committed
  3a03014
- feat(pack): harden stream_chunks_to_tar against all real failure modes
  
  mohamedelabbas1996
  and
  claude
  committed
  d7e19f2
- feat(pack): rename stream_chunks_to_tar → merge_sqfs_chunks, full hardening
  
  mohamedelabbas1996
  and
  claude
  committed
  b704aee
- test(pack): rename and update tests for merge_sqfs_chunks.py
  
  mohamedelabbas1996
  and
  claude
  committed
  1e28438
- feat(download): merge failed status into training_images
  
  mohamedelabbas1996
  and
  claude
  committed
  0126297
- feat(pack): add job_bq_pack_per_task.sh with verify-then-delete
  
  mohamedelabbas1996
  and
  claude
  committed
  d51fef7
- Add BQ training_images dedup clean script
  
  mohamedelabbas1996
  and
  claude
  committed
  506bb3c
- Add test suite for clean.py (68 tests)
  
  mohamedelabbas1996
  and
  claude
  committed
  8358966
Commits on Jun 5, 2026
- feat(download): add --dataset flag and NULL fetch_status support
  
  mohamedelabbas1996
  and
  claude
  committed
  5a7e437
- feat(jobs): point bq_download job to global_all_leps_2605
  
  mohamedelabbas1996
  and
  claude
  committed
  f7cfc5e
- fix(download): retry BQ MERGE on serialization conflicts
  
  mohamedelabbas1996
  and
  claude
  committed
  979c3de
- feat(test-tables): add --dataset flag, random sampling, _test_ naming
  
  mohamedelabbas1996
  and
  claude
  committed
  f127c47
- feat(jobs): replace bq_download with staged download→merge→upload pipeline
  
  mohamedelabbas1996
  and
  claude
  committed
  f80482f
Commits on Jun 11, 2026
- chore: remove job_bq_pack_per_task.sh from tracking (superseded by job_bq_download.sh)
  
  mohamedelabbas1996
  and
  claude
  committed
  e738458
- chore: add google-cloud-bigquery to dev dependencies
  
  mohamedelabbas1996
  and
  claude
  committed
  b7d3368
- style: apply black and isort formatting
  
  mohamedelabbas1996
  and
  claude
  committed
  caa04de

Navigation Menu

Repository navigation

BQ to SquashFS download pipeline with test suite - #79

BQ to SquashFS download pipeline with test suite#79
mohamedelabbas1996 wants to merge 26 commits into
mainfrom
feature/bq-squashfs-pipeline

Commits on May 1, 2026

Add BQ/SquashFS pipeline scripts and documentation

Commits on Jun 3, 2026

chore: ignore image data dirs, logs, stats, and scratch outputs

feat(download): retry backoff, session pooling, inline MERGE, mksquashfs error handling

test: add BQ test table creation script for download_images.py

test(download): failure scenario tests for download_images.py

test(download): add MOD split correctness tests for --num-jobs/--task-id

Commits on Jun 4, 2026

test: add shared conftest.py with fixtures for all pipeline stage tests

docs(download): improve --help descriptions for all arguments

test(download): add multi-task distribution and merge tests

feat(pack): add stream_chunks_to_tar.py with 18 tests

fix(pack): handle BrokenPipeError cleanly when sqfstar is OOM killed

feat(pack): harden stream_chunks_to_tar against all real failure modes

feat(pack): rename stream_chunks_to_tar → merge_sqfs_chunks, full hardening

test(pack): rename and update tests for merge_sqfs_chunks.py

feat(download): merge failed status into training_images

feat(pack): add job_bq_pack_per_task.sh with verify-then-delete

Add BQ training_images dedup clean script

Add test suite for clean.py (68 tests)

Commits on Jun 5, 2026

feat(download): add --dataset flag and NULL fetch_status support

feat(jobs): point bq_download job to global_all_leps_2605

fix(download): retry BQ MERGE on serialization conflicts

feat(test-tables): add --dataset flag, random sampling, _test_ naming

feat(jobs): replace bq_download with staged download→merge→upload pipeline

Commits on Jun 11, 2026

chore: remove job_bq_pack_per_task.sh from tracking (superseded by job_bq_download.sh)

chore: add google-cloud-bigquery to dev dependencies

style: apply black and isort formatting