Repository navigation
Commit source sync state and jobs atomically - #1451
Open
Floating-Y wants to merge 1 commit into
Open
Floating-Y wants to merge 1 commit into
Floating-Y wants to merge 1 commit into
Conversation
Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com>
WaylandYang
requested changes
Oct 26, 2026
WaylandYang
left a comment
Contributor
There was a problem hiding this comment.
Thanks, the fix is right. Please retitle to a sentence stating the behaviour, e.g. "A source's queued status and its sync job commit together, so a failed job insert no longer leaves the source stuck as queued". Please also keep only the failure/rollback regressions (the concurrent and Drive follow-up cases cover unchanged behaviour), drop the trigger and function when the test finishes instead of leaving them on jobs, and write the test comments in Chinese.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
If a
sync_sourceinsertion fails aftermark_queuedcommits, the source staysqueuedwithout a task and ordinary sync requests and the due-source scheduler cannot schedule it again.What changes
jobs::enqueue_with_max_attempts_tx. The payload and three-attempt budget remain the same.Existing orphan states need separate recovery. Read-only candidates are sources marked queued/running with no matching active
sync_sourcejob; check job history and active workers before changing them. This PR does not reset existing states or change checkpoints, sync completion, or retry policy.How it was checked
Validated the exact diff committed as
d9bb2c8on Windows with Rust 1.98.0 and a fresh disposable PostgreSQL 16/pgvector database with all 106 migrations.UTOPIA_TEST_REQUIRE_DB=1andUTOPIA_TEST_REQUIRE_PDFTOTEXT=1were set.queuedwith zero jobs; the two existing-behavior tests passed. All five final source regressions ran and passed, including the real HTTP router.cargo fmt --all --checkandgit diff --check: passed.cargo clippy --workspace --all-targets -- -D warnings: passed.cargo test --locked --workspace --no-fail-fast -- --test-threads=1: exit 0; the harness reported 1,746 OK, zero failures, and six pre-existing ignored tests. This includes the five source regressions and the three existing worker-notification regressions.pnpm install --frozen-lockfile --offline,pnpm build, andpnpm test --maxWorkers=2: passed; 327 frontend tests across 48 files.node --test scripts/bench/lib.test.mjs: 15 passed, none skipped.Harness OK includes optional early-return cases: eight MySQL/ClickHouse/Trino live tests were not run because their service URLs were not configured. Optional real Drive/Notion/S3/Azure/GCS/WebDAV probes and the gated large benchmark are also not claimed as executed integration coverage.
The full run used standalone Poppler 26.09.0 with its CJK data and process-only localhost proxy bypass. One harness thread avoided unrelated wall-clock contention; concurrent source requests and embedding batches still ran inside their tests. Both previously failing PDF GBK and embedding timing tests passed in the final full run, with their source and assertions unchanged.
Before review
git commit -s).crates/utopia-store/was tested withUTOPIA_DATABASE_URLset andUTOPIA_TEST_REQUIRE_DB=1.