feat: parallelize data-link chunk uploads

[georgi-seqera]

Summary

Uploads via tw data-links upload now transfer a file's parts concurrently (default 4 at a time, configurable), instead of strictly one-at-a-time.
Along the way this makes the shared progress tracker safe under concurrency, adds a --silent flag to upload and download, shows elapsed time on the progress bar, and reduces per-part allocation overhead.

This is the CLI companion to the Platform expired-credential refresh work — large uploads that previously crawled through parts sequentially now saturate available bandwidth while still recovering from expired signing credentials mid-upload.

Why

• Throughput. A multi-part upload was fully sequential: each 250 MB part waited for the previous one to finish. On high-bandwidth links this leaves most of the pipe idle — wall-clock scaled with the number of parts even when the network could do several at once.
• Ergonomics. Scripted/logged runs want no progress spam (--silent), and long uploads benefit from an elapsed-time readout.

How

Parallel execution. A shared uploadPartsInParallel helper in AbstractProviderUploader runs part tasks on a bounded pool sized min(concurrency, totalParts) (so a small file never spawns more threads than parts). It fails fast — the first failing part cancels the remaining in-flight uploads and propagates the cause (re-asserting the interrupt flag on interruption) so the caller can finalize/abort — and always shuts the pool down.

• AWS / Seqera Compute: each part returns its ETag keyed by part number; the completed-parts list is assembled in ascending order after all parts finish (completion order is nondeterministic).
• Azure: blocks upload concurrently, then finalize sends the ordered block list.
• Google: unchanged — resumable-session uploads are inherently sequential.

Thread-safe progress (the enabler). Global snapshot/restore is replaced with per-part delta accounting: ProgressTracker.newPart() hands out a PartProgress that accumulates only its own bytes and forwards them to the tracker through a small ProgressSink interface. A failed attempt rolls back only that part's bytes, so concurrent parts can't corrupt each other's progress. The terminating newline is latched to print exactly once.

Also included:

• Updates to the upload progress-bar to make to display thread-safe progress.
• --silent on upload and download to suppress progress output.
• Elapsed-time field on the progress bar (plus a guard against a NaN/Infinity ETA on the first tick).
• Body read buffer raised 8 KB → 256 KB to cut allocation churn on large parts (the whole part is already resident in memory, so this costs nothing extra).
• TOWER_UPLOAD_SIZE_PART_BYTES override for the part size (env var, or same-named system property for in-process tests) — lets us exercise multi-part uploads without multi-hundred-MB fixtures.

Usage

# Default: 4 parts in parallel
tw data-links upload -w <workspace> -c <credentials> -n <data-link> ./big-file.bam

# Tune parallelism (higher throughput on fast links)
tw data-links upload -w <workspace> -c <credentials> -n <data-link> --concurrency 8 ./big-file.bam

# Sequential (old behavior)
tw data-links upload ... --concurrency 1 ./big-file.bam

# Silent — no per-file line, no progress bar (scripting / log files)
tw data-links upload ... --silent ./big-file.bam
tw data-links download ... --silent s3://bucket/path

# Progress bar now shows elapsed time alongside ETA:
#  Progress: [==============      ]  62% (4194304/6710886 KBs, ETA: 12.4s, Elapsed: 20s)

Tradeoffs / operational notes

• Memory: each in-flight part buffers up to 250 MB, so peak heap ≈ concurrency × 250 MB (default 4 ≈ 1 GB). This is documented in the --concurrency help text. There's intentionally no hard cap — --concurrency 8 needs ~2 GB, so give the JVM headroom (-Xmx) on constrained machines. --concurrency 0 (or negative) is rejected with a clear error.
• Ordering: finalization is order-correct regardless of completion order (S3 ETag list and Azure block list are both assembled in ascending part-number order).

Compatibility

No changes to command syntax or existing behavior beyond the new opt-in --concurrency (defaults preserve prior semantics aside from running in parallel) and the additive --silent. --concurrency 1 reproduces the previous sequential behavior exactly.

[sabulous]

I bumped into an issue while testing the following. Basically there is a silent truncation when the CLI part size ≠ the backend part size.

The uploader takes the part count from the backend's returned URL list but slices each chunk using its own MULTI_UPLOAD_PART_SIZE_IN_BYTES. If the two ever diverge, it uploads urlCount × cliPartSize bytes and finalizes anyway — producing a truncated object and reporting success.

I reproduced locally a 1 GB upload with TOWER_UPLOAD_SIZE_PART_BYTES=5 MB against a 250 MB-part backend wrote a 25 MB object (all 5 parts identical) and still printed "Successfully uploaded." In production both are 250 MB so it's currently safe, but the new TOWER_UPLOAD_SIZE_PART_BYTES hook makes divergence a one-env-var mistake, and a backend part-size change would corrupt uploads silently. Maybe it's worth deriving the chunk size from contentLength / urlCount (so the CLI always matches the URLs it was given), or at minimum asserting ceil(contentLength / partSize) == urlCount and failing loudly.

[sabulous]

Besides my only comment, I tested it and it works as expected. Approving to unblock

[georgi-seqera]

The uploader takes the part count from the backend's returned URL list but slices each chunk using its own MULTI_UPLOAD_PART_SIZE_IN_BYTES

Good catch! I removed the MULTI_UPLOAD_PART_SIZE_IN_BYTES env var since I mostly needed just a way to configure the part size for unit tests so I can test on tiny files - but exposing it as configurable for production code doesn't make sense if it will always mismatch the hardcoded platform 250MB.

I'm changed it to a property setting instead for test purposes as I don't want to add to the scope of this PR and I put the guard in for an early warning in case Platform's partSize does ever change.