Profile gzip- and zstd-compressed CSV and JSON Lines #892

Activity

1. AndreaBozzo

   added this to the 0.15.0 milestone on Oct 8, 2026
2. AndreaBozzo

   added

   enhancementNew feature or requestNew feature or request

   area: rustPull requests that update rust codePull requests that update rust code

   on Oct 8, 2026
3. github-project-automation

   added this to dataprofon Oct 8, 2026
4. AndreaBozzo commented on Oct 9, 2026

   

   AndreaBozzo

   on Oct 9, 2026

   OwnerAuthor

   More actions

   From #901 (#893): file routes now refuse gzip, zstd, zip and misnamed Parquet with a binary_input error before a text reader sees them. The routes below take the same bytes without a path and are not checked yet; measured on the #901 build with gzip-compressed CSV text:

   route	answer today
   Python dp.profile(data, format="csv") (bytes or BytesIO)	UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1
   dataprof.asyncio.profile_bytes(data, format="csv")	ValueError: CSV parsing failed: ... invalid utf-8 ... Suggestion: Check CSV formatting in the stream data
   dataprof.asyncio.infer_schema_stream(data, format="csv")	RuntimeError: Async schema inference failed: ... Suggestion: This CSV file has inconsistent column counts

   Also unchecked: analyze_csv_from_reader / analyze_json_from_reader, URL profiling (same async reader as profile_bytes), and the engines' own IncrementalProfiler::analyze_file / AdaptiveProfiler::analyze_file, which Profiler checks before calling.

   When this issue adds decompression, those routes either decompress or reuse dataprof_core::binary_input on the first bytes of the stream. The infer_schema_stream row is also a RuntimeError where the file routes raise ValueError: that binding formats the error itself instead of going through analysis_error_to_py.
5. AndreaBozzo commented on Oct 12, 2026

   

   AndreaBozzo

   on Oct 12, 2026

   OwnerAuthor

   More actions

   One more scope point, from Arrow 26.0.0 (GH-50313, "Add guidance about memory bombs", for formats with optional compression): once input can be compressed, its size on disk stops bounding what a single record costs in memory.

   Streaming decompression keeps the decoder's own buffer small, so the codec is not the problem. The record buffers downstream of it are. Neither text reader bounds one record today: the CSV path uses csv 1.4, which has no record-size cap, and the JSONL path reads each line into a growing String (crates/dataprof-json/src/lib.rs, the FileFormat::Jsonl loop). For uncompressed input a 10 GB line needs a 10 GB file. Through gzip or zstd it needs a few megabytes.

   Suggested additions to the acceptance criteria:

   • a high-ratio fixture (a long run of one byte inside a single field, and inside a single JSONL line) whose peak memory is measured, not assumed;
   • a documented limit on decompressed bytes per record or per input, or an explicit statement that none exists, so a caller profiling untrusted drops knows which one they have;
   • if a limit is hit, a typed error naming the limit. A truncated record must not become a plausible row.

   Whether bytes_consumed counts compressed or decompressed bytes (already a question above) decides which of the two numbers such a limit would be expressed in.
6. AndreaBozzo

   moved this to Backlog in dataprofon Oct 25, 2026
7. AndreaBozzo commented on Oct 31, 2026

   

   AndreaBozzo

   on Oct 31, 2026

   Last edited by AndreaBozzo

   OwnerAuthor

   More actions

   Until this lands, the binary-input hint fails for a compressed file without its suffix. For export.csv holding gzip, gunzip -k stops with "unknown suffix -- ignored" (exit 2), and the Windows hint from #1175 derives its output name by stripping .gz, so it would write to the input itself and its exclusive open refuses. Reading the file here would remove that hint for gzip and zstd. If this waits, the hint could name an output for an unsuffixed file (gunzip -c 'export.csv' > 'export.decompressed.csv' on POSIX, and the same output name in the Windows one-liner).