fix(arrow-ipc): return an error for a DictionaryBatch without its data - #11020
fix(arrow-ipc): return an error for a DictionaryBatch without its data#11020PlenoraETL wants to merge 1 commit into
Conversation
I quattro contributi sono partiti, e la verifica sulle codifiche che li
accompagnava ha trovato un difetto nostro.
## Il contratto, misurato
Che cosa il driver KML accetta **oggi**, provato file per file:
| ingresso | esito |
|---|---|
| UTF-8, con o senza accenti | letto, valore conservato |
| ISO-8859-1 dichiarata e reale | letto, e `citta` con l'accento torna indietro **giusto** |
| dichiarazione che mente su byte validi in entrambe | letto |
| UTF-16LE | rifiutato, «nome di elemento XML non valido» |
Il latin-1 non e' tollerato: e' **supportato**, con i valori conservati. Il
supporto viene dalla feature `encoding` di `quick-xml`, che legge la
dichiarazione XML e transcodifica con `encoding_rs`.
## Il difetto
Quella feature non la chiedevamo. La accendeva `calamine`, che entra dal driver
XLSX, e il nostro manifesto diceva soltanto `quick-xml = "=0.41.0"`.
Percio' il supporto **dipendeva da quali altri crate stavano nello stesso
build**. Nel workspace intero il latin-1 si legge; con `cargo test -p
driver-kml` la feature non c'e' e lo stesso file torna con il byte non
decodificato. Il binario spedito e i test di quel crate dicevano cose diverse
sullo stesso ingresso, e nessuno se ne era accorto.
L'ho trovato perche' la regressione nuova falliva in `-p driver-kml` e passava
dalla CLI, sugli **stessi byte**. Due misure in disaccordo sono un fatto da
spiegare, non un test da aggiustare.
## La correzione
Dichiarare cio' che usiamo:
quick-xml = { version = "=0.41.0", features = ["encoding"] }
`Cargo.lock` non si muove -- `encoding_rs` era gia' nel grafo -- e le cinque
misure del fuzzing non cadono. Verificato: dopo la modifica `-p driver-kml`
costruisce con `encoding` e il test passa in entrambe le forme di build.
## La regressione
`le_codifiche_accettate_restano_quelle` fissa i quattro casi **misurati**, non
quelli desiderati. Se una riga va cambiata, e' una decisione da dichiarare.
Serve a questo: `quick-xml 0.42.0` toglie la decodifica dal lettore e la sposta
in un `DecodingReader` esplicito. Per **preservare** gli ingressi supportati va
adottato quello; aggiornare senza restringerebbe il supporto, e una restrizione
va proposta prima di introdurla invece che scoperta dopo. Ora c'e' un metro.
## I quattro invii
Aperti con l'identita' `PlenoraETL
<4678670+PlenoraETL@users.noreply.github.com>`, **recuperata e non costruita**:
nel repository non c'e' nessuna configurazione git, e l'indirizzo e' stato
trovato in uso in sette commit su altri due repository dell'account. La storia
esistente non e' stata riscritta.
* ixmilia/dxf-rs#105
* tmontaigu/shapefile-rs#56 e #57
* apache/arrow-rs#11020, con l'issue #11019 aperto per primo come chiede il loro
CONTRIBUTING
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012PRhkFKzTjuqBjaUXCou96
| // `DictionaryBatch.data` is optional in the flatbuffer grammar and | ||
| // required by the format, so a message that omits it panics here | ||
| // rather than producing an error. This mirrors the check the footer | ||
| // metadata already got: report it, don't unwrap it. |
There was a problem hiding this comment.
| // `DictionaryBatch.data` is optional in the flatbuffer grammar and | |
| // required by the format, so a message that omits it panics here | |
| // rather than producing an error. This mirrors the check the footer | |
| // metadata already got: report it, don't unwrap it. |
these LLM comments are really not necessary
| /// A dictionary message that omits its `data` must report an error. | ||
| /// | ||
| /// `DictionaryBatch.data` is optional in the flatbuffer grammar and required | ||
| /// by the format, so a file that omits it is structurally decodable but | ||
| /// semantically incomplete. `get_dictionary_values` used to open it with | ||
| /// `unwrap`, which turned a malformed file into a panic. | ||
| /// | ||
| /// The fixture is the original 2169-byte input from a fuzzing corpus, not a | ||
| /// reduced one. |
There was a problem hiding this comment.
| /// A dictionary message that omits its `data` must report an error. | |
| /// | |
| /// `DictionaryBatch.data` is optional in the flatbuffer grammar and required | |
| /// by the format, so a file that omits it is structurally decodable but | |
| /// semantically incomplete. `get_dictionary_values` used to open it with | |
| /// `unwrap`, which turned a malformed file into a panic. | |
| /// | |
| /// The fixture is the original 2169-byte input from a fuzzing corpus, not a | |
| /// reduced one. |
same here
| /// The fixture is the original 2169-byte input from a fuzzing corpus, not a | ||
| /// reduced one. | ||
| #[test] | ||
| fn dictionary_batch_without_data_is_an_error_not_a_panic() { |
There was a problem hiding this comment.
| fn dictionary_batch_without_data_is_an_error_not_a_panic() { | |
| fn test_invalid_dictionary_batch_without_data() { |
| #[test] | ||
| fn dictionary_batch_without_data_is_an_error_not_a_panic() { | ||
| let bytes: &[u8] = include_bytes!("../test/data/dictionary_batch_without_data.arrow"); | ||
| let err = crate::reader::FileReader::try_new(std::io::Cursor::new(bytes), None) |
There was a problem hiding this comment.
please move this test down; it shouldnt be above the imports
There was a problem hiding this comment.
could we refactor the test to somehow create this data in memory as part of the test? instead of commiting a separate file
`DictionaryBatch.data` is optional in the flatbuffer grammar and required by the format. `get_dictionary_values` opened it with `unwrap`, so a file that omits it panicked instead of producing an `ArrowError`. This is the same class of field, in the same file, that the footer's custom metadata was just fixed for - `let ... else` returning a `ParseError`. The patch uses that shape. A panic matters more than usual here: under `libfuzzer-sys` a caught panic still becomes `abort()` before unwinding, so a fuzz target reading untrusted IPC goes into quarantine rather than reporting a rejected input. The test builds the malformed message in memory with `FlatBufferBuilder`, following `test_missing_buffer_metadata_error`. Closes apache#11019
77bbad9 to
940dd1d
Compare
|
Thanks for the review — all five points addressed, branch force-pushed.
One note in case it matters to you: I had to |
Which issue does this PR close?
Closes #11019.
Rationale for this change
DictionaryBatch.datais optional in the flatbuffer grammar and required by the format.get_dictionary_valuesopened it withunwrap, so a file that omits it panicked instead of producing anArrowError.This is the same class of field, in the same file, that the footer's custom metadata was just fixed for:
The patch uses that shape. The proposal is essentially "finish the round".
A panic matters more than usual on this path: under
libfuzzer-sysa caught panic still becomesabort()before unwinding, so a fuzz target reading untrusted IPC goes into quarantine rather than reporting a rejected input.What changes are included in this PR?
One
let ... elseinget_dictionary_values, plus a regression test.Measured before and after on
main: the fixture panics without the change and returnsParser error: Dictionary batch is missing its datawith it.cargo test -p arrow-ipcpasses — 132, 7 and 11.Are these changes tested?
Yes. The fixture is the original 2169-byte input from the fuzzing corpus of plenora-IO-tools, not a reduced one, added under
arrow-ipc/test/data/.Are there any user-facing changes?
A file that used to panic now returns an
ArrowError::ParseError. No API change.