Skip to content

fix(arrow-ipc): return an error for a DictionaryBatch without its data - #11020

Open
PlenoraETL wants to merge 1 commit into
apache:mainfrom
PlenoraETL:fix/dictionary-batch-without-data
Open

fix(arrow-ipc): return an error for a DictionaryBatch without its data#11020
PlenoraETL wants to merge 1 commit into
apache:mainfrom
PlenoraETL:fix/dictionary-batch-without-data

Conversation

@PlenoraETL

Copy link
Copy Markdown

Which issue does this PR close?

Closes #11019.

Rationale for this change

DictionaryBatch.data is optional in the flatbuffer grammar and required by the format. get_dictionary_values opened it with unwrap, so a file that omits it panicked instead of producing an ArrowError.

This is the same class of field, in the same file, that the footer's custom metadata was just fixed for:

let (Some(key), Some(value)) = (kv.key(), kv.value()) else {
    return Err(ArrowError::ParseError(
        "Custom metadata in the IPC footer is missing a key or a value".to_string(),
    ));
};

The patch uses that shape. The proposal is essentially "finish the round".

A panic matters more than usual on this path: under libfuzzer-sys a caught panic still becomes abort() before unwinding, so a fuzz target reading untrusted IPC goes into quarantine rather than reporting a rejected input.

What changes are included in this PR?

One let ... else in get_dictionary_values, plus a regression test.

Measured before and after on main: the fixture panics without the change and returns Parser error: Dictionary batch is missing its data with it. cargo test -p arrow-ipc passes — 132, 7 and 11.

Are these changes tested?

Yes. The fixture is the original 2169-byte input from the fuzzing corpus of plenora-IO-tools, not a reduced one, added under arrow-ipc/test/data/.

Are there any user-facing changes?

A file that used to panic now returns an ArrowError::ParseError. No API change.

@github-actions github-actions Bot added arrow Changes to the arrow crate arrow-ipc labels Sep 8, 2026
PlenoraETL pushed a commit to PlenoraETL/plenora-IO-tools that referenced this pull request Sep 8, 2026
I quattro contributi sono partiti, e la verifica sulle codifiche che li
accompagnava ha trovato un difetto nostro.

## Il contratto, misurato

Che cosa il driver KML accetta **oggi**, provato file per file:

| ingresso | esito |
|---|---|
| UTF-8, con o senza accenti | letto, valore conservato |
| ISO-8859-1 dichiarata e reale | letto, e `citta` con l'accento torna indietro **giusto** |
| dichiarazione che mente su byte validi in entrambe | letto |
| UTF-16LE | rifiutato, «nome di elemento XML non valido» |

Il latin-1 non e' tollerato: e' **supportato**, con i valori conservati. Il
supporto viene dalla feature `encoding` di `quick-xml`, che legge la
dichiarazione XML e transcodifica con `encoding_rs`.

## Il difetto

Quella feature non la chiedevamo. La accendeva `calamine`, che entra dal driver
XLSX, e il nostro manifesto diceva soltanto `quick-xml = "=0.41.0"`.

Percio' il supporto **dipendeva da quali altri crate stavano nello stesso
build**. Nel workspace intero il latin-1 si legge; con `cargo test -p
driver-kml` la feature non c'e' e lo stesso file torna con il byte non
decodificato. Il binario spedito e i test di quel crate dicevano cose diverse
sullo stesso ingresso, e nessuno se ne era accorto.

L'ho trovato perche' la regressione nuova falliva in `-p driver-kml` e passava
dalla CLI, sugli **stessi byte**. Due misure in disaccordo sono un fatto da
spiegare, non un test da aggiustare.

## La correzione

Dichiarare cio' che usiamo:

    quick-xml = { version = "=0.41.0", features = ["encoding"] }

`Cargo.lock` non si muove -- `encoding_rs` era gia' nel grafo -- e le cinque
misure del fuzzing non cadono. Verificato: dopo la modifica `-p driver-kml`
costruisce con `encoding` e il test passa in entrambe le forme di build.

## La regressione

`le_codifiche_accettate_restano_quelle` fissa i quattro casi **misurati**, non
quelli desiderati. Se una riga va cambiata, e' una decisione da dichiarare.

Serve a questo: `quick-xml 0.42.0` toglie la decodifica dal lettore e la sposta
in un `DecodingReader` esplicito. Per **preservare** gli ingressi supportati va
adottato quello; aggiornare senza restringerebbe il supporto, e una restrizione
va proposta prima di introdurla invece che scoperta dopo. Ora c'e' un metro.

## I quattro invii

Aperti con l'identita' `PlenoraETL
<4678670+PlenoraETL@users.noreply.github.com>`, **recuperata e non costruita**:
nel repository non c'e' nessuna configurazione git, e l'indirizzo e' stato
trovato in uso in sette commit su altri due repository dell'account. La storia
esistente non e' stata riscritta.

* ixmilia/dxf-rs#105
* tmontaigu/shapefile-rs#56 e #57
* apache/arrow-rs#11020, con l'issue #11019 aperto per primo come chiede il loro
  CONTRIBUTING

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012PRhkFKzTjuqBjaUXCou96
Comment thread arrow-ipc/src/reader.rs Outdated
Comment on lines +926 to +929
// `DictionaryBatch.data` is optional in the flatbuffer grammar and
// required by the format, so a message that omits it panics here
// rather than producing an error. This mirrors the check the footer
// metadata already got: report it, don't unwrap it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// `DictionaryBatch.data` is optional in the flatbuffer grammar and
// required by the format, so a message that omits it panics here
// rather than producing an error. This mirrors the check the footer
// metadata already got: report it, don't unwrap it.

these LLM comments are really not necessary

Comment thread arrow-ipc/src/reader.rs Outdated
Comment on lines +2021 to +2029
/// A dictionary message that omits its `data` must report an error.
///
/// `DictionaryBatch.data` is optional in the flatbuffer grammar and required
/// by the format, so a file that omits it is structurally decodable but
/// semantically incomplete. `get_dictionary_values` used to open it with
/// `unwrap`, which turned a malformed file into a panic.
///
/// The fixture is the original 2169-byte input from a fuzzing corpus, not a
/// reduced one.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// A dictionary message that omits its `data` must report an error.
///
/// `DictionaryBatch.data` is optional in the flatbuffer grammar and required
/// by the format, so a file that omits it is structurally decodable but
/// semantically incomplete. `get_dictionary_values` used to open it with
/// `unwrap`, which turned a malformed file into a panic.
///
/// The fixture is the original 2169-byte input from a fuzzing corpus, not a
/// reduced one.

same here

Comment thread arrow-ipc/src/reader.rs Outdated
/// The fixture is the original 2169-byte input from a fuzzing corpus, not a
/// reduced one.
#[test]
fn dictionary_batch_without_data_is_an_error_not_a_panic() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fn dictionary_batch_without_data_is_an_error_not_a_panic() {
fn test_invalid_dictionary_batch_without_data() {

Comment thread arrow-ipc/src/reader.rs Outdated
#[test]
fn dictionary_batch_without_data_is_an_error_not_a_panic() {
let bytes: &[u8] = include_bytes!("../test/data/dictionary_batch_without_data.arrow");
let err = crate::reader::FileReader::try_new(std::io::Cursor::new(bytes), None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please move this test down; it shouldnt be above the imports

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we refactor the test to somehow create this data in memory as part of the test? instead of commiting a separate file

`DictionaryBatch.data` is optional in the flatbuffer grammar and required by the
format. `get_dictionary_values` opened it with `unwrap`, so a file that omits it
panicked instead of producing an `ArrowError`.

This is the same class of field, in the same file, that the footer's custom
metadata was just fixed for - `let ... else` returning a `ParseError`. The patch
uses that shape.

A panic matters more than usual here: under `libfuzzer-sys` a caught panic still
becomes `abort()` before unwinding, so a fuzz target reading untrusted IPC goes
into quarantine rather than reporting a rejected input.

The test builds the malformed message in memory with `FlatBufferBuilder`,
following `test_missing_buffer_metadata_error`.

Closes apache#11019
@PlenoraETL
PlenoraETL force-pushed the fix/dictionary-batch-without-data branch from 77bbad9 to 940dd1d Compare September 8, 2026 13:16
@PlenoraETL

Copy link
Copy Markdown
Author

Thanks for the review — all five points addressed, branch force-pushed.

  • Dropped the comment above the fix and the doc comment on the test.
  • Renamed to test_invalid_dictionary_batch_without_data and moved it down, next to test_missing_buffer_metadata_error.
  • Removed the committed .arrow file: the test now builds the malformed DictionaryBatch in memory with FlatBufferBuilder, following the shape of test_missing_buffer_metadata_error. It calls read_dictionary directly, so there's no file and no byte-patching involved.

cargo test -p arrow-ipc passes (132 / 7 / 11) and clippy is clean.

One note in case it matters to you: I had to #[allow(deprecated)] on Field::new_dict, since read_dictionary looks the field up by dict id and that's the constructor that sets one. Happy to build the schema differently if there's a preferred way now that dict ids are on their way out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate arrow-ipc

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow-ipc: reading a DictionaryBatch without its data panics instead of returning an error

2 participants