Skip to content

Continue a full XLSX sheet on another sheet instead of dropping its rows - #362

Open
dchaudhari7177 wants to merge 1 commit into
RyanDFIR:mainfrom
dchaudhari7177:fix/359-xlsx-row-limit
Open

dchaudhari7177 wants to merge 1 commit into
RyanDFIR:mainfrom
dchaudhari7177:fix/359-xlsx-row-limit

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #359.

What changed

This takes the preferred fix from the issue: a sheet that fills up continues on another sheet, so nothing is dropped.

RolloverWorksheet in analysis.py wraps a worksheet and takes the same calls the writers already make: write, write_string, write_number, merge_range, set_column, freeze_panes and autofilter. Those seven are every method generate_excel calls on a sheet. The writers keep one running row number for a whole collection, and the wrapper maps it onto the sheet with room for it:

  • Rows 0 and 1 (title bar and column headers), column widths and frozen panes are recorded and applied to every sheet, including continuations created later.
  • When a data row lands past the last row, a new sheet named by get_unique_sheet_name (Timeline (2), Storage (2), …) gets the recorded setup replayed. The row then goes there and a warning is logged: XLSX sheet "Storage" reached 1,048,576 rows; continuing on "Storage (2)".
  • autofilter gives each full sheet a filter down to its last row, and the last sheet keeps the writer's row, so a single-sheet workbook gets exactly the filter it got before.
  • _tab_order now strips a trailing (N) before matching, so continuations sort next to the sheet they continue instead of falling to the end.

All eight sheets use it, not only Timeline and Storage. Every one of them is filled from a collection with a running row number and uses the same two header rows, so the change is one line per sheet. A merged range in the data has to fit on one sheet, which holds for the single-row group headers on Service Workers, Sessions and Extensions.

I left the per-store IndexedDB cap from #348 as it is. It bounds memory and parse time as well as the sheet, so removing it deserves its own decision.

Checks

At the boundary, stock xlsxwriter 3.2.9 behaves as the issue describes:

stock write at row 1048575 -> 0 ; at row 1048576 -> -1 ; sheets: ['Storage']

tests/test_xlsx_row_limit.py:

  • TestXlsxRowLimit patches XLSX_MAX_ROWS to 6 (two header rows plus four data rows) and runs generate_excel with 10 Timeline cookies and 10 Local Storage records. It checks that Timeline, Timeline (2), Timeline (3) and StorageStorage (3) sit together in tab order, that every record appears exactly once and in order across the three sheets, that no sheet exceeds the limit, and that continuations repeat the header row, title bar, frozen panes and column width. Each sheet's autofilter must be A2:…6, A2:…6, A2:…5, and four rollovers must be logged.
  • TestXlsxRowLimitBoundary uses the real limit. Writing at zero-based row 1,048,575 returns 0 on the first sheet, and 1,048,576 returns 0 and lands on Storage (2) under the repeated header. Because the wrapper maps row numbers directly, this needs only three writes, not a million.
$ python -m pytest tests/test_xlsx_row_limit.py
# analysis.py at main: 1 failed, 6 errors (there is no XLSX_MAX_ROWS or RolloverWorksheet to patch)
# with this change:    7 passed, 14 subtests passed
$ python -m pytest
285 passed, 2 skipped, 94 subtests passed

The existing XLSX tests (test_xlsx_none_safety, and TestXlsxColumns in test_hashing, which reads column positions and the autofilter range) pass unchanged. Wrapped sheets are still NoneSafeWorksheets, because they come from workbook.add_worksheet with the same worksheet_class.

xlsxwriter returns -1 for a write past row 1,048,576 and leaves the cell
out of the file without raising, so a Timeline or Storage sheet filled
from a large run silently lost everything past the limit.

Every XLSX sheet now goes through RolloverWorksheet, which maps the
writer's running row number onto the sheet with room for it and, when
one fills, adds "Name (2)" with the same title bar, headers, column
widths and frozen panes. Each sheet gets its own autofilter, the
rollover is logged, and tab ordering keeps continuations next to the
sheet they continue.

Closes RyanDFIR#359
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

XLSX output silently drops rows past Excel's 1,048,576-row sheet limit

1 participant