[billfreeman44]

What

csgo-ppp is about to write player_vector and player_status in a compact format: integers as int8, int16 or int32, floats as float32, place_name as a dictionary, and no pandas metadata. Files already written keep int64, float64 and strings. Older files also hold some flags (burst_mode and is_silenced, and flags in player_death, other_death and bomb_defuse) as 0 and 1 in int64 columns declared Int64, where newer ones hold bool. This PR makes dsdk's tomes handle all of these, alone or mixed.

• Mixed tomes are narrowed, not widened. Where some matches or pages hold a column as int64 or float64 and others narrower, build_basic_tomes and the loaders (get_dataframe with pandas or polars, and scan) cast the wide values to the narrow type. Integers go to the widest narrow type seen, float64 to float32, and a float64 that holds whole numbers (player_id_fixed in some csds versions) to the integer type. Before this, pyarrow and pd.concat widened everything back to int64 and float64, which gives up the memory the compact files save.
• Flags are narrowed to bool. Where some files hold a column as bool and others as integers, the integers are cast to bool: 0 to False, 1 to True. Before this, a page holding both couldn't be written (pd.concat made an object column of bools and ints, which pyarrow refused), so a tome spanning those csds versions failed to build, on master too.
• Every cast is checked. A value that doesn't fit raises NarrowingError, which names the column, the match (on load, the page and row) and the value. That covers overflow, underflow, a float that isn't a whole number (or is NaN or infinite) for an integer column, a finite float beyond float32's range, and a flag other than 0 or 1.
• current_ammo stays int64 in a mixed tome, on every page (the compact files' int16 is widened to it, exactly). Old files hold 4294967295 there for an empty magazine, which int16 can't hold, and dsdk doesn't change values.
• Values don't change, except that a float64 narrowed to float32 is rounded to the nearest float32. The raw floats read from the demo convert exactly; the derived ones move by at most 2^-24 relative, as they do in the compact files.
• Missing values stay missing. A missing column (player_controller_id is absent from some 2023 files) is filled with nulls at the narrow type: pandas' nullable Int8, not float64. A flag narrowed from a column declared Int64 is pandas' nullable boolean.
• Dictionary columns are decoded as each match is read, so a page holding compact player_status is built with pyarrow instead of falling back to pandas. pandas reads place_name as strings, which is what pd.concat gives categories that differ between matches.
• widen=True, opt-in, on get_dataframe and scan returns 64-bit integers and float64 for analysis code; flags stay bool. Narrow integers wrap in element-wise arithmetic: an int16 money * 5 gives 14,464 for 16,000.
• Tomes of old files only that built before build and load exactly as before (checked below).

Where narrowing happens

• Builder. Each page narrows its tables before joining them, on the pyarrow path and in the pandas fallback. When the tome is finished, every page not yet at the tome's types is read back, cast to them and written again. That covers a page written before the first compact match turned up, and one narrowed to int8 before a later match brought int16. It also covers a page holding a value int8 or int16 can't hold: that page keeps the column wide until the end, since a later match may bring a wider type. So every page holds the tome's types, and a value that doesn't fit them stops the build instead of every later load.
• Loaders. A column whose type differs across pages this way is read page by page with pyarrow and cast (pandas). For polars, the differing pages' values are checked with pyarrow when the scan is created, then cast lazily.
• make_tome is unchanged. It joins a page's frames with pd.concat, so a page that mixes old and compact matches is widened there, and a page mixing bool with Int64 flags still fails there. The loaders still narrow across its pages.

docs/tome-data-model.md has a new section with the rules. CHANGELOG: v3.3.1 was tagged with the entries of #39 to #41 still under "Unreleased", so this titles them 3.3.1 and starts a new Unreleased section.

How it was tested

• make test: 306 passed on pandas 3.0.6 with polars, 306 on pandas 2.3.3, and 260 passed and 46 skipped without polars. make lint: 10.00/10, black clean.

• New tests in narrowing_test.py:

  • compact player_vector and player_status matches derived from dsdk's own fixtures, with the old fixtures' Int64 0/1 flags against compact bool flags;
  • a mixed tome (an old page, a mixed page, a compact page) built and read with pandas and polars;
  • overflow, underflow, float32 overflow and a non-whole player_id_fixed each stop the build, both when a page is narrowed as it is built and when it is narrowed at the end;
  • the widest narrow type wins: an int8 match and an old one on a page, an int16 match later, gives int16 on every page (from the first Copilot review);
  • flags: 0/1 Int64 with bool narrows to bool in pandas and polars, a missing value stays missing, a 2 stops the build and fails a load, and tomes whose flags are all Int64 or all bool load as before;
  • current_ammo's 4294967295 kept, and a missing column;
  • pages built with pandas narrowed the same as pages built with pyarrow;
  • narrowing and its errors on load, and widen.

• Existing tests changed on purpose:

  • Categories that differ now build with pyarrow (still equal to make_tome).
  • The loader cases "int32 then int64" and "bool then int" moved to the narrowing tests: the first now loads as int32, and the second, which holds a 2, now raises.
  • The pandas-fallback builder test forces the fallback, since bool against int64 no longer triggers it.

• The same output as master. I built tomes with master and with this branch, then compared every page file (data and metadata) and every column as loaded by master's loaders and by this branch's (pandas and polars, on both builds' pages):

  • dsdk's 3 fixtures, all 31 channels, with 256 MB and 0.05 MB pages: identical.
  • 14 real old-format matches (4 each from the 2023, 2024 and 2026 archive, and 2 written by today's csgo-ppp), all 43 channels: 39 identical. The other 4 (player_status, player_death, other_death, bomb_defuse) fail on master because of the bool and Int64 flags; they build here, and are checked against their sources below.

• Real tomes, checked against their sources. The 14 old matches, plus 4 compact copies where marked "mixed". Each match's rows in the tome equal its source file cast to the tome's type, in pandas and in polars, for every column (0 mismatches in every row of the table):

  tome	matches	pages	rows	notes
  player_vector, 256 MB pages	mixed, 18	10 (1 mixed, 9 rewritten at the end)	13.35M	every page at the compact types, with current_ammo int64 on every page (from the second Copilot review)
  player_status, 256 MB pages	mixed, 18	10 (1 mixed, 8 narrowed at the end)	13.35M	failed before; burst_mode and is_silenced are boolean, 9.33M values each narrowed from 0/1
  player_status, one match a page	mixed, 18	18 (14 narrowed at the end)	13.35M	player_controller_id is Int8, with nulls for the 2 matches that lack it
  player_status, 256 MB pages	old, 14	9 (7 narrowed at the end)	11.50M	failed on master; only the flags narrow, the rest stays int64/float64
  player_death	old 14 / mixed 18	1	1,946 / 2,225	failed on master; 4 flags narrowed, 1,584 values each
  other_death	old 14 / mixed 18	1	433 / 491	failed on master; 3 flags narrowed, 414 values each
  bomb_defuse	old 14 / mixed 18	1	40 / 45	failed on master; has_kit narrowed, 35 values

  Float32 rounding shows up only in player_vector's derived columns, for example speed_2d: 6.61M of the 11.5M old values, by at most 5.96e-8 relative. No raw float changed. widen=True gave the same values at int64, Int64 and float64, and left flags bool.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Page-local narrowing can reject values that fit the eventual tome-wide target and leave pages at inconsistent widths.

Review effort: Balanced
Findings: 1

Open (1)

• Final pass skips narrowed pages, causing incorrect tome-wide type · New

What changed in this PR

Adds checked narrowing for tomes mixing legacy and compact CSDS player tables, plus optional 64-bit widening for analysis.

Changes:

• Narrows mixed schemas with overflow validation across builders and loaders.
• Adds widen=True to pandas and polars APIs.
• Documents and tests compact types, nullable columns, and dictionary decoding.

File	Description
pureskillgg_dsdk/​tome/​page_frames.py	Narrows and widens loaded pages.
pureskillgg_dsdk/​tome/​narrowing.py	Implements checked type conversions.
pureskillgg_dsdk/​tome/​narrowing_test.py	Tests mixed-schema behavior and errors.
pureskillgg_dsdk/​tome/​loader.py	Exposes loader widening.
pureskillgg_dsdk/​tome/​loader_test.py	Updates schema-drift cases.
pureskillgg_dsdk/​tome/​curator.py	Passes widening through curator APIs.
pureskillgg_dsdk/​tome/​constants.py	Centralizes the match-key column.
pureskillgg_dsdk/​tome/​builder.py	Narrows built pages and decodes dictionaries.
pureskillgg_dsdk/​tome/​builder_test.py	Tests updated builder behavior.
pureskillgg_dsdk/​tome/​__init__.py	Exports NarrowingError.
docs/​tome-data-model.md	Documents narrowing rules.
CHANGELOG.md	Records the public behavior changes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Compact-only pages in mixed tomes can retain current_ammo as int16, violating the documented tome-wide int64 invariant.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Final pass skips narrowed pages, causing incorrect tome-wide type

Previously missed (1)

In code that hasn't changed since last review

Promote current_ammo to int64 across mixed-width pages

pureskillgg_dsdk/​tome/​narrowing.py:80

KEEP_WIDE currently removes current_ammo from every plan, so the final page-rewrite pass cannot make it wide. In the multi-page mixed tome exercised above, the compact-only page therefore remains int16 while old pages are int64 (the schema assertion in test_a_mixed_tome_holds_the_compact_types explicitly skips this column). The combined loader happens to promote it, but this violates the stated invariant that every built page uses the tome-wide types and forces future loads down the schema-drift path. Please plan an int64 promotion for this column when wide and narrow sources coexist, while continuing to prohibit narrowing its old values.

[billfreeman44]

On the second Copilot review (the one point, "Promote current_ammo to int64 across mixed-width pages"): agreed, fixed in e2753d8.

Where current_ammo is int64 in some tables or pages and a narrower integer in others, the plan now targets int64 instead of leaving the column out. Narrow pages are widened, which is exact; old values are still never narrowed. The builder's end-of-build pass therefore rewrites a compact-only page to int64 too, so every page holds the same type, and the loaders cast likewise for tomes whose pages differ.

test_a_mixed_tome_holds_the_compact_types now asserts current_ammo is int64 on every page, including the compact one, and test_narrow_target covers the rule. Full suite: 306 passed on pandas 3 and 2.3.3, and 260 passed with 46 skipped without polars. On the real data, the mixed player_vector tome (18 matches, 13.35M rows) now has int64 current_ammo on all 10 pages, with 0 mismatches against the sources; fixture tomes are still identical to master.