writestat to .sav/.por emits incorrect "A{n}" format width for short string columns

Summary

When writing an SPSS file, writestat sets the variable format to "A9" for any string column with content ≤ 8 chars, regardless of the actual max content width. The format "A{n}" is the canonical place SPSS stores the string-column max width (since storage_width is padded to 8-byte blocks for short strings), so emitting the wrong n loses that information on roundtrip.

Reproducer

using ReadStatTables, DataFrames, Tables

df = DataFrame(
    s1  = ["a", "b", "c"],
    s5  = ["hello", "world", "foo  "],
    s24 = ["a quite long string here", "another long-ish string", "x"],
)

writestat("/tmp/widths.sav", df)
d = readstat("/tmp/widths.sav")
[colmetadata(d, c)[:format] for c in Tables.columnnames(d)]
# => ["A9", "A9", "A24"]

Expected

["A1", "A5", "A24"]

i.e. "A{maximum(length, col)}" per string column. Canonical writers (SPSS itself, R haven, Python pyreadstat) all behave this way.

Why it matters

Loses per-column max-width info on roundtrip through ReadStatTables.jl.
Blocks any future change that uses the format string to pick a tight InlineString{N} at read time (since the value emitted by this package would over-allocate vs. the same data written by other tools).
Causes the format field of column metadata to disagree across writers for the same data.

@junyuan-chen

@junyuan-chen, based on our discussion in #71 (comment), this is the issue to be solve, right?

These are related issues to be addressed. Thank you for bringing them up.

After searching a bit, I thought the n in the A{n} format refers to the display width instead of storage width of the string variables? There is something weird about writing the storage width to SPSS files. Somehow, the generated SPSS files are not following the storage width requested via libreadstat API. This could be a bug in libreadstat if I didn't get something wrong on my side.

After searching a bit, I thought the n in the A{n} format refers to the display width instead of storage width of the string variables?

I tried to run some examples

A .sav file stores these as two different records, so they can hold different numbers at the same time:
Variable record (rec_type = 2) — one per variable. Carries the declared string length (type field) and the print/write format packed into an int32 as (fmt_type << 16) | (width << 8) | decimals. fmt_type = 1 is A. This is the A{n}.
Extension record rec_type = 7, subtype 11 ("variable display parameters") — a separate flat array of (measure, width, alignment) triples, one per variable. That width is the display/column width SPSS uses in the Data Editor.

Example 1 — pyreadstat's file: A{n} varies, display width is constant

py.sav
  var    rec2.type  print fmt  subtype11 display
  S1             1         A1                  8
  S5             5         A5                  8
  S24           24        A24                  8
  S255         255       A255                  8

Same file, same columns. The A{n} tracks the declared string length exactly (1, 5, 24, 255) while the display record is a flat 8 for every one of them. If A{n} were the display width, S24 would read A8. It reads A24. haven's file is byte-for-byte equivalent on these fields.

Example 2 — set them to deliberately different values

I wrote one column with format = "A24" and display_width = 12:

exp_indep.sav
  var    rec2.type  print fmt  subtype11 display
  S24           24        A24                 12

Both land, independently, and read back as format=A24, display_width=12. Two fields, two storage slots.

Example 3 — why ReadStatTables makes them look coupled

ReadStatTables never calls variable_set_format for string columns (writer.jl:119-120 skips it when the format string is empty), so libreadstat hits its fallback in readstat_spss.c:237-245:

spss_format->type = SPSS_FORMAT_TYPE_A;
if (r_variable->display_width)   spss_format->width = r_variable->display_width;
else if (r_variable->user_width) spss_format->width = r_variable->user_width;
else                             spss_format->width = r_variable->storage_width;

Display width is checked first, and it's non-zero because writestat.jl:230 sets it to clamp(w, 9, 60). (I think this may be the issue) So the format ends up copying it:

jl_v035.sav
  var    rec2.type  print fmt  subtype11 display
  s1             1         A9                  9   <- true width 1
  s5             5         A9                  9   <- true width 5
  s24           24        A24                 24
  s255         255        A60                 60   <- true width 255

The two columns are identical only because one was derived from the other. The moment a format string is supplied (Example 2, and the whole of Examples 1), the if (r_variable->format[0]) branch at line 259 parses it and overrides the fallback — which is why the fix is just to set "A$(width)" and leave display_width alone.

@RagavRajan

@RagavRajan I think I got what you mean and made a PR for that. Now by default the max length of the strings will be found and used to set the n in the A{n} format string.

Do you want to take a look at #76 before I merge it?

Navigation Menu

Repository navigation

writestat to .sav/.por emits incorrect "A{n}" format width for short string columns #73

Summary

Reproducer

Expected

Why it matters

Example 1 — pyreadstat's file: A{n} varies, display width is constant

Example 2 — set them to deliberately different values

Example 3 — why ReadStatTables makes them look coupled

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

writestat to .sav/.por emits incorrect "A{n}" format width for short string columns #73

Description

Summary

Reproducer

Expected

Why it matters

Activity

RagavRajan commented on Jun 11, 2026

junyuan-chen commented on Jun 11, 2026

junyuan-chen commented on Aug 10, 2026

RagavRajan commented on Aug 11, 2026

Example 1 — pyreadstat's file: A{n} varies, display width is constant

Example 2 — set them to deliberately different values

Example 3 — why ReadStatTables makes them look coupled

junyuan-chen commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions