mstackhouse · Sep 4, 2026 · Sep 4, 2026
diff --git a/DESCRIPTION b/DESCRIPTION
@@ -1,7 +1,7 @@
 Package: datasetjson
 Type: Package
 Title: Read and Write CDISC Dataset JSON Files
-Version: 0.3.0
+Version: 0.4.0
 Authors@R: c(
     person(given = "Mike",
            family = "Stackhouse",

diff --git a/NEWS.md b/NEWS.md
@@ -1,9 +1,63 @@
 # datasetjson 0.4.0
 
-- Updated `read_dataset_json()`: `format.sas` is now set from the `displayFormat` attribute on columns (#87)=
-- `decimals_as_floats` parameter removed from `read_dataset_json()`. Variables with `dataType = "decimal"` and `targetDataType = "decimal"` are now always converted to numeric on read, per the Dataset JSON v1.1 specification.
-- Fixed a bug in the `converting_files` vignette where `extract_xpt_meta()` could return `-Inf` for the `length` field of empty or all-NA string columns; minimum length is now 1 (#71)
-- Fixed a bug where `write_dataset_json()` with `float_as_decimals = TRUE` would write `NA` values as padded strings (e.g., `"    NA"`) instead of JSON `null` (#76)
+## JSON handling moved into the package
+
+Reading and writing are now handled by the `yyjson` C library, bundled with the
+package, rather than by an R JSON package. The practical effects:
+
+- **Numbers survive a write and read unchanged.** Previously a float below
+  roughly 5e-7 was read back as `0`, and larger values could lose their final
+  digits. Values are now written at the shortest precision that reads back
+  identically, and parsed to a double in C rather than through R. Files written
+  by earlier versions are unaffected - the loss happened on read, so re-reading
+  them with this version recovers the original values (#97).
+- **Reading is substantially faster** - roughly 15-18x on files of a few hundred
+  thousand rows and up, and it scales linearly with file size.
+- `yyjsonr` and `jsonvalidate` are no longer required. `Imports` is now just
+  `hms`; `jsonvalidate` moved to `Suggests` and is needed only for the
+  `validate_*()` functions (#70).
+
+## New formats
+
+- `read_dataset_ndjson()`, `write_dataset_ndjson()` and
+  `validate_dataset_ndjson()` support the Dataset NDJSON representation, where
+  the dataset metadata forms the first line and each row follows as a JSON
+  array.
+- `read_dataset_dsjc()`, `write_dataset_dsjc()` and `validate_dataset_dsjc()`
+  support Dataset JSON Compressed (`.dsjc`), a zLib stream of Dataset NDJSON
+  content. Rows are compressed as they are written, so a large dataset never
+  needs its uncompressed form held in memory (#78).
+- `datasetjson_example()` now ships `dm.json`, `dm.ndjson` and `dm.dsjc`.
+
+## Behavior changes
+
+- `decimals_as_floats` has been removed from `read_dataset_json()`. Columns with
+  `dataType = "decimal"` and `targetDataType = "decimal"` are always converted to
+  numeric on read, as the Dataset JSON v1.1 specification requires (#77).
+- `digits` is deprecated in `write_dataset_json()` and `write_dataset_ndjson()`.
+  It is ignored, and supplying it warns. Decimals are written at whatever
+  precision reads back as the same value, so there is no precision left for it
+  to control. To render values at a fixed precision, format the column to
+  character yourself and declare it as `decimal`/`decimal` in the column
+  metadata.
+- `float_as_decimals` is no longer needed to protect precision and setting it
+  now warns to say so. It remains available as an interoperability choice for
+  systems that require the `decimal` type.
+- Reading a file whose columns lack a `name`, or whose `records` value is
+  missing, now reports the problem rather than failing obscurely or silently
+  substituting a value.
+
+## Fixes
+
+- `read_dataset_json()` sets `format.sas` from the `displayFormat` column
+  attribute (#87).
+- Reading from a URL returned only the first line of the file, so any
+  pretty-printed JSON lost everything after it and NDJSON returned no rows.
+- `write_dataset_json()` with `float_as_decimals = TRUE` wrote `NA` as a padded
+  string rather than JSON `null` (#76).
+- `extract_xpt_meta()` in the `converting_files` vignette could return `-Inf`
+  for the `length` of an empty or all-`NA` string column; the minimum is now 1
+  (#71).
 
 # datasetjson 0.3.0
 

diff --git a/README.Rmd b/README.Rmd
@@ -17,7 +17,7 @@ library(datasetjson)
 
 Welcome to **datasetjson**. **datasetjson** is an R package built to read and write [CDISC Dataset JSON](https://www.cdisc.org/standards/data-exchange/dataset-json) formatted datasets. 
 
-If you're stumbling into the world of Dataset JSON, you might be wondering "Why JSON?", as many have asked this question. We highly recommend you take a pit stop to read [this blog post](https://swhume.github.io/why-json-for-datasets) by Sam Hume one of the creators of the Dataset JSON standard. 
+If you're stumbling into the world of Dataset JSON, you might be wondering "Why JSON?", as many have asked this question. We highly recommend you take a pit stop to read [this blog post](https://humedatalabs.com/why-json-for-datasets/) by Sam Hume one of the creators of the Dataset JSON standard. 
 
 As always, we welcome your feedback. If you spot a bug, would like to see a new feature, or if any documentation is unclear - submit an issue through GitHub right [here](https://github.com/atorus-research/datasetjson/issues). 
 

diff --git a/README.md b/README.md
@@ -17,8 +17,8 @@ formatted datasets.
 If you’re stumbling into the world of Dataset JSON, you might be
 wondering “Why JSON?”, as many have asked this question. We highly
 recommend you take a pit stop to read [this blog
-post](https://swhume.github.io/why-json-for-datasets) by Sam Hume one of
-the creators of the Dataset JSON standard.
+post](https://humedatalabs.com/why-json-for-datasets/) by Sam Hume one
+of the creators of the Dataset JSON standard.
 
 As always, we welcome your feedback. If you spot a bug, would like to
 see a new feature, or if any documentation is unclear - submit an issue
@@ -99,95 +99,95 @@ cat(js_text)
 ```
 
     ## {
-    ##   "datasetJSONCreationDateTime": "2025-01-27T16:45:36",
-    ##   "datasetJSONVersion": "1.1.0",
-    ##   "fileOID": "/some/path",
-    ##   "dbLastModifiedDateTime": "2025-01-21T13:34:50",
-    ##   "originator": "Some Org",
-    ##   "sourceSystem": {
-    ##     "name": "source system",
-    ##     "version": "1.0"
-    ##   },
-    ##   "studyOID": "SOMESTUDY",
-    ##   "metaDataVersionOID": "MDV.MSGv2.0.SDTMIG.3.3.SDTM.1.7",
-    ##   "metaDataRef": "some/define.xml",
-    ##   "itemGroupOID": "IG.IRIS",
-    ##   "records": 5,
-    ##   "name": "IRIS",
-    ##   "label": "Iris",
-    ##   "columns": [
-    ##     {
-    ##       "itemOID": "IT.IR.Sepal.Length",
-    ##       "name": "Sepal.Length",
-    ##       "label": "Sepal Length",
-    ##       "dataType": "float",
-    ##       "keySequence": 2
+    ##     "datasetJSONCreationDateTime": "2026-09-03T20:28:51",
+    ##     "datasetJSONVersion": "1.1.0",
+    ##     "fileOID": "/some/path",
+    ##     "dbLastModifiedDateTime": "2025-01-21T13:34:50",
+    ##     "originator": "Some Org",
+    ##     "sourceSystem": {
+    ##         "name": "source system",
+    ##         "version": "1.0"
     ##     },
-    ##     {
-    ##       "itemOID": "IT.IR.Sepal.Width",
-    ##       "name": "Sepal.Width",
-    ##       "label": "Sepal Width",
-    ##       "dataType": "float"
-    ##     },
-    ##     {
-    ##       "itemOID": "IT.IR.Petal.Length",
-    ##       "name": "Petal.Length",
-    ##       "label": "Petal Length",
-    ##       "dataType": "float",
-    ##       "keySequence": 3
-    ##     },
-    ##     {
-    ##       "itemOID": "IT.IR.Petal.Width",
-    ##       "name": "Petal.Width",
-    ##       "label": "Petal Width",
-    ##       "dataType": "float"
-    ##     },
-    ##     {
-    ##       "itemOID": "IT.IR.Species",
-    ##       "name": "Species",
-    ##       "label": "Flower Species",
-    ##       "dataType": "string",
-    ##       "length": 10,
-    ##       "keySequence": 1
-    ##     }
-    ##   ],
-    ##   "rows": [
-    ##     [
-    ##       5.1,
-    ##       3.5,
-    ##       1.4,
-    ##       0.2,
-    ##       "setosa"
-    ##     ],
-    ##     [
-    ##       4.9,
-    ##       3.0,
-    ##       1.4,
-    ##       0.2,
-    ##       "setosa"
-    ##     ],
-    ##     [
-    ##       4.7,
-    ##       3.2,
-    ##       1.3,
-    ##       0.2,
-    ##       "setosa"
-    ##     ],
-    ##     [
-    ##       4.6,
-    ##       3.1,
-    ##       1.5,
-    ##       0.2,
-    ##       "setosa"
+    ##     "studyOID": "SOMESTUDY",
+    ##     "metaDataVersionOID": "MDV.MSGv2.0.SDTMIG.3.3.SDTM.1.7",
+    ##     "metaDataRef": "some/define.xml",
+    ##     "itemGroupOID": "IG.IRIS",
+    ##     "records": 5,
+    ##     "name": "IRIS",
+    ##     "label": "Iris",
+    ##     "columns": [
+    ##         {
+    ##             "itemOID": "IT.IR.Sepal.Length",
+    ##             "name": "Sepal.Length",
+    ##             "label": "Sepal Length",
+    ##             "dataType": "float",
+    ##             "keySequence": 2
+    ##         },
+    ##         {
+    ##             "itemOID": "IT.IR.Sepal.Width",
+    ##             "name": "Sepal.Width",
+    ##             "label": "Sepal Width",
+    ##             "dataType": "float"
+    ##         },
+    ##         {
+    ##             "itemOID": "IT.IR.Petal.Length",
+    ##             "name": "Petal.Length",
+    ##             "label": "Petal Length",
+    ##             "dataType": "float",
+    ##             "keySequence": 3
+    ##         },
+    ##         {
+    ##             "itemOID": "IT.IR.Petal.Width",
+    ##             "name": "Petal.Width",
+    ##             "label": "Petal Width",
+    ##             "dataType": "float"
+    ##         },
+    ##         {
+    ##             "itemOID": "IT.IR.Species",
+    ##             "name": "Species",
+    ##             "label": "Flower Species",
+    ##             "dataType": "string",
+    ##             "length": 10,
+    ##             "keySequence": 1
+    ##         }
     ##     ],
-    ##     [
-    ##       5.0,
-    ##       3.6,
-    ##       1.4,
-    ##       0.2,
-    ##       "setosa"
+    ##     "rows": [
+    ##         [
+    ##             5.1,
+    ##             3.5,
+    ##             1.4,
+    ##             0.2,
+    ##             "setosa"
+    ##         ],
+    ##         [
+    ##             4.9,
+    ##             3.0,
+    ##             1.4,
+    ##             0.2,
+    ##             "setosa"
+    ##         ],
+    ##         [
+    ##             4.7,
+    ##             3.2,
+    ##             1.3,
+    ##             0.2,
+    ##             "setosa"
+    ##         ],
+    ##         [
+    ##             4.6,
+    ##             3.1,
+    ##             1.5,
+    ##             0.2,
+    ##             "setosa"
+    ##         ],
+    ##         [
+    ##             5.0,
+    ##             3.6,
+    ##             1.4,
+    ##             0.2,
+    ##             "setosa"
+    ##         ]
     ##     ]
-    ##   ]
     ## }
 
 To read a Dataset JSON file, you can use `read_dataset_json()`. You can

diff --git a/cran-comments.md b/cran-comments.md
@@ -1,8 +1,31 @@
-# Submission 0.3.0
+# Submission 0.4.0
 
-- Updates to support a new version of the Dataset JSON standard. 
-- Changes to dependency JSON engine
-- Expanded user documentation
+- JSON parsing and serialization moved into the package, using the bundled
+  `yyjson` C library. This is the first release of `datasetjson` to contain
+  compiled code.
+- Removes the `yyjsonr` dependency, and moves `jsonvalidate` to `Suggests`.
+  `Imports` is now only `hms`.
+- Adds support for the Dataset NDJSON and Dataset JSON Compressed (`.dsjc`)
+  representations of the CDISC Dataset JSON standard.
+- Corrects a loss of numeric precision when reading files.
+
+## Bundled and linked code
+
+- `src/yyjson.c` and `src/yyjson.h` are the `yyjson` C library
+  (https://github.com/ibireme/yyjson), version 0.12.0, included unmodified.
+  It is released under the MIT license. The copyright holder is recorded in
+  `Authors@R` with role `cph`, in the `Copyright` field of DESCRIPTION, and in
+  `inst/COPYRIGHTS`, which reproduces the license in full.
+- The package links `zlib` (`PKG_LIBS = -lz`), which the Dataset JSON
+  Compressed format requires. `zlib` is required by R itself on all platforms.
+
+## Test environments
+
+- local macOS (aarch64), R 4.5.1
+- GitHub Actions: Ubuntu (release, devel, oldrel-1, oldrel-3), macOS (release),
+  Windows (release)
+- win-builder (release, devel)
+- macOS builder
 
 ## R CMD CHECK Results