From caab53aa174e002178fe90e9f69327278b26646b Mon Sep 17 00:00:00 2001 From: "mike.stackhouse" Date: Thu, 3 Sep 2026 20:29:34 -0400 Subject: [PATCH] Prepare 0.4.0 for CRAN - DESCRIPTION version 0.4.0 - NEWS.md rewritten for the release: the partial entry covered only the four PRs that landed before the JSON rewrite, and said nothing about the precision fix, the new NDJSON and DSJC formats, the dropped dependencies, or the deprecations - cran-comments.md records the two things CRAN will look at here: the bundled yyjson library and where its MIT attribution lives, and the zlib link that the compressed format needs - Fixed a moved URL in the README, and three real misspellings the spell check found (defintions, leiu, and Behaviour against the declared Language: en-US). Technical terms added to inst/WORDLIST --- DESCRIPTION | 2 +- NEWS.md | 62 +++++++++++- README.Rmd | 2 +- README.md | 176 ++++++++++++++++----------------- cran-comments.md | 31 +++++- inst/WORDLIST | 71 ++++++++++--- vignettes/converting_files.Rmd | 2 +- vignettes/odm_details.Rmd | 2 +- 8 files changed, 235 insertions(+), 113 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index bac4f8c..5530778 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,7 +1,7 @@ Package: datasetjson Type: Package Title: Read and Write CDISC Dataset JSON Files -Version: 0.3.0 +Version: 0.4.0 Authors@R: c( person(given = "Mike", family = "Stackhouse", diff --git a/NEWS.md b/NEWS.md index 17a421b..5249bf9 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,9 +1,63 @@ # datasetjson 0.4.0 -- Updated `read_dataset_json()`: `format.sas` is now set from the `displayFormat` attribute on columns (#87)= -- `decimals_as_floats` parameter removed from `read_dataset_json()`. Variables with `dataType = "decimal"` and `targetDataType = "decimal"` are now always converted to numeric on read, per the Dataset JSON v1.1 specification. -- Fixed a bug in the `converting_files` vignette where `extract_xpt_meta()` could return `-Inf` for the `length` field of empty or all-NA string columns; minimum length is now 1 (#71) -- Fixed a bug where `write_dataset_json()` with `float_as_decimals = TRUE` would write `NA` values as padded strings (e.g., `" NA"`) instead of JSON `null` (#76) +## JSON handling moved into the package + +Reading and writing are now handled by the `yyjson` C library, bundled with the +package, rather than by an R JSON package. The practical effects: + +- **Numbers survive a write and read unchanged.** Previously a float below + roughly 5e-7 was read back as `0`, and larger values could lose their final + digits. Values are now written at the shortest precision that reads back + identically, and parsed to a double in C rather than through R. Files written + by earlier versions are unaffected - the loss happened on read, so re-reading + them with this version recovers the original values (#97). +- **Reading is substantially faster** - roughly 15-18x on files of a few hundred + thousand rows and up, and it scales linearly with file size. +- `yyjsonr` and `jsonvalidate` are no longer required. `Imports` is now just + `hms`; `jsonvalidate` moved to `Suggests` and is needed only for the + `validate_*()` functions (#70). + +## New formats + +- `read_dataset_ndjson()`, `write_dataset_ndjson()` and + `validate_dataset_ndjson()` support the Dataset NDJSON representation, where + the dataset metadata forms the first line and each row follows as a JSON + array. +- `read_dataset_dsjc()`, `write_dataset_dsjc()` and `validate_dataset_dsjc()` + support Dataset JSON Compressed (`.dsjc`), a zLib stream of Dataset NDJSON + content. Rows are compressed as they are written, so a large dataset never + needs its uncompressed form held in memory (#78). +- `datasetjson_example()` now ships `dm.json`, `dm.ndjson` and `dm.dsjc`. + +## Behavior changes + +- `decimals_as_floats` has been removed from `read_dataset_json()`. Columns with + `dataType = "decimal"` and `targetDataType = "decimal"` are always converted to + numeric on read, as the Dataset JSON v1.1 specification requires (#77). +- `digits` is deprecated in `write_dataset_json()` and `write_dataset_ndjson()`. + It is ignored, and supplying it warns. Decimals are written at whatever + precision reads back as the same value, so there is no precision left for it + to control. To render values at a fixed precision, format the column to + character yourself and declare it as `decimal`/`decimal` in the column + metadata. +- `float_as_decimals` is no longer needed to protect precision and setting it + now warns to say so. It remains available as an interoperability choice for + systems that require the `decimal` type. +- Reading a file whose columns lack a `name`, or whose `records` value is + missing, now reports the problem rather than failing obscurely or silently + substituting a value. + +## Fixes + +- `read_dataset_json()` sets `format.sas` from the `displayFormat` column + attribute (#87). +- Reading from a URL returned only the first line of the file, so any + pretty-printed JSON lost everything after it and NDJSON returned no rows. +- `write_dataset_json()` with `float_as_decimals = TRUE` wrote `NA` as a padded + string rather than JSON `null` (#76). +- `extract_xpt_meta()` in the `converting_files` vignette could return `-Inf` + for the `length` of an empty or all-`NA` string column; the minimum is now 1 + (#71). # datasetjson 0.3.0 diff --git a/README.Rmd b/README.Rmd index c439fb1..debf0a4 100644 --- a/README.Rmd +++ b/README.Rmd @@ -17,7 +17,7 @@ library(datasetjson) Welcome to **datasetjson**. **datasetjson** is an R package built to read and write [CDISC Dataset JSON](https://www.cdisc.org/standards/data-exchange/dataset-json) formatted datasets. -If you're stumbling into the world of Dataset JSON, you might be wondering "Why JSON?", as many have asked this question. We highly recommend you take a pit stop to read [this blog post](https://swhume.github.io/why-json-for-datasets) by Sam Hume one of the creators of the Dataset JSON standard. +If you're stumbling into the world of Dataset JSON, you might be wondering "Why JSON?", as many have asked this question. We highly recommend you take a pit stop to read [this blog post](https://humedatalabs.com/why-json-for-datasets/) by Sam Hume one of the creators of the Dataset JSON standard. As always, we welcome your feedback. If you spot a bug, would like to see a new feature, or if any documentation is unclear - submit an issue through GitHub right [here](https://github.com/atorus-research/datasetjson/issues). diff --git a/README.md b/README.md index 3d1f6a4..57c3d99 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ formatted datasets. If you’re stumbling into the world of Dataset JSON, you might be wondering “Why JSON?”, as many have asked this question. We highly recommend you take a pit stop to read [this blog -post](https://swhume.github.io/why-json-for-datasets) by Sam Hume one of -the creators of the Dataset JSON standard. +post](https://humedatalabs.com/why-json-for-datasets/) by Sam Hume one +of the creators of the Dataset JSON standard. As always, we welcome your feedback. If you spot a bug, would like to see a new feature, or if any documentation is unclear - submit an issue @@ -99,95 +99,95 @@ cat(js_text) ``` ## { - ## "datasetJSONCreationDateTime": "2025-01-27T16:45:36", - ## "datasetJSONVersion": "1.1.0", - ## "fileOID": "/some/path", - ## "dbLastModifiedDateTime": "2025-01-21T13:34:50", - ## "originator": "Some Org", - ## "sourceSystem": { - ## "name": "source system", - ## "version": "1.0" - ## }, - ## "studyOID": "SOMESTUDY", - ## "metaDataVersionOID": "MDV.MSGv2.0.SDTMIG.3.3.SDTM.1.7", - ## "metaDataRef": "some/define.xml", - ## "itemGroupOID": "IG.IRIS", - ## "records": 5, - ## "name": "IRIS", - ## "label": "Iris", - ## "columns": [ - ## { - ## "itemOID": "IT.IR.Sepal.Length", - ## "name": "Sepal.Length", - ## "label": "Sepal Length", - ## "dataType": "float", - ## "keySequence": 2 + ## "datasetJSONCreationDateTime": "2026-09-03T20:28:51", + ## "datasetJSONVersion": "1.1.0", + ## "fileOID": "/some/path", + ## "dbLastModifiedDateTime": "2025-01-21T13:34:50", + ## "originator": "Some Org", + ## "sourceSystem": { + ## "name": "source system", + ## "version": "1.0" ## }, - ## { - ## "itemOID": "IT.IR.Sepal.Width", - ## "name": "Sepal.Width", - ## "label": "Sepal Width", - ## "dataType": "float" - ## }, - ## { - ## "itemOID": "IT.IR.Petal.Length", - ## "name": "Petal.Length", - ## "label": "Petal Length", - ## "dataType": "float", - ## "keySequence": 3 - ## }, - ## { - ## "itemOID": "IT.IR.Petal.Width", - ## "name": "Petal.Width", - ## "label": "Petal Width", - ## "dataType": "float" - ## }, - ## { - ## "itemOID": "IT.IR.Species", - ## "name": "Species", - ## "label": "Flower Species", - ## "dataType": "string", - ## "length": 10, - ## "keySequence": 1 - ## } - ## ], - ## "rows": [ - ## [ - ## 5.1, - ## 3.5, - ## 1.4, - ## 0.2, - ## "setosa" - ## ], - ## [ - ## 4.9, - ## 3.0, - ## 1.4, - ## 0.2, - ## "setosa" - ## ], - ## [ - ## 4.7, - ## 3.2, - ## 1.3, - ## 0.2, - ## "setosa" - ## ], - ## [ - ## 4.6, - ## 3.1, - ## 1.5, - ## 0.2, - ## "setosa" + ## "studyOID": "SOMESTUDY", + ## "metaDataVersionOID": "MDV.MSGv2.0.SDTMIG.3.3.SDTM.1.7", + ## "metaDataRef": "some/define.xml", + ## "itemGroupOID": "IG.IRIS", + ## "records": 5, + ## "name": "IRIS", + ## "label": "Iris", + ## "columns": [ + ## { + ## "itemOID": "IT.IR.Sepal.Length", + ## "name": "Sepal.Length", + ## "label": "Sepal Length", + ## "dataType": "float", + ## "keySequence": 2 + ## }, + ## { + ## "itemOID": "IT.IR.Sepal.Width", + ## "name": "Sepal.Width", + ## "label": "Sepal Width", + ## "dataType": "float" + ## }, + ## { + ## "itemOID": "IT.IR.Petal.Length", + ## "name": "Petal.Length", + ## "label": "Petal Length", + ## "dataType": "float", + ## "keySequence": 3 + ## }, + ## { + ## "itemOID": "IT.IR.Petal.Width", + ## "name": "Petal.Width", + ## "label": "Petal Width", + ## "dataType": "float" + ## }, + ## { + ## "itemOID": "IT.IR.Species", + ## "name": "Species", + ## "label": "Flower Species", + ## "dataType": "string", + ## "length": 10, + ## "keySequence": 1 + ## } ## ], - ## [ - ## 5.0, - ## 3.6, - ## 1.4, - ## 0.2, - ## "setosa" + ## "rows": [ + ## [ + ## 5.1, + ## 3.5, + ## 1.4, + ## 0.2, + ## "setosa" + ## ], + ## [ + ## 4.9, + ## 3.0, + ## 1.4, + ## 0.2, + ## "setosa" + ## ], + ## [ + ## 4.7, + ## 3.2, + ## 1.3, + ## 0.2, + ## "setosa" + ## ], + ## [ + ## 4.6, + ## 3.1, + ## 1.5, + ## 0.2, + ## "setosa" + ## ], + ## [ + ## 5.0, + ## 3.6, + ## 1.4, + ## 0.2, + ## "setosa" + ## ] ## ] - ## ] ## } To read a Dataset JSON file, you can use `read_dataset_json()`. You can diff --git a/cran-comments.md b/cran-comments.md index 10e6045..d5968f1 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -1,8 +1,31 @@ -# Submission 0.3.0 +# Submission 0.4.0 -- Updates to support a new version of the Dataset JSON standard. -- Changes to dependency JSON engine -- Expanded user documentation +- JSON parsing and serialization moved into the package, using the bundled + `yyjson` C library. This is the first release of `datasetjson` to contain + compiled code. +- Removes the `yyjsonr` dependency, and moves `jsonvalidate` to `Suggests`. + `Imports` is now only `hms`. +- Adds support for the Dataset NDJSON and Dataset JSON Compressed (`.dsjc`) + representations of the CDISC Dataset JSON standard. +- Corrects a loss of numeric precision when reading files. + +## Bundled and linked code + +- `src/yyjson.c` and `src/yyjson.h` are the `yyjson` C library + (https://github.com/ibireme/yyjson), version 0.12.0, included unmodified. + It is released under the MIT license. The copyright holder is recorded in + `Authors@R` with role `cph`, in the `Copyright` field of DESCRIPTION, and in + `inst/COPYRIGHTS`, which reproduces the license in full. +- The package links `zlib` (`PKG_LIBS = -lz`), which the Dataset JSON + Compressed format requires. `zlib` is required by R itself on all platforms. + +## Test environments + +- local macOS (aarch64), R 4.5.1 +- GitHub Actions: Ubuntu (release, devel, oldrel-1, oldrel-3), macOS (release), + Windows (release) +- win-builder (release, devel) +- macOS builder ## R CMD CHECK Results diff --git a/inst/WORDLIST b/inst/WORDLIST index 8e052b1..fb48d82 100644 --- a/inst/WORDLIST +++ b/inst/WORDLIST @@ -1,29 +1,74 @@ -ADXX +Acknowledgements ADaM +admiraltemplate +admiralxxx +adxx +ADXX +anonymized Arancibia +Atorus +autoset +BDAT +BestPharmaceuticals Biologics CDISC Changelog +DatasetJSON +datasetJSONVersion +dataType +dbLastModifiedDateTime +der +dev +displayFormat DM +DSJC Farrugia +fileOID +FileOID +FileOIDs +Github GSK +hackathon +hms +https +itemData +ItemDef +itemGroupData +ItemGroupDef +itemGroupOID +itemOID +json +keySequence +Lifecycle LinkedIn +metaDataRef +MetaDataVersion +metaDataVersionOID +NDJSON +ODM +OID Onboarding -README -SDTM -TAs -USUBJIDs -admiralxxx -admiraltemplate -adxx -anonymized -der -dev -https onboarding -pkgdown +ORCID pharmaverse +pkgdown +README renv repo roxygen +RStudio +SDTM +Simms +sourceSystem +Straub +studyOID +sys +targetDataType +TAs Template +USUBJIDs +XPT +XPTs +yyjson +yyjsonr +zLib diff --git a/vignettes/converting_files.Rmd b/vignettes/converting_files.Rmd index 338b029..a97e6d4 100644 --- a/vignettes/converting_files.Rmd +++ b/vignettes/converting_files.Rmd @@ -25,7 +25,7 @@ Some users may be interested in converting SAS Version 5 Transport files into XP ## Converting from XPT -Ideally if you're converting from an XPT file or a SAS7BDAT file, the appropriate metadata would be maintained separate from the dataset itself. Inevitably some information will get lost while being exchanged because of differences of what information was applied or how R interprets SAS datasets. This is one reason why **{datasetjson}** improves upon XPTs as an exchange format. In leiu of external metadata, here's an example of how you can make a best effort conversion. +Ideally if you're converting from an XPT file or a SAS7BDAT file, the appropriate metadata would be maintained separate from the dataset itself. Inevitably some information will get lost while being exchanged because of differences of what information was applied or how R interprets SAS datasets. This is one reason why **{datasetjson}** improves upon XPTs as an exchange format. In lieu of external metadata, here's an example of how you can make a best effort conversion. ```{r XPT conversion} adsl <- haven::read_xpt(file.path(system.file(package='datasetjson'), "adsl.xpt")) diff --git a/vignettes/odm_details.Rmd b/vignettes/odm_details.Rmd index a4a93d7..6a02032 100644 --- a/vignettes/odm_details.Rmd +++ b/vignettes/odm_details.Rmd @@ -18,7 +18,7 @@ Dataset JSON inherently pulls some standards from the [ODM v2.0](https://wiki.cd > The Operational Data Model (ODM) is a vendor-neutral, platform-independent data exchange format, intended primarily for interchange and archival of clinical study data pertaining to individual subjects, aggregated collections of subjects, and integrated research studies. The model includes the clinical data along with its associated metadata, administrative data, reference data, and audit information. All information that needs to be shared among different software systems during the setup, operation, analysis, submission, and archiving of a study are included in the model. -Without having a background in ODM, some fields within the Dataset JSON file may be hard to understand without the ODM references available. This vignette provides some quick defintions that may prove useful. +Without having a background in ODM, some fields within the Dataset JSON file may be hard to understand without the ODM references available. This vignette provides some quick definitions that may prove useful. ## fileOID