Repository navigation

develop => main - #285

Merged

ialarmedalien merged 162 commits into

mainfrom

develop

Oct 20, 2026

Merged

develop => main#285
ialarmedalien merged 162 commits into
mainfrom
develop

Commits on Jun 22, 2026
- Rename functions/variables to remove references to delta lake format
  ialarmedalien
  committed
  104126d
- rm extra word in error msg
  ialarmedalien
  committed
  6016d2a
- Merge pull request #196 from kbase/rename_spark_delta
  ialarmedalien
  authored
  6065adf
- Renaming spark_delta => spark
  ialarmedalien
  committed
  ed62453
- Allow bucket-only ls with s3 CLI (#195)
  
  authored
  ee68e52
- Merge branch 'develop' into rename_spark_delta_II
  ialarmedalien
  authored
  44d07a0
- Merge pull request #197 from kbase/rename_spark_delta_II
  ialarmedalien
  authored
  df9258d
Commits on Jun 30, 2026
Commits on Jul 1, 2026
- update ceph health check
  mattldawson
  committed
  c21811f
- Potential fix for pull request finding
  
  mattldawson
  and
  Copilot
  authored
  1527701
- undo copilot changes
  mattldawson
  committed
  628f83b
- Refactor NCBI FTP Promote into CLI Tool (#201)
  
  mattldawson
  and
  Copilot
  authored
  78e41f4
- Merge branch 'develop' into develop-update-ceph-scripts
  mattldawson
  committed
  c01480f
- Merge pull request #202 from kbase/develop-update-ceph-scripts
  ialarmedalien
  authored
  301f583
Commits on Jul 8, 2026
- Add `xsv-validate.sh` to Dockerfile (#203)
  
  mattldawson
  and
  Copilot
  authored
  b60f3bf
Commits on Jul 9, 2026
- Moving the settings objects to cdm_data_loaders.core (#204)
  
  ialarmedalien
  and
  github-code-quality[bot]
  authored
  ae0a5ce
Commits on Jul 10, 2026
- Add xSV validation module (#205)
  
  authored
  697c878
Commits on Jul 31, 2026
- Add PDB manifest pipeline (#206)
  mattldawson
  authored
  bbcfbe1
- file transfer / download / upload reorg, part I (#208)
  ialarmedalien
  authored
  2ff014e
Commits on Aug 3, 2026
Commits on Aug 4, 2026
- Checksums module (#213)
  ialarmedalien
  authored
  28d7f53
- Adding transfer config (#214)
  ialarmedalien
  authored
  04cf11c
Commits on Aug 5, 2026
- Adding in missing tests and an interface to upload_fileobj (#215)
  ialarmedalien
  authored
  8a088b0
Commits on Aug 7, 2026
- Adding JSON schema for various different file types (#216)
  ialarmedalien
  authored
  9d99cfc
Commits on Aug 10, 2026
- Minor reorg: (#217)
  ialarmedalien
  authored
  92c8b14
Commits on Aug 12, 2026
- Adding new xsv validator mode plus support, with tests. (#220)
  ialarmedalien
  authored
  4bfb0c7
Commits on Aug 13, 2026
Commits on Aug 16, 2026
- Adding script to filter ncbi updates by date and IDs of interest (#224)
  ialarmedalien
  authored
  b30bd3e
Commits on Aug 19, 2026
- add rclone instructions (#225)
  
  mattldawson
  and
  ialarmedalien
  authored
  082923e
Commits on Aug 20, 2026
Commits on Aug 21, 2026
- Updating pipelines that use XML parsers to reflect the new code (#229)
  ialarmedalien
  authored
  f0e038c
Commits on Aug 25, 2026
- Editing standard settings model config to autopopulate the dlt_config (#230)
  ialarmedalien
  authored
  03a3192
Commits on Aug 26, 2026
- Adding in alias checks and trimming fat from pipelines. (#231)
  ialarmedalien
  authored
  265aa05
- make input/output names symmetrical (#232)
  ialarmedalien
  authored
  a5e16ae
Commits on Aug 28, 2026
- Adding CLI shortcuts (#233)
  ialarmedalien
  authored
  1d151fe
Commits on Aug 31, 2026
- Updating interface for qsv validation to include custom metaschema (#234)
  ialarmedalien
  authored
  eeed07d
Commits on Sep 8, 2026
- splitting out tests into unit and integration/e2e (#235)
  ialarmedalien
  authored
  00c48fe
Commits on Sep 10, 2026
- Extract e2e (#236)
  ialarmedalien
  authored
  b1ec217
Commits on Sep 11, 2026
Commits on Sep 14, 2026
- Adding generic JSONLines and XML to Dict pipelines (#240)
  ialarmedalien
  authored
  f7f36bb
Commits on Sep 15, 2026
- switch out tests that mock dlt with real lightweight dlt pipeline tests (#241)
  ialarmedalien
  authored
  112d763
- Minor file reorg (#242)
  ialarmedalien
  authored
  33b184f
Commits on Sep 17, 2026
- Adding arrow buffers for saving structured data (#243)
  ialarmedalien
  authored
  b190ea7
Commits on Sep 18, 2026
- Misc fixes, duplicate code removal, etc. (#244)
  ialarmedalien
  authored
  d62ded0
- Adding more thorough end to end tests for the xml pipeline (#245)
  ialarmedalien
  authored
  b6856f3
Commits on Sep 22, 2026
- Tests for xmltodict pipeline against reference dataset (#246)
  ialarmedalien
  authored
  5da9c26
- Adding in XSD capabilities (#247)
  ialarmedalien
  authored
  a7bfd3d
Commits on Sep 23, 2026
- Adding pyiceberg to jsonschema clieant for even more data parsing fun! (#248)
  ialarmedalien
  authored
  8b6db3a
Commits on Sep 24, 2026
Commits on Sep 25, 2026
- minor settings change
  ialarmedalien
  committed
  63af77b
- Minor config file changes (#252)
  ialarmedalien
  authored
  2bc5a58
- Merge branch 'develop' into converter_chaos_step_1
  ialarmedalien
  authored
  483f5ab
- Add converter refactor plan with completed inventory
  ialarmedalien
  committed
  b1eb808
- Extract shared schema conversion core and isolate column metadata
  ialarmedalien
  committed
  9e1921e
- Extract dlt schema tree normalization and shared nested paths
  ialarmedalien
  committed
  7518f26
- Add immutable schema IR, validated extensions, and source readers
  ialarmedalien
  committed
  213f5fa
- Implement direct IR emitters for JSON Schema, dlt, and PySpark
  ialarmedalien
  committed
  af4b57c
- Compose converter facades from readers and emitters
  ialarmedalien
  committed
  3336d65
- Fix converter code-review findings and switch docstrings to Sphinx style
  ialarmedalien
  committed
  09a4886
- Refactor metadata collection in emitters and add PySpark union-as-JSON support
  ialarmedalien
  committed
  012621e
- Add linkml emitter
  ialarmedalien
  committed
  4d961ce
- Revert unrequired commits
  ialarmedalien
  committed
  f471c55
- Fixing annotation
  ialarmedalien
  committed
  fe9a8b6
- Fixing a couple of issues causing test failures.
  ialarmedalien
  committed
  f1455f7
Commits on Sep 26, 2026
- adding validator-based tests
  ialarmedalien
  committed
  5d7facd
- ridiculous .md file formatting
  ialarmedalien
  committed
  7555c38
- Fix logger init errors, reorganize files under the converters tree (#256)
  ialarmedalien
  authored
  f1b4ade
- Merge pull request #255 from kbase/converter_cleanup
  ialarmedalien
  authored
  996861d
Commits on Sep 28, 2026
- Adding pyiceberg to linkml catalog converter
  ialarmedalien
  committed
  92146a0
- Minor refactor to get rid of redundancies
  ialarmedalien
  committed
  6bf5de0
Commits on Sep 29, 2026
- Redo linkml emitter to use official linkml packages. Remove redundancy in pyspark emission.
  ialarmedalien
  committed
  6988678
Commits on Sep 30, 2026
- Emit linkml schemas by namespace
  ialarmedalien
  committed
  6fcb93d
- Merge pull request #259 from kbase/refactor/converters-cleanup
  ialarmedalien
  authored
  4c8465a
- Remove old / superseded code
  ialarmedalien
  committed
  c7e61e0
- More cruft getting the boot
  ialarmedalien
  committed
  5d89fe1
- remove the audit packages
  ialarmedalien
  committed
  1842c54
- Merge pull request #260 from kbase/remove_cruft
  ialarmedalien
  authored
  2db8730
- Adding grouped emitter for jsonschema output
  ialarmedalien
  committed
  0c048c8
- More cruft removal; editing test runner script to install its own dependencies
  ialarmedalien
  committed
  18cba36
- Merge pull request #261 from kbase/group_jsonschema_outputs
  ialarmedalien
  authored
  eb23c9b
Commits on Oct 1, 2026
- Adding a JSONSchema to pyarrow converter
  ialarmedalien
  committed
  b4d4eae
- Merge pull request #262 from kbase/jsonschema_to_pyarrow
  ialarmedalien
  authored
  0d0526a
- xMinor housekeeping on JSONlines importers
  ialarmedalien
  committed
  bb062e8
- Update the compaction module to run whenever compact_reused_tables is True
  ialarmedalien
  committed
  5baa0d8
- Adding all-important openapi_pydantic.py
  ialarmedalien
  committed
  395f571
- Test fixes, missing fixtures
  ialarmedalien
  committed
  90165ca
Commits on Oct 7, 2026
- Update dependabot.yml
  ialarmedalien
  authored
  6904d2e
- Merge pull request #265 from kbase/update_dependabot_config
  ialarmedalien
  authored
  aed6ac5
- formatting
  ialarmedalien
  committed
  0a263e2
Commits on Oct 8, 2026
- Test fixes
  ialarmedalien
  committed
  9ac5788
- isolate failing test file
  ialarmedalien
  committed
  075c4b9
- Merge branch 'develop' into pipelines_again
  ialarmedalien
  authored
  0864fcd
- Misc copilot review fixes and test fix
  ialarmedalien
  committed
  abd346a
- More fixes...
  ialarmedalien
  committed
  a0acc49
- Close sqlite db connection to prevent memory leaks
  ialarmedalien
  committed
  ea4d415
- Merge pull request #267 from kbase/pipelines_again
  ialarmedalien
  authored
  c026b15
- Minor formatting updates
  ialarmedalien
  committed
  d054983
- More updates
  ialarmedalien
  committed
  1f7f8ae
- More fixes
  ialarmedalien
  committed
  868ecf6
- Rework fields, add destination.py, CliApp.run(), etc.
  ialarmedalien
  committed
  ddde65b
Commits on Oct 9, 2026
- Add xml2db changes, fix all tests
  ialarmedalien
  committed
  90edb94
- Bump the actions group across 1 directory with 5 updates
  dependabot[bot]
  authored
  d1f28a9
Commits on Oct 10, 2026
Commits on Oct 14, 2026
Commits on Oct 16, 2026
- Bump github/codeql-action/upload-sarif
  dependabot[bot]
  authored
  e78f6f8
Commits on Oct 17, 2026
- update dockerfile and entrypoint to simplify calling pipelines and allow dev branch usage
  ialarmedalien
  committed
  c76f3df
- Dedupe NCBI REST API test files
  ialarmedalien
  committed
  1486084
- Small standalone fixes and housekeeping
  ialarmedalien
  committed
  8f2c82a
- Rename _ingest pipelines and test files
  ialarmedalien
  committed
  975ddef
- Merge pull request #273 from kbase/dependabot/github_actions/actions-902dd7c283
  ialarmedalien
  authored
  2f56de5
- Merge branch 'develop' into housekeeping
  ialarmedalien
  authored
  1a911f0
- formatting
  ialarmedalien
  committed
  28d491f
- Split up container tests
  ialarmedalien
  committed
  9c6b70d
- Fix run_registry_ingest ref
  ialarmedalien
  committed
  c250b55
- More housekeeping updates
  ialarmedalien
  committed
  a4c4f37
- Merge pull request #276 from kbase/housekeeping
  ialarmedalien
  authored
  5fdb258
- Convert the xsv import pipeline to using the filesystem resource
  ialarmedalien
  committed
  4197c30
- Merge branch 'develop' into xsv_reader_to_filesystem
  ialarmedalien
  authored
  f2ff1d4
- Merge pull request #277 from kbase/xsv_reader_to_filesystem
  ialarmedalien
  authored
  1a4137f
Commits on Oct 18, 2026
- Pipeline cli (#278)
  ialarmedalien
  authored
  b7c7cd8
- Introduce FileFormatIngestSettings and standardise
  ialarmedalien
  committed
  1250219
- Fix failing tests and apply ruff lint/format fixes
  ialarmedalien
  committed
  cd0f31a
- Merge pull request #279 from kbase/field_reorg
  ialarmedalien
  authored
  c3632b5
- Decouple CtsSettings.output_dir from dlt destination bucket_url, enforce LocalInputOutputSettings
  ialarmedalien
  committed
  d0a35ac
- Merge pull request #281 from kbase/remove_destination
  ialarmedalien
  authored
  7ec664d
- Merge branch 'develop' into use_develop
  ialarmedalien
  authored
  447d302
- Merge pull request #275 from kbase/use_develop
  ialarmedalien
  authored
  be394ed
- Adding extra codecov coverage uploads
  ialarmedalien
  committed
  043e6ee
- Merge pull request #282 from kbase/codecov
  ialarmedalien
  authored
  099bb99
- Removing commented-out code
  ialarmedalien
  committed
  0f0ef39
- Merge pull request #283 from kbase/rm_comments
  ialarmedalien
  authored
  b2bd3ef
Commits on Oct 19, 2026
- Adding pipeline report file to output
  ialarmedalien
  committed
  28de00c
- Formatting
  ialarmedalien
  committed
  510a4c6
- Adding in pipeline report code
  ialarmedalien
  committed
  0987e54
- Merge pull request #284 from kbase/pipeline_report
  ialarmedalien
  authored
  f086928
- Bump version from 0.1.10 to 0.1.11
  ialarmedalien
  authored
  3fadfaa
- Merge pull request #286 from kbase/version-bump
  ialarmedalien
  authored
  9dfc402
- Merge branch 'main' into develop
  ialarmedalien
  authored
  cd5f88f
- remove infernal spark-based annoyances
  ialarmedalien
  committed
  cbe981d
Commits on Oct 20, 2026
- Merge pull request #287 from kbase/no_more_bright_sparks
  ialarmedalien
  authored
  2ec93fb

Commits

Commits on Jun 22, 2026

Commits on Jun 30, 2026

Commits on Jul 1, 2026

Commits on Jul 8, 2026

Commits on Jul 9, 2026

Commits on Jul 10, 2026

Commits on Jul 31, 2026

Commits on Aug 3, 2026

Commits on Aug 4, 2026

Commits on Aug 5, 2026

Commits on Aug 7, 2026

Commits on Aug 10, 2026

Commits on Aug 12, 2026

Commits on Aug 13, 2026

Commits on Aug 16, 2026

Commits on Aug 19, 2026

Commits on Aug 20, 2026

Commits on Aug 21, 2026

Commits on Aug 25, 2026

Commits on Aug 26, 2026

Commits on Aug 28, 2026

Commits on Aug 31, 2026

Commits on Sep 8, 2026

Commits on Sep 10, 2026

Commits on Sep 11, 2026

Commits on Sep 14, 2026

Commits on Sep 15, 2026

Commits on Sep 17, 2026

Commits on Sep 18, 2026

Commits on Sep 22, 2026

Commits on Sep 23, 2026

Commits on Sep 24, 2026

Commits on Sep 25, 2026

Commits on Sep 26, 2026

Commits on Sep 28, 2026

Commits on Sep 29, 2026

Commits on Sep 30, 2026

Commits on Oct 1, 2026

Commits on Oct 7, 2026

Commits on Oct 8, 2026