Repository navigation
feat: skip unchanged folders on import, and check data before publishing - #221
Merged
Merged
Conversation
-import-data and -create-db import every dataset folder whole, and that import was neither checked nor recorded, so each run re-ran citydb-tool over every file of every folder. --import-mode=skip kept it free of duplicates, but a scheduled import that only has to pick up new files re-read everything each time. A whole-folder import is now recorded by the fingerprint of the folder's files, as bbox imports already were, with an extent covering the whole world: it holds every feature, so it covers any bbox in any mode. A later whole-folder import, or a bbox import, of unchanged files skips citydb-tool; adding, replacing or removing a file changes the fingerprint and the folder is imported again. An import with IMPORT_LIMIT set stays unrecorded, as before, and so does one without a database connection, as in unit tests. docs/deployment/adding-data.md describes the scheduled import, the staging of new files by rename, and that one invalid attribution.json stops the country's whole import.
A data folder was only checked when it was imported, so files with a broken attribution.json or a CRS City2TABULA does not use reached the data folder before anyone found out, and the attribution rules lived nowhere a publishing step could run them. c2t -check-data <dir> checks a staged dataset folder for COUNTRY without importing it or connecting to a database. It runs the import's own checks (attribution.json, a gml/ or cityjson/ folder) and adds: no model file outside those folders, hidden ones skipped as the import skips them; no published folder of the country using the dataset_id with a different attribution.json (the same one means the staged files extend that dataset); and for every model file, gzipped or not and in any subfolder, that it is CityGML or CityJSON and that its declared CRS (srsName in the first MiB, or the CityJSON metadata.referenceSystem) resolves to the SRID the import uses. Accepted CRS names are the EPSG urn, OGC URI and EPSG:<code> forms, the AdV names for UTM 32 and 33, and the compound EPSG:7415 for RD New with NAP. Anything else, such as "unknown" or a misspelt urn, fails. The command lists every problem with its path and exits 1, or prints "ready to publish" and exits 0. Run on real folders: Bavaria, NRW, 3DBAG, Vienna and Brno pass; Prague fails on its missing attribution.json and srsName "unknown".
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes for publishing data region by region into a running deployment, one commit each.
Skip unchanged dataset folders on a whole-folder import
-import-dataand-create-dbre-ran citydb-tool over every file of every dataset folder on each run, because a whole-folder import was neither checked nor recorded. A whole-folder import is now recorded by the fingerprint of the folder's files (path, size, modification time), as bbox imports already were, with an extent covering the whole world. A later whole-folder or bbox import of unchanged files skips citydb-tool; a changed folder is imported again, with existing buildings skipped. Imports withIMPORT_LIMITstay unrecorded.c2t -check-data <dir>Checks a staged dataset folder for
COUNTRYwithout importing it or connecting to a database:attribution.jsongml/orcityjson/, or a model file outside them (hidden folders skipped)dataset_idattribution.jsonAccepted CRS names:
urn:ogc:def:crs:EPSG::<code>(also compound),https://www.opengis.net/def/crs/EPSG/0/<code>,EPSG:<code>, the AdV namesurn:adv:crs:ETRS89_UTM32*/UTM33*, and EPSG:7415 (RD New + NAP) as 28992. It lists every problem with its path and exits 1, or printsready to publish.Docs
docs/deployment/adding-data.md(new, in the nav as Adding Data): the scheduled import command, what an import does per folder state, the attribution check that stops a country's import,-check-data, and staging by rename.CHANGELOG.md.Tests
TestImportCityDBData_SkipsUnchangedWholeFolder(integration, fake citydb-tool counting calls): whole-folder import, the same again, a bbox import, a file added, the same again. Before the change it fails:same files again: citydb import called 2 times in total, want 1.TestResolveCRS,TestCheckDataset(11 cases, including gzipped CityGML in a region subfolder, CityJSON with metadata after 1.5 MB ofCityObjects, a foreign CRS,unknown, a stray model file, hidden folders, bothdataset_idcases),TestCheckDataset_UsesConfiguredSRID,TestCheckDataset_MissingAttributionAndModelFolders.go vet ./...andgo test ./...pass; the integration suites for importer, process, onrequest, db and api pass;mkdocs build --strictpasses.-check-dataon real folders: Bavaria, NRW, 3DBAG, Vienna and Brno pass (each under a second); Prague fails on its missingattribution.jsonandsrsName "unknown". It also runs from an empty directory with no database settings.