feat: skip unchanged folders on import, and check data before publishing

CHANGELOG.md

@@ -3,6 +3,23 @@
 Each release is listed with its breaking changes first. Releases before
 v0.8.0 are described in their tag messages.
 
+## Unreleased
+
+Features:
+- -import-data and -create-db record a whole-folder import by its
+  files' fingerprint, as bbox imports already were, so a later import
+  skips citydb-tool for every dataset folder whose files are unchanged.
+  A scheduled `c2t -import-data` is then cheap when nothing is new. A
+  bbox run after a whole-folder import of unchanged files is skipped
+  too. docs/deployment/adding-data.md covers the scheduled import, the
+  staging of new files and the attribution check.
+- c2t -check-data <dir> checks a staged dataset folder for COUNTRY
+  without importing it or connecting to a database: attribution.json,
+  the gml/ and cityjson/ folders, a dataset_id no published folder uses
+  with another attribution, and each model file's format and declared
+  CRS, which must be the SRID the import uses. It lists every problem
+  and exits with status 1, or prints "ready to publish".
+
 ## v0.13.0 (2026-10-25)
 
 Features:

cmd/c2t/main.go

@@ -42,6 +42,11 @@ func main() {
 	utils.InitLogger()
 	config := config.LoadConfig()
 
+	// Before Validate: a check needs only COUNTRY, not database settings.
+	if f.CheckData != "" {
+		os.Exit(checkData(&config, f.CheckData))
+	}
+
 	if err := config.Validate(); err != nil {
 		utils.Error.Fatal("Invalid configuration:", err)
 	}
@@ -122,3 +127,18 @@ func main() {
 		utils.Info.Println(flagMessages.LinkPylovo.Success)
 	}
 }
+
+// checkData prints every problem importer.CheckDataset finds in dir and returns
+// the process exit code: 0 when the folder is ready to publish, 1 otherwise.
+func checkData(cfg *config.Config, dir string) int {
+	problems := importer.CheckDataset(cfg, dir)
+	for _, p := range problems {
+		fmt.Fprintln(os.Stderr, p)
+	}
+	if len(problems) > 0 {
+		fmt.Fprintf(os.Stderr, "%s: %d problem(s); not ready to publish under %s\n", dir, len(problems), cfg.Country)
+		return 1
+	}
+	fmt.Printf("%s: ready to publish under %s\n", dir, cfg.Country)
+	return 0
+}

docs/deployment/adding-data.md

@@ -0,0 +1,62 @@
+---
+audience: developer
+---
+
+# Adding data to a running deployment
+
+A country's database grows by adding source files under `data/lod2/<country>/` and importing them. The import can run on a schedule: a run over unchanged files does nothing.
+
+```bash
+COUNTRY=germany c2t -create-db     # once per country: creates the database and imports every folder
+COUNTRY=germany c2t -import-data   # afterwards, as often as needed: imports what changed
+```
+
+`-import-data` only imports. Feature extraction and the PyLovo link run when a region is requested (`POST /api/v1/runs`), or with `-extract-features` and `-link-pylovo`.
+
+## What an import does
+
+| Folder state since the last import | Result |
+|---|---|
+| No files added, replaced or removed | citydb-tool is not run for the folder |
+| Files added or replaced | The folder is imported again; buildings already in the database are skipped, so none is duplicated |
+| New dataset folder | The folder is imported |
+
+A folder counts as changed when the path, size or modification time of any file in its `gml/` or `cityjson/` folder changes. A run for a bbox (`POST /api/v1/runs`, or `-import-data -bbox`) after a whole-folder import of unchanged files does not run citydb-tool either.
+
+!!! warning "One invalid attribution file stops the country's import"
+    Before anything is imported, every dataset folder of the country is checked: it needs a valid `attribution.json` ([Dataset attribution file](../code/attribution/index.md)) and a `gml/` or `cityjson/` folder, and no model file may lie directly in `data/lod2/<country>/`. One failing folder stops the whole import for that country, with an error naming the folder; nothing is imported until it is fixed. A scheduled job should report the error.
+
+## Checking files before publishing
+
+`-check-data` checks a staged dataset folder for the country without importing it or connecting to a database. It needs only `COUNTRY`.
+
+```bash
+COUNTRY=germany c2t -check-data data/lod2/germany/.bavaria-upload
+```
+
+| Check | Fails when |
+|---|---|
+| `attribution.json` | It is missing or breaks a rule of the [attribution file](../code/attribution/index.md) |
+| Model folders | There is neither a `gml/` nor a `cityjson/` folder, or a model file lies outside them (hidden folders are not read) |
+| `dataset_id` | A published folder of the country uses it with a different `attribution.json`. The same file means the staged files extend that dataset |
+| Model files | A file, gzipped or not and in any subfolder, is not CityGML or CityJSON, or its declared CRS is not the SRID the import uses: `CITYDB_SRID`, or the country's default ([Configuration](../configuration/index.md)) |
+
+The CRS is read from a CityGML file's `srsName` values in its first MiB and from a CityJSON file's `metadata.referenceSystem`. EPSG names are accepted in the `urn:ogc:def:crs:EPSG::<code>`, `https://www.opengis.net/def/crs/EPSG/0/<code>` and `EPSG:<code>` forms, as are the AdV names `urn:adv:crs:ETRS89_UTM32*` and `urn:adv:crs:ETRS89_UTM33*` and the compound EPSG:7415 (Amersfoort / RD New + NAP). Any other name, such as `unknown`, fails.
+
+The command lists every problem, each naming its file or folder, and exits with status 1; a folder that passes prints `ready to publish` and exits with status 0.
+
+## Staging files
+
+A file that is still being written must not be in a model folder when an import runs. Write files outside the model folders, then rename them into place on the same filesystem, where a rename is atomic. No marker file is needed.
+
+| Adding | Steps |
+|---|---|
+| A new dataset | Write the folder under a hidden name, e.g. `data/lod2/germany/.bavaria-upload/` with `attribution.json` and `gml/`, then rename it to `data/lod2/germany/bavaria/`. Hidden folders are not read. |
+| Files to an existing dataset | Write each file to `data/lod2/germany/.staging/`, then rename it into `data/lod2/germany/bavaria/gml/`. |
+
+A file renamed into place during an import is picked up by the next one.
+
+## Limitations
+
+- A provider's corrected tile is not applied: buildings already in the database are skipped on re-import, and their extracted features are kept.
+- An import with `IMPORT_LIMIT` set is partial by design and is never skipped.

internal/flags/flags.go

@@ -14,6 +14,7 @@ type Flags struct {
 	ShowVersion     bool
 	ShowV           bool
 	Bbox            string
+	CheckData       string
 	BboxMode        string
 }
 
@@ -30,6 +31,7 @@ func ParseFlags() *Flags {
 	flag.BoolVar(&f.ShowVersion, "version", false, "print version and exit")
 	flag.BoolVar(&f.ShowV, "v", false, "print version and exit (shorthand)")
 	flag.StringVar(&f.Bbox, "bbox", "", "Bounding box spatial filter for -import-data, format xmin,ymin,xmax,ymax[,srid] (passed through to citydb-tool)")
+	flag.StringVar(&f.CheckData, "check-data", "", "Check a staged dataset folder for publishing under COUNTRY (attribution.json, model folders, dataset_id, each model file's format and CRS) without importing it or connecting to a database; exits 1 listing every problem")
 	flag.StringVar(&f.BboxMode, "bbox-mode", "intersects", "Bounding box filter mode for -bbox: intersects, contains, or on_tile")
 	flag.Parse()
 	return f