Settle the lower tail of flying_height per roll from the logbooks (#60)

CLAUDE.md

@@ -50,6 +50,12 @@ the note and NEWS quote. Ships `inst/extdata/flying_height_sweep.csv` and
 `flying_height_population.csv`, which `test-fly_footprint_height.R` reads so the three constants
 are checked against the data. Use a PSOCK cluster for remote DEM reads — `mclapply()` forks
 abort on GDAL's curl handles on macOS while the wrapper exits 0
+- `data-raw/height_calibrate-lower_tail_rolls.R` — settles the lower tail of `flying_height`
+per roll (fly#60) with adjacent-frame spacing and the hand transcription of the province's
+logbook scans in `data-raw/flying_height_logbooks.csv`, which is an input and is never
+regenerated. Writes `inst/extdata/flying_height_rolls.csv` and
+`flying_height_rolls_excluded.csv`; `test-fly_footprint_height_rolls.R` holds both to the
+sweep
 - `height_fixture()` and `flat_dem()` in `tests/testthat/setup.R` — eight frames over level
 ground, each reaching one height check by a stated route; rows 7 and 8 exist because two
 deliberate defects survived the first six
@@ -269,6 +275,30 @@ first DEM pass and **before** the second, and a camera-table frame over the ceil
 seeded — classifying after both passes returns the right footprint and still fetches 110 km of
 terrain per frame, so the test asserts on the grids. Read `inst/notes/terrain-correction.md`
 
+- **The lower tail of `flying_height` is settled per roll, from the logbooks, and 389 of
+the 1,001 frames it settled were never a height error** (v0.15.0, #60) — #54 left 1,962 film frames
+under half their nominal height `"implausible"` because ×10.764, ×10 and ×2 fit equally
+pooled. That was the wrong unit: they are 42 rolls, nearly each carrying one round-feet
+height, so 77 roll-heights. Two witnesses that read none of the disputed fields — spacing
+between frames adjacent by number against the ~60% designed overlap, and the TRUE HEIGHT
+column of the logbook scans at `flight_log_url` — settle 22 roll-heights (1,001 frames):
+10 dropped a digit, 1 dropped two, and **11 carry the right height beside a wrong
+`scale`**, where #54's fallback to nominal was the defect. `fly_footprint()` reads
+`flying_height_rolls.csv` and marks them `"corrected_roll_table"`.
+
+  **Four things are load-bearing.** The shipped height is the **logbook's**, converted,
+never the catalogue's times the factor — `bc7280`'s 20,000 ft is catalogued as 60 m and
+×100 of that is 96 m short. The key is roll, height, lens and scale with numbers
+**formatted identically** on both sides, since `paste()` spells 100000L and 100000
+differently. The table reaches frames catalogued **below the terrain**, which is how a
+height missing two digits arrives, so it is gated on the band and not on `disputed`. And
+every excluded roll-height carries the reason that actually fired — no page, rows that
+disagree, a row not read are three states, and an earlier draft folded them into one.
+Spacing separates readings ~1.6× apart and no closer, so it cannot tell ×10 from ×10.764;
+the logbook can. The control roll `bc78065` suggests #54's own factor is ×10 on pre-2000
+rolls (fly#71), and half the r ≈ 2 mass is a wrong scale, not a mislabelled lens (fly#72).
+Do not change `fly_height_slip_factor()` from this; read `inst/notes/terrain-correction.md`
+
 ## Gotchas
 
 - `.lintr` must be single-line DCF format — multi-line breaks newer lintr versions

DESCRIPTION

@@ -1,7 +1,7 @@
 Package: fly
 Title: Historic Airphoto Footprints, Selection and Georeferencing for
     British Columbia
-Version: 0.14.1
+Version: 0.15.0
 Date: 2026-09-21
 Authors@R: c(
     person("Allan", "Irvine", , "al@newgraphenvironment.com", role = c("aut", "cre"),

NEWS.md

@@ -1,5 +1,13 @@
 # fly (development version)
 
+## 0.15.0 (2026-09-26)
+
+- **Half of the lower tail of `flying_height` is now settled per roll, and 389 of those 1,001 frames were never a height error** ([#60](https://github.com/NewGraphEnvironment/fly/issues/60)). #54 left 1,962 film frames reading under half the height their scale implies as `"implausible"`, because ×10.764, ×10 and ×2 each explained a comparable share when pooled. Per roll they do not: the tail is 42 rolls, nearly each carrying one height, so it is 77 roll-heights rather than 1,962 frames. Two instruments that read none of the three disputed fields settle 22 of them (1,001 frames): the province's **scanned flight logbooks** at `flight_log_url`, whose TRUE HEIGHT column the catalogue transcribed, and the **spacing between frames adjacent by number**, which must imply the ~60% forward overlap a flight is designed to
+- New `height_source` value **`"corrected_roll_table"`**, read from the new `inst/extdata/flying_height_rolls.csv` and keyed on `film_roll`, `flying_height`, `focal_length` and `scale` together, which reaches exactly the 1,001 measured frames. Ten roll-heights (511 frames) dropped a digit, a logbook "20.0" in thousands of feet catalogued as 2,000 ft; one (`bc7280`, 101 frames) dropped two, 60 m catalogued where 20,000 ft was flown. **Eleven (389 frames) carry the right height beside a wrong `scale`**, which is a remedy the issue did not list: for those the fallback #54 chose was the defect, drawing them two to ten times too wide. The height used is always the logbook's, converted, never the catalogue's times the factor, since the catalogue's rounding survives the multiplication and left `bc7280` 96 m short. Needs a `film_roll` column and a `dem`; without a DEM a wrong-scale frame is still sized from its scale
+- **Every lower-tail roll-height the table does not correct is listed with its reason** in `flying_height_rolls_excluded.csv` (55 roll-heights, 961 frames — 30 with no logbook page, 8 where spacing rejects the logbook's height, 5 where the covering row was not read, 4 no named multiple, 4 where the logbook names a different lens and nominal scale is already right, 2 where covering rows disagree, 2 under half covered), so an unlisted roll reads as unmeasured rather than clean. The test suite recomputes every row from the shipped sweep and asserts the two files partition the census exactly; every one of the 77 labels was re-derived from the per-frame logbook states by an independent predicate, and all agree
+- **Both instruments were held to known answers first.** Spacing reads 0.63 on random in-band frames and 0.62 on #54's slipped frames after repair, against 0.97 before. The logbooks were transcribed with three control rolls' catalogue values withheld, and the two clean ones read back to the metre. The rule — logbook covers half the frames and 90% agree on a factor of 1, 10 or 100; no legible logbook lens contradicts the catalogue; spacing inside the random frames' own 2.5–97.5% — was fixed before any roll was classified. `data-raw/height_calibrate-lower_tail_rolls.R` reproduces all of it, and `data-raw/flying_height_logbooks.csv` is the transcription
+- **×10.764 is refused wherever a logbook reads the lower tail**, and the third control suggests it is not the whole story on the upper side either: #54's slipped roll `bc78065` logs 1,350 ft where the catalogue holds 13,500, a decimal point rather than a unit conversion, and the planned-altitude rolls #54 repairs land on round feet under ÷10 but not ÷10.764. Filed as [#71](https://github.com/NewGraphEnvironment/fly/issues/71), not changed here. And the note's claim that the mass beyond r ≈ 2 is a mislabelled lens, so nominal scale is right there, holds for about half of it: spacing puts 105 of its 209 sampled frames, identified per roll, at a wrong scale that the fallback draws at half width — [#72](https://github.com/NewGraphEnvironment/fly/issues/72)
+
 ## 0.14.1 (2026-09-21)
 
 - **`R CMD check` now runs in CI, on three platforms** ([#52](https://github.com/NewGraphEnvironment/fly/issues/52)). `.github/workflows/` held only `pkgdown.yaml`, so a green PR check meant the docs site built — not that the package checks, and not that the test suite runs anywhere but the author's machine. The new workflow runs on **ubuntu, macOS and Windows at R release**, on every push to `main` and every PR, gating at the r-lib action's default `error-on: "warning"`. No exported behaviour changes

R/fly_footprint.R

@@ -210,7 +210,9 @@ fly_height_slip_factor <- function() 3.28084^2
 # frames land at 0.80-1.32 once repaired against 10.0-15.8 before. The UPPER edge sits in
 # the trough between that mass and the next one out, centred on 2.0, which is a 305 mm lens
 # catalogued as 153 — a frame the DEM route would draw at twice its true width, and the
-# nominal route gets right because it never reads `focal_length`. The lower edge is that
+# nominal route gets right because it never reads `focal_length`. For about half of that
+# mass: fly#60's frame spacing shows 105 of its 209 sampled frames carry the right height
+# and a wrong scale, which the nominal route draws at half width (fly#72). The lower edge is that
 # factor inverted, because the error is a ratio either way; it is set by symmetry and not
 # by a trough, since ordinary frames simply thin out below 0.8 with no second mass at 0.5.
 # See `inst/notes/terrain-correction.md`.
@@ -226,6 +228,32 @@ fly_height_ratio_band <- function() c(1 / 1.6, 1.6)
 # a DEM window is ever built from the height in question.
 fly_flying_height_max <- function() 16000
 
+# Roll-heights whose `flying_height` disagreement was settled by measurement (fly#60).
+#
+# fly#54 left the lower tail — film frames reading under half the height their scale
+# implies — refused, because pooled, x10.764, x10 and x2 each explained a comparable share.
+# Per roll they do not: the lower tail is 42 rolls, nearly each carrying one height, and two
+# instruments that read none of the disputed fields settle 22 roll-heights — the scanned
+# logbooks at `flight_log_url`, whose TRUE HEIGHT column the catalogue transcribed, and the
+# spacing between frames adjacent by number, which must imply the ~60% forward overlap the
+# flight was designed to. `factor` names the defect: 10 or 100 where digits were dropped (a
+# logbook "20.0", thousands of feet, catalogued as 2,000 ft), 1 where the logbook confirms
+# the height and the SCALE is the wrong field. `height_m` is the height used — the logbook's,
+# converted, since the catalogue's rounding survives multiplying by the factor.
+#
+# Keyed on roll, height, lens and scale together, so the table reaches only the frames it
+# was measured on. Every lower-tail roll-height it does not correct is listed with its
+# reason in `flying_height_rolls_excluded.csv`: an unlisted roll is unmeasured, not clean.
+# Produced by `data-raw/height_calibrate-lower_tail_rolls.R`; see
+# `inst/notes/terrain-correction.md`.
+fly_height_roll_table <- function() {
+  path <- system.file("extdata/flying_height_rolls.csv", package = "fly")
+  if (!nzchar(path)) {
+    stop("`flying_height_rolls.csv` is missing from the installed package.", call. = FALSE)
+  }
+  utils::read.csv(path, colClasses = c(film_roll = "character", cause = "character"))
+}
+
 # The DEM-aligned grid a single footprint is counted against.
 #
 # Named and separate so the "one frame at a time" invariant can be asserted
@@ -655,6 +683,14 @@ fly_is_square <- function(footprints) {
 #'     `height_agl` is the height used. Your `flying_height` column is **not**
 #'     overwritten, so on these rows `flying_height - height_agl` is not the ground
 #'     elevation}
+#'   \item{`"corrected_roll_table"`}{they disagree, and the frame sits on one of
+#'     the roll-heights in `inst/extdata/flying_height_rolls.csv`, which the
+#'     province's flight logbooks and the spacing between adjacent frames settled
+#'     (fly#60): a height with one or two digits dropped (a logbook "20.0",
+#'     thousands of feet, catalogued as 2,000 ft), or a correct height beside a
+#'     wrong `scale`. Sized from the measured height; matched on `film_roll`,
+#'     `flying_height`, `focal_length` and `scale` together, so it needs a
+#'     `film_roll` column. As with the slip, `flying_height` is not overwritten}
 #'   \item{`"implausible"`}{they disagree some other way, or `flying_height` is
 #'     above 16,000 m or below the terrain. Nothing can say whether the height, the
 #'     scale or the focal length is the wrong one — a 305 mm lens catalogued as
@@ -1081,8 +1117,48 @@ fly_footprint <- function(centroids_sf, negative_size = 9, format_size = NULL,
     # `r_reported > 0`: terrain at or above the aircraft is already its own case below,
     # with its own warning, and stays there.
     disputed <- comparable & is.finite(r_reported) & r_reported > 0 & !in_band(r_reported)
-    slipped <- disputed & in_band(r_repaired)
+
+    # A frame on a roll-height the logbooks and frame spacing have settled (fly#60).
+    # Consulted before #54's repair, since it names the cause for these frames rather than
+    # inferring one. A height slip must still reconcile THIS frame's ratio, as #54's does;
+    # a factor of 1 says the scale is the wrong field, so the ratio against that scale is
+    # exactly what cannot be asked of it — but that height must still clear the ground.
+    #
+    # Not only `disputed` frames: a height with two digits dropped sits BELOW the terrain
+    # (bc7280 is catalogued at 60 m), which `disputed` hands to the terrain-above-aircraft
+    # case, so the table would never reach the frames it was measured on.
+    out_of_band <- comparable & is.finite(r_reported) & !in_band(r_reported)
+    tab_factor <- rep(NA_real_, n)
+    tab_height <- rep(NA_real_, n)
+    if ("film_roll" %in% names(centroids_sf) && any(out_of_band)) {
+      tab <- fly_height_roll_table()
+      # Every number formatted the same way on both sides. `read.csv()` gives the table's
+      # columns as integers and the frame's arrive as doubles, and `paste()` writes those
+      # differently once they are round enough: 100000L is "100000", 100000 is "1e+05".
+      num <- function(x) formatC(as.numeric(x), format = "f", digits = 3)
+      tab_key <- paste(tab$film_roll, num(tab$flying_height), num(tab$focal_length),
+                       num(tab$scale_n))
+      frame_key <- paste(as.character(centroids_sf$film_roll), num(fh),
+                         num(centroids_sf$focal_length), num(scale_num))
+      hit <- match(frame_key, tab_key)
+      # `match()` pairs NA with NA, and `paste()` turns NA into the string "NA", so a frame
+      # missing any key column is kept off the table explicitly rather than by luck.
+      keyed <- !is.na(centroids_sf$film_roll) & is.finite(fh) &
+        is.finite(centroids_sf$focal_length) & is.finite(scale_num)
+      hit[!keyed | !out_of_band] <- NA_integer_
+      tab_factor <- tab$factor[hit]
+      tab_height <- tab$height_m[hit]
+    }
+    # The logbook's height, converted — not the catalogue's times the factor, which carries
+    # the catalogue's rounding: bc7280's 20,000 ft is catalogued as 60 m, and x100 of that
+    # is 96 m short.
+    r_tabled <- (tab_height - first$elev) / nominal_agl
+    tabled <- out_of_band & is.finite(tab_factor) &
+      ((tab_factor == 1 & r_reported > 0) | (tab_factor > 1 & in_band(r_tabled)))
+
+    slipped <- disputed & !tabled & in_band(r_repaired)
     fh_used <- fh
+    fh_used[tabled] <- tab_height[tabled]
     fh_used[slipped] <- fh[slipped] / fly_height_slip_factor()
     # The ceiling needs no terrain, so it applies whether or not the DEM covers the frame \u2014
     # except to a frame that HAS a scale to be held against and could not be, because the
@@ -1092,7 +1168,7 @@ fly_footprint <- function(centroids_sf, negative_size = 9, format_size = NULL,
     # findable as `no_dem_coverage`, which is the documented way to learn the DEM needs to
     # be bigger.
     untested <- comparable & !is.finite(first$elev)
-    implausible <- (disputed & !slipped) |
+    implausible <- (disputed & !slipped & !tabled) |
       (dem_eligible & !untested & is.finite(fh_used) & fh_used > fly_flying_height_max())
     height_refused <- implausible
 
@@ -1212,6 +1288,18 @@ fly_footprint <- function(centroids_sf, negative_size = 9, format_size = NULL,
         call. = FALSE
       )
     }
+    roll_tabled <- corrected & tabled
+    if (any(roll_tabled)) {
+      warning(
+        sum(roll_tabled), " of ", sum(dem_eligible), " frames sit on a roll whose ",
+        "`flying_height` disagreement was settled by the flight logbooks and the spacing ",
+        "between frames (`flying_height_rolls.csv`): a height with dropped digits, or a ",
+        "correct height beside a wrong `scale`. Sized from the measured height; ",
+        "`flying_height` is left as supplied and `height_agl` is the height used. ",
+        "See `height_source`.",
+        call. = FALSE
+      )
+    }
     if (any(implausible)) {
       warning(
         sum(implausible), " of ", sum(dem_eligible), " frames have a `flying_height` that ",
@@ -1279,6 +1367,7 @@ fly_footprint <- function(centroids_sf, negative_size = 9, format_size = NULL,
 
     height_source[corrected] <- "reported"
     height_source[repaired] <- "corrected_unit_slip"
+    height_source[roll_tabled] <- "corrected_roll_table"
     height_source[implausible] <- "implausible"
     # Terrain at or above the aircraft is a height that was supplied and rejected, so it is
     # findable the same way. Missing or zero metadata is not: there was no height to doubt.