Skip to content

fix(createSSURGO): type conversion for logical column stored as Yes/No - #482

Closed
AndrewBrown-USDA wants to merge 1 commit into
masterfrom
fix/createssurgo-lgl1
Closed

fix(createSSURGO): type conversion for logical column stored as Yes/No#482
AndrewBrown-USDA wants to merge 1 commit into
masterfrom
fix/createssurgo-lgl1

Conversation

@AndrewBrown-USDA

Copy link
Copy Markdown
Collaborator

No description provided.

@brownag brownag closed this Sep 5, 2026
@AndrewBrown-USDA

Copy link
Copy Markdown
Collaborator Author

Unfortunately, this fix was not as general as I would like.

While SQL dialects generally support Yes/No, 0/1, TRUE/FALSE equivalency, R does not. And different write implementations i.e. DuckDB vs. RSQLite handle logical values differently.

This explicit logical conversion seems to work as expected with DuckDB (where it writes Yes/No logical values as TRUE/FALSE) whereas in SQLite they are written as 1/0.

Either seems "fine" as a representation of logical until one realizes that many queries in the wild are written with some_lgl_val = 'Yes', which works properly on DuckDB (when stored as T/F) but not in SQLite (when stored as 1/0). :/

Passing through R as.logical() converts Yes/No to NA; which snuck by me in the #477 which included enforcement of declared data model types. As it turns out some Yes/No columns in SSURGO are not actually coded in the metadata as "logical" but some really important/common ones like majcompflag are.

Metadata-based type conversion seemed like a good idea, but it isn't as currently (naively) implemented. This was essentially a workaround for not defining a template database with all tables and column data types declared a priori (like SSURGO Portal does). If the schema was more consistent maybe it would be OK to do but we are running up against R vs various SQL dialect interpretation of logicals.

I am thinking removing the type conversion added in #477 entirely may be best, but I hesitate there as well. This was designed to make sure we had conformant data types when appending to existing tables, notably where the table creation is initialized with incomplete/empty source data that may not have sufficient data for DBI packages to use heuristics to identify the relevant data type (something we heavily rely on in lieu of template CREATE TABLE statements and to support various backends). Left entirely unmanaged can lead to inconsistency between builds depending on which tables and SSAs you added first. Possibly just need to handle logical differently than other data types, knowing the schema itself is inconsistent. Back to the drawing board!

AndrewBrown-USDA added a commit that referenced this pull request Sep 5, 2026
- some are Yes/No, some are 0/1 in source; best to leave them alone! cf. #482
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants