fix(createSSURGO): type conversion for logical column stored as Yes/No - #482
fix(createSSURGO): type conversion for logical column stored as Yes/No#482AndrewBrown-USDA wants to merge 1 commit into
Conversation
|
Unfortunately, this fix was not as general as I would like. While SQL dialects generally support Yes/No, 0/1, TRUE/FALSE equivalency, R does not. And different write implementations i.e. DuckDB vs. RSQLite handle logical values differently. This explicit logical conversion seems to work as expected with DuckDB (where it writes Yes/No logical values as TRUE/FALSE) whereas in SQLite they are written as 1/0. Either seems "fine" as a representation of logical until one realizes that many queries in the wild are written with Passing through R Metadata-based type conversion seemed like a good idea, but it isn't as currently (naively) implemented. This was essentially a workaround for not defining a template database with all tables and column data types declared a priori (like SSURGO Portal does). If the schema was more consistent maybe it would be OK to do but we are running up against R vs various SQL dialect interpretation of logicals. I am thinking removing the type conversion added in #477 entirely may be best, but I hesitate there as well. This was designed to make sure we had conformant data types when appending to existing tables, notably where the table creation is initialized with incomplete/empty source data that may not have sufficient data for DBI packages to use heuristics to identify the relevant data type (something we heavily rely on in lieu of template CREATE TABLE statements and to support various backends). Left entirely unmanaged can lead to inconsistency between builds depending on which tables and SSAs you added first. Possibly just need to handle logical differently than other data types, knowing the schema itself is inconsistent. Back to the drawing board! |
- some are Yes/No, some are 0/1 in source; best to leave them alone! cf. #482
No description provided.