Part of SmartMet Server. See the SmartMet Server documentation for a full overview of the ecosystem.
The querydata engine (QEngine) provides shared access to gridded weather forecast data in QueryData format. It memory-maps data from NFS, supports spatial and temporal interpolation, DEM-based temperature correction, and automatic selection of the best data source for a requested region.
verbose = true/false- in verbose mode the engine will report newly loaded datamaxthreads = N- the number of threads used to read data on start up
cache.values_size = N- how many processed grids to cache, default is 5000cache.coordinates_size = N- how many projected grid coordinates to cache, default is 100cache.lat_lon_size = N- how many latlon grids to cache, default is 500
The maxthreads and verbose settings can be overridden for groups of hosts using an overrides group. Sample configuration:
overrides:
(
{
name = [ "test1", "test2.fmi" ]; # host name prefixes
maxthreads = 5; # do not use too many threads on test machines
verbose = true; # report what is being done on test machines
},
{
name = [ "super1", "super2" ];
maxthreads = 50; # use more threads on powerful servers
}
)
The engine can be loaded in a dummy mode when it is needed only because a plugin looks it
up at startup, and the plugin is not configured to use querydata. The dummy reads no
data: it reports an empty producer list, hasProducer() returns false, and data access
throws. There are several ways to enable it:
- set
disabled = truein the engine's configuration file (no other settings are read) - leave
configfileout of theengines.querydatasection of the server configuration - set
configfile = "";in that section
A log message is printed when the engine is disabled. Setting disabled = true in the
server's engines.querydata section is different: the server then does not load
querydata.so at all, and plugins that require the engine fail at startup.
The producers setting will list the producers in the order they will be used for finding a producer for the requested coordinates if no producer is otherwise set in the request. Producers may be grouped using an alias command to limit the search.
Sample configuration:
producers =
[
"local_default_model", # local high resolution model
"hirlam_europe",
"ecmwf_world", # this will match any coordinate
.... # and hence the remaining producers must be requested explicitly by name
]
Each producer is configured in more detail using a top level block as follows:
hirlam_scandinavia:
{
mmap = false;
alias = "hirlam";
directory = "/path/to/hirlam/scandinavia/querydata";
pattern = ".*_hirlam_scandinavia\.sqd$";
forecast = true;
type = "grid";
leveltype = "surface";
fullgrid = false;
refresh_interval_secs = 45;
number_to_keep = 2;
update_interval = "PT1H";
minimum_expires = "PT10M";
relative_uv = false;
};
The individual settings are as follows
alias- optional grouping for the producerdirectory- path to the querydata filespattern- required filename patternforecast- true for forecasts, false for observationsclimatology- false by default. If true, the data can be queried for any year, only the date part matterstype- (default: grid) grid, points. Some operations are permitted only for grids or points.leveltype- (default: surface) surface, pressure, model, points, or ensemble for legacy ensembles whose members are stored as levels (the EDR plugin then lists the levels as ensemble members)fullgrid- (default: true) true if data is valid for all points, saves speed when server startsstaticgrid- (default: false) true if the valid grid points never change, for example a land-only model, as opposed to data whose valid points change with the seasonmultifile- (default: false) treat all the files as one data set, for example observations split into daily fileslazy- (default: false) catalogue the files cheaply at start up and decode the data only when first accessed, used for radar producers, see docs/radar.mdmaxdistance- (default: -1, not set) producer specific maximum distance in kilometers when searching for a producer for a location, used when the caller allows itrefresh_interval_secs- (default: 60) How often to check the directory for changesnumber_to_keep- (default: 2) How many newest models to keep in the engine. Should be at least two for directories that change over time in server clusters. If the data is static and never updates, the value can be set to 1. When using "multifile mode" this value should be greater than two, or if old data is otherwise often requested using origintime-settings.update_interval- (default: 3600, minimum 60) Estimated update interval for the data, used for expiration headersminimum_expires- (default: 600) Minimum expiration header even though the model might be just a minute or two latemmap- true by default, often set to false for the most important local modelmax_age- time when the data should be dropped from the engine even if it still exists on the diskmax_latest_age- (default: off) the producer is skipped when searching for a producer for a location if its latest model is older than thisrelative_uv- are wind U- and V-components relative to the local grid orientation or east/north components
For historical reasons durations can be specified using ISO8601 or as simple offsets:
- 0, 0m, 0h (zero offset with or without units)
- (+|-){N}{unit} where N is a positive integer and unit is one of 'm' (minutes), 'h' (hours), 'w' (weeks) or 'y' (years). If the units are omitted, minutes will be used.
The Querydata Engine enables converting a numeric WeatherSymbol3 to a textual WeatherText parameter, and a SmartSymbol to a textual SmartSymbolText parameter, on the fly. The code assumes the translations are defined in the engine configuration for specific ISO2 language codes as follows (parameter_translations is accepted as the name of the group too):
language = "fi";
translations:
{
WeatherText:
(
{
value = 1;
en = "sunny";
sv = "klart";
fi = "selke\\u00e4\\u00e4";
},
{
value = 2;
en = "partly cloudy";
sv = "halvklart";
fi = "puolipilvist\\u00e4";
},
{
value = 3;
en = "cloudy";
sv = "mulet";
fi = "pilvist\\u00e4";
},
...
The SmartSymbolText translations are given the same way in a SmartSymbolText list. A value without a translation for the requested language is translated to the default language instead, and is missing if there is no translation for that either.
QEngine exposes several parameters that are not stored in the querydata files but computed on the fly from the underlying model fields. These include derived wind components, feels-like temperature, and weather symbol parameters.
The most notable is SmartSymbol, a single integer code that summarises the dominant weather condition (clear, cloudy, fog, drizzle, rain, sleet, snow, hail, thunderstorm) at a location and time. It is computed from cloud cover, precipitation rate, precipitation form, precipitation type, fog intensity, and thunderstorm probability. A day/night variant is produced by adding 100 when solar position indicates darkness. See docs/smartsymbol.md for a full description of the logic and symbol values.
SmartMet Server can be dockerized. This tutorial explains how to explains how to configure the querydata engine (QEngine) of the SmartMet Server when using Docker.
- Developer guide — models and producers, the plugin API, producer search, hash values and expiration, caches, reload, binary compatibility
- Radar producers
MIT — see LICENSE
Bug reports and pull requests are welcome on GitHub.