diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 0b16ac0eb8..a0327fcf93 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -22,7 +22,7 @@ v1.1.0 | September XX, 2026 New features ------------- -* Automations: recurring tasks defined per asset, which compute forecasts or schedules on a cron recurrence read in the automation's own timezone, defined from the CLI, the API or the UI, and dispatched once a minute by ``flexmeasures jobs run-automations``. See :ref:`automations` for what they do, and `Automations, in detail`_ for what each pull request contributed. +* Automations: recurring tasks defined per asset, which compute forecasts, schedules or reports on a cron recurrence read in the automation's own timezone, defined from the CLI, the API or the UI, and dispatched once a minute by ``flexmeasures jobs run-automations``. See :ref:`automations` for what they do, and `Automations, in detail`_ for what each pull request contributed. * In the UI, the full record of the data source selected on a sensor page can be inspected, backed by a new API endpoint (``[GET] /sources/(id)``) [see `PR #2290 `_] * Try out a forecast without recording it, using ``flexmeasures add forecasts --dry-run``, which computes the forecast in full and reports the sensor, data source, number of beliefs and event range it would have saved [see `PR #2483 `_] * Run one-off reports as background jobs from the CLI or the asset API, with sensor-level authorization and a dedicated reporting worker queue [see `PR #2298 `_] @@ -67,6 +67,7 @@ Automations arrived over several pull requests. This is what each of them contri * A single automation can now be run on demand, from the CLI (``flexmeasures jobs run-automation``), the API (``POST /assets//automations//trigger``) and the asset's *Automations* page (a *Run now* button), which is useful to try out a new automation, to re-run one after fixing what made it fail, or to refresh its results after late input data arrived [see `PR #2460 `_] * Automations can be created, edited and deleted in the UI and through new API endpoints (``[POST|PATCH|DELETE] /assets/(id)/automations``), by whoever may add data under the asset, with their recurrence expressed in a selectable IANA timezone, and only involving sensors they can access themselves (read access to the sensors an automation reads, and permission to record data on the sensors it writes to). The *Automations* page shows when each active automation is due to run next, as a clock time in the automation's own timezone, so a recurrence no longer has to be read back from its cron string; the same time is available as ``next_run`` on the automations API endpoints, and is null while an automation is inactive. It is the next scheduled clock time, so it excludes catch-up work still pending, and it follows the dispatcher's daylight-saving rules, taking the first fold of a repeated local time and the first valid minute after a skipped one. Each row's *Run now*, *Edit*, *Activate*/*Deactivate* and *Delete* controls are collected into a single *Actions* menu, leaving *Details* beside it, so a long listing carries two controls per row instead of five [see `PR #2294 `_] * An automation's output sensors are checked against its creator's permissions when the automation is created, and the schedules it computes are held to exactly those sensors: a scheduler that returns results for any other sensor is refused, rather than recording on a sensor that was only ever checked for read access [see `PR #2536 `_] +* Reports as well as forecasts and schedules: a report automation stores report parameters, with its reporter and the reporter's configuration on a data source, and reports on a period resolved afresh on each run, either from ``start-offset`` and ``end-offset`` applied to the run time in the automation's timezone, or since the last successful report ended, while a fixed ``start`` or ``end`` is refused; a report job records only on the sensors the automation was checked against [see `PR #2297 `_] v1.0.1 | September 9, 2026 diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst index 26bfbdb900..865e21c09d 100644 --- a/documentation/cli/change_log.rst +++ b/documentation/cli/change_log.rst @@ -10,6 +10,12 @@ since v1.1.0 | September XX, 2026 * Fix ``flexmeasures add schedule --dry-run`` saving a schedule when combined with ``--as-job``: the combination is now rejected, and a dry run reports the beliefs and events it would have saved. * Add ``flexmeasures add forecasts --dry-run`` to compute a forecast and see what it would record, without saving anything to the database. It cannot be combined with ``--as-job``. * Add ``flexmeasures add report --as-job`` and the ``reporting`` worker queue for asynchronous one-off reports. +* Add ``flexmeasures add automation``, ``flexmeasures edit automation`` and ``flexmeasures delete automation`` to manage automations (recurring tasks on an asset, with ``--type forecasting``, ``--type scheduling`` or ``--type reporting`` saying which task to automate). Each automation carries its own IANA timezone (``--timezone``), in which its cron expression is interpreted. +* ``flexmeasures add automation --type scheduling`` refuses a flex config field which fixes a moment in time, such as ``soc-at-start`` or a ``soc-targets`` entry with a ``datetime``, naming the field: a recurring schedule automation computes a fresh schedule on every run, so such a value would be stale on the next one. +* ``flexmeasures add automation --type reporting`` refuses a fixed ``start`` or ``end`` in the report parameters, as every run would then report on the same period, and points to ``start-offset`` and ``end-offset`` instead. +* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. It makes at most one queueing attempt per automation per minute, including when an attempt fails after partially queueing jobs. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once. +* Add ``flexmeasures jobs run-automation --automation `` to queue the jobs for a single run of one automation, now, on top of its recurring runs. This leaves the automation's cursor alone, so its next recurring run still happens as scheduled, and inactive automations can be run this way, too. +* ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run. since v1.0.1 | September 9, 2026 ================================= @@ -24,11 +30,6 @@ since v1.0.0 | August 11, 2026 * Add ``flexmeasures add plan``, ``flexmeasures show plans`` and ``flexmeasures edit plan``, to manage the rate limits and quotas which apply to the accounts on a plan. * Add ``flexmeasures edit secret`` to store an encrypted secret on an account or asset. * Add ``flexmeasures delete secret`` to remove an encrypted secret from an account or asset. -* Add ``flexmeasures add automation``, ``flexmeasures edit automation`` and ``flexmeasures delete automation`` to manage automations (recurring tasks on an asset, with ``--type forecasting`` or ``--type scheduling`` saying which task to automate). Each automation carries its own IANA timezone (``--timezone``), in which its cron expression is interpreted. -* ``flexmeasures add automation --type scheduling`` refuses a flex config field which fixes a moment in time, such as ``soc-at-start`` or a ``soc-targets`` entry with a ``datetime``, naming the field: a recurring schedule automation computes a fresh schedule on every run, so such a value would be stale on the next one. -* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. It makes at most one queueing attempt per automation per minute, including when an attempt fails after partially queueing jobs. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once. -* Add ``flexmeasures jobs run-automation --automation `` to queue the jobs for a single run of one automation, now, on top of its recurring runs. This leaves the automation's cursor alone, so its next recurring run still happens as scheduled, and inactive automations can be run this way, too. -* ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run. * ``flexmeasures show data-sources`` now shows the account a data source belongs to, and lists the sensors holding data recorded by a single source with ``--show-sensors``. since v0.33.0 | June 01, 2026 diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst index 7d81a1e309..205b9c41d6 100644 --- a/documentation/cli/commands.rst +++ b/documentation/cli/commands.rst @@ -41,7 +41,7 @@ of which some are referred to in this documentation. ``flexmeasures add annotation`` Add annotation to accounts, assets and/or sensors. ``flexmeasures add toy-account`` Create a toy account, for tutorials and trying things. ``flexmeasures add report`` Create a report. -``flexmeasures add automation`` Add an automation: a recurring task (computing forecasts or schedules) on an asset, with its own cron timezone. +``flexmeasures add automation`` Add an automation: a recurring task (computing forecasts, schedules or reports) on an asset, with its own cron timezone. ================================================= ======================================= diff --git a/documentation/features/automations.rst b/documentation/features/automations.rst index 91b9e1ea12..559a4b519d 100644 --- a/documentation/features/automations.rst +++ b/documentation/features/automations.rst @@ -4,9 +4,9 @@ Automations ============ An **automation** is a recurring task defined on an asset. -For now, an automation computes forecasts or schedules; automating reports is planned. +An automation computes forecasts, schedules or reports. -On each run, the automation queues jobs (so make sure a worker is processing the ``forecasting`` or ``scheduling`` queue, whichever the automation needs, see :ref:`redis-queue`). +On each run, the automation queues jobs (so make sure a worker is processing the ``forecasting``, ``scheduling`` or ``reporting`` queue, whichever the automation needs, see :ref:`redis-queue`). The parameters of the task were stored when the automation was created, and validated with the same schema that the CLI and API use. Timing parameters are resolved on each run — for instance, the forecast or schedule start defaults to the time the automation runs, so each run produces fresh results. @@ -20,7 +20,7 @@ Here is how you create an automation in the CLI, asking for daily (at 6 AM) fore flexmeasures add automation --asset 3 --name "Daily PV forecasts" --type forecasting \ --cron "0 6 * * *" --timezone Europe/Amsterdam --sensor 12 -``--type`` says which task to automate (``forecasting`` or ``scheduling``, matching the queue the jobs go to), and defaults to ``forecasting``. +``--type`` says which task to automate (``forecasting``, ``scheduling`` or ``reporting``, matching the queue the jobs go to), and defaults to ``forecasting``. The remaining options are the ones the task itself needs: a forecast automation accepts everything `flexmeasures add forecast` accepts, such as ``--forecaster`` to pick the forecaster and ``--config`` to configure it (see :ref:`forecasting`). The forecaster and its configuration are stored on a data source, so you can also pass ``--source`` to reuse the data source of an existing forecaster, in which case ``--forecaster`` and ``--config`` (and the individual configuration options) are not needed — the data source already determines them. That data source is required while the automation exists, so it cannot be deleted until the automation is removed. @@ -68,6 +68,39 @@ For example, this automation queues a scheduling job every hour, each time sched echo 'duration: "PT12H"' > trigger-message.yml flexmeasures add automation --asset 3 --name "Hourly schedules" --cron "0 * * * *" --type scheduling --parameters trigger-message.yml +Automating reports +------------------ + +A report automation's parameters are report parameters, as ``flexmeasures add report`` accepts them (see :ref:`reporting`), passed in a file through ``--parameters``. +Name the reporter with ``--reporter``, and configure it with ``--config``. +As for a forecast automation, the reporter and its configuration are stored on a data source, so ``--source`` can reuse the data source of an existing reporter instead. + +The sensors a report is recorded on must belong to the automation's asset or one of its descendants. +This is checked when the automation is created and immediately before each run. +A report job only records on those sensors, so a reporter that returns results for any other sensor is refused. + +Because the report is computed afresh on every run, its parameters cannot fix the period it covers: an absolute ``start`` or ``end`` is refused when the automation is created. +Say instead how the period relates to the run, in one of two ways: + +- Give ``start-offset`` and ``end-offset``, as comma-separated Pandas offsets plus ``DB`` (day begin) and ``HB`` (hour begin), applied to the run time on the automation's own clock, the one its cron string is read in. + For instance, ``start-offset: "-1D,DB"`` with ``end-offset: "DB"`` reports on the whole of the previous day. + Leave out ``end-offset`` to report up to the run time. +- Leave the timing out to report on the period since the last successful report ended, falling back to the previous cron period on the first run. + That end is recorded by the reporting job itself, once it has succeeded, so a failed report leaves no gap: the next run starts where the last successful one ended. + +The two differ after the runner was down. +It catches up with only the latest missed run (see below), so offsets then report on the period around that run alone, +while leaving the timing out reports on everything since the last successful report. + +For example, this automation reports on the previous day, with the offsets above in ``report-parameters.yml``. +It runs every night at 1 AM, an hour after midnight, so that the day's last readings have had time to arrive: + +.. code-block:: bash + + flexmeasures add automation --asset 3 --name "Daily self-consumption report" --type reporting \ + --cron "0 1 * * *" --timezone Europe/Amsterdam \ + --reporter PandasReporter --config reporter-config.yml --parameters report-parameters.yml + Running automations ------------------- @@ -83,7 +116,7 @@ Use one dispatcher for a deployment; the Docker Compose service already runs the Each due automation then queues its jobs. If the runner misses runs, because it was down or overloaded, it catches up when it resumes: it queues only the latest missed run of each automation, rather than replaying stale ones. -Timing parameters that default to the run time are resolved when that catch-up run is queued, so it produces a current forecast or schedule. +Timing parameters that default to the run time are resolved when that catch-up run is queued, so it produces a current forecast, schedule or report. Each scheduled run receives at most one automatic queueing attempt. If the process crashes, or queueing fails after creating some jobs, that run is not retried automatically, because a retry could duplicate partial work. diff --git a/documentation/features/reporting.rst b/documentation/features/reporting.rst index 5262af925c..cf4bff86bf 100644 --- a/documentation/features/reporting.rst +++ b/documentation/features/reporting.rst @@ -129,3 +129,11 @@ Here, the ``ProfitOrLossReporter`` used as source (with Id 6) is the one we conf With the offsets, we control the timing ― we indicate that we want the new report to encompass the day of tomorrow (see Pandas offset strings). The report sensor will now store all costs which we know will be made tomorrow by the schedule. + +.. _automating_reports: + +Automating reports +-------------------- + +Instead of computing reports one at a time, you can set up an *automation*: a recurring task defined on an asset, which queues reporting jobs on a cron schedule. +See :ref:`automations`. diff --git a/flexmeasures/api/v3_0/tests/test_automations_api.py b/flexmeasures/api/v3_0/tests/test_automations_api.py index d54514fb4e..c3020272b6 100644 --- a/flexmeasures/api/v3_0/tests/test_automations_api.py +++ b/flexmeasures/api/v3_0/tests/test_automations_api.py @@ -109,7 +109,7 @@ def test_get_automations( assert intraday["next-run"] is None # generator and parameters are not listed assert "generator_id" not in day_ahead - assert "generator" not in day_ahead + assert "source" not in day_ahead assert "parameters" not in day_ahead @@ -230,6 +230,170 @@ def test_post_automation( fresh_db.session.flush() +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user_2@seita.nl"], indirect=True +) +def test_post_automation_with_foreign_sensor( + app, + db, + setup_accounts, + add_battery_assets, + requesting_user, +): + """Referencing a sensor outside the caller's reach is forbidden.""" + from datetime import timedelta + + from flexmeasures.data.models.generic_assets import GenericAsset + from flexmeasures.data.models.time_series import Sensor + + battery = add_battery_assets["Test battery"] + foreign_asset = GenericAsset( + name="Foreign asset", + generic_asset_type=battery.generic_asset_type, + owner=setup_accounts["Dummy"], + ) + foreign_sensor = Sensor( + "foreign power", + generic_asset=foreign_asset, + event_resolution=timedelta(minutes=15), + unit="MW", + ) + db.session.add(foreign_sensor) + db.session.flush() + with app.test_client() as client: + response = client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={ + "name": "Sneaky forecasts", + "cron": "0 6 * * *", + "type": "forecasting", + "parameters": {"sensor": foreign_sensor.id}, + }, + ) + assert response.status_code == 403 + assert ( + db.session.execute( + select(Automation).filter_by(name="Sneaky forecasts") + ).scalar_one_or_none() + is None + ) + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user_2@seita.nl"], indirect=True +) +def test_post_report_automation_with_foreign_config_sensor( + app, + db, + setup_accounts, + add_battery_assets, + requesting_user, +): + """Reporter configuration may not read a sensor outside the caller's reach.""" + from flexmeasures.data.models.generic_assets import GenericAsset + + battery = add_battery_assets["Test battery"] + foreign_asset = GenericAsset( + name="Foreign price asset", + generic_asset_type=battery.generic_asset_type, + owner=setup_accounts["Dummy"], + ) + foreign_price_sensor = Sensor( + "private foreign price", + generic_asset=foreign_asset, + event_resolution=timedelta(hours=1), + unit="EUR/MWh", + ) + report_sensor = Sensor( + "profit report", + generic_asset=battery, + event_resolution=timedelta(hours=1), + unit="EUR", + ) + db.session.add_all([foreign_price_sensor, report_sensor]) + db.session.flush() + + with app.test_client() as client: + response = client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={ + "name": "Cross-organisation profit report", + "cron": "0 1 * * *", + "type": "reporting", + "data-generator": "ProfitOrLossReporter", + "config": { + "consumption_price_sensor": foreign_price_sensor.id, + }, + "parameters": { + "input": [{"sensor": battery.sensors[0].id}], + "output": [{"sensor": report_sensor.id}], + }, + }, + ) + + assert response.status_code == 403 + assert foreign_price_sensor.name not in response.text + assert ( + db.session.execute( + select(Automation).filter_by(name="Cross-organisation profit report") + ).scalar_one_or_none() + is None + ) + db.session.commit() + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user_2@seita.nl"], indirect=True +) +def test_post_report_automation_rejects_output_outside_asset_subtree( + app, + db, + add_battery_assets, + requesting_user, +): + """Report output must stay on the automation asset or a descendant.""" + battery = add_battery_assets["Test battery"] + sibling_battery = add_battery_assets["Test small battery"] + report_sensor = Sensor( + "sibling report output", + generic_asset=sibling_battery, + event_resolution=timedelta(hours=1), + unit="MW", + ) + db.session.add(report_sensor) + db.session.flush() + + with app.test_client() as client: + response = client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={ + "name": "Misplaced report output", + "cron": "0 1 * * *", + "type": "reporting", + "data-generator": "PandasReporter", + "config": { + "required_input": [{"name": "flow"}], + "required_output": [{"name": "copied_flow"}], + "transformations": [ + { + "df_input": "flow", + "df_output": "copied_flow", + "method": "copy", + } + ], + }, + "parameters": { + "input": [{"name": "flow", "sensor": battery.sensors[0].id}], + "output": [{"name": "copied_flow", "sensor": report_sensor.id}], + }, + }, + ) + + assert response.status_code == 422 + assert "must belong to asset" in response.text + db.session.commit() + + @pytest.mark.parametrize( "requesting_user", ["test_prosumer_user_2@seita.nl"], indirect=True ) diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py index 4ed0ef0039..c9c80f8feb 100755 --- a/flexmeasures/cli/data_add.py +++ b/flexmeasures/cli/data_add.py @@ -1458,7 +1458,8 @@ def _assemble_forecaster_config_and_parameters( config = _load_yaml_mapping(config_file, "--config") for field_name, field in TrainPredictPipelineConfigSchema._declared_fields.items(): field_value = kwargs.pop(field_name, None) - if field_value is not None: + # Skip unset options: click passes None, or an empty tuple for a multiple-value option. + if field_value is not None and field_value != (): if field_name in { "future_regressors", "past_regressors", @@ -1510,8 +1511,8 @@ def _assemble_forecaster_config_and_parameters( if kebab_key not in parameters: parameters[kebab_key] = v - # Drop None values - parameters = {k: v for k, v in parameters.items() if v is not None} + # Drop unset values + parameters = {k: v for k, v in parameters.items() if v is not None and v != ()} return config, parameters @@ -1788,15 +1789,15 @@ def add_forecast( # noqa: C901 "source", required=False, type=DataSourceIdField(), - help="DataSource ID of the `Forecaster`. The forecaster class and its configuration are read from" - " the data source's data generator attributes, so --forecaster and --config are not needed (or allowed) with it.", + help="DataSource ID of the data generator (`Forecaster` or `Reporter`). The generator class and its configuration are read from" + " the data source's attributes, so --forecaster/--reporter and --config are not needed (or allowed) with it.", ) @click.option( "--config", "config_file", required=False, type=click.File("r"), - help="Path to the JSON or YAML file with the configuration of the forecaster." + help="Path to the JSON or YAML file with the configuration of the forecaster or reporter." " Cannot be combined with --source, which already determines the configuration.", ) @click.option( @@ -1805,7 +1806,8 @@ def add_forecast( # noqa: C901 required=False, type=click.File("r"), help="Path to the JSON or YAML file with the parameters used on each run of the automation:" - " forecast parameters for --type forecasting, or a schedule trigger message for --type scheduling.", + " forecast parameters for --type forecasting, a schedule trigger message for --type scheduling," + " or report parameters for --type reporting.", ) @add_cli_options_from_schema( ForecasterParametersSchema(), hidden=True, force_optional=True @@ -1827,7 +1829,7 @@ def add_automation( **kwargs, ): """ - Add an automation: a recurring task (computing forecasts or schedules) on an asset. + Add an automation: a recurring task (computing forecasts, schedules or reports) on an asset. \b Examples @@ -1836,12 +1838,17 @@ def add_automation( --parameters forecast-parameters.yml flexmeasures add automation --asset 3 --name "Hourly schedules" --cron "0 * * * *" --type scheduling --parameters trigger-message.yml - - For forecasts, the forecaster configuration is stored on a data source, and - the forecast parameters are validated and stored on the automation itself. - For schedules, the parameters form a schedule trigger message (as accepted by - the [POST] /assets/(id)/schedules/trigger API endpoint, without the asset id); - omit its "start" field to schedule from the run time on each run. + flexmeasures add automation --asset 3 --name "Daily self-consumption report" + --cron "0 1 * * *" --type reporting --reporter PandasReporter + --config reporter-config.yml --parameters report-parameters.yml + + For forecasts and reports, the data generator configuration is stored on a data source, + and the parameters are validated and stored on the automation itself. + For schedules, the parameters form a schedule trigger message, as accepted by the [POST] /assets/(id)/schedules/trigger API endpoint, + without the asset id; omit its "start" field to schedule from the run time on each run. + For reports, a fixed "start" or "end" is refused: use "start-offset" and "end-offset", + which take comma-separated Pandas offsets applied to the run time in the automation's timezone, + or leave the timing out to report on the period since the last successful report. Each time the automation runs, jobs are queued (see `flexmeasures jobs run-automations`). Alternatively, pass an existing data source (--source) to reuse the forecaster @@ -1854,7 +1861,9 @@ def add_automation( A configuration option given on the command line overrides the same setting from --config, while a parameter from --parameters takes precedence over the matching command-line option. """ - if generator_class is None: + # Only a forecast automation has a default generator: + # a report automation has to name its reporter, and the service says so, while a schedule automation resolves its own. + if generator_class is None and automation_type == "forecasting": generator_class = "TrainPredictPipeline" config, parameters = _assemble_forecaster_config_and_parameters( diff --git a/flexmeasures/cli/jobs.py b/flexmeasures/cli/jobs.py index c674267914..c45a111e48 100644 --- a/flexmeasures/cli/jobs.py +++ b/flexmeasures/cli/jobs.py @@ -116,7 +116,9 @@ def run_automations(): ) continue try: - returns = run_automation(automation) + returns = run_automation( + automation, scheduled_at=due_automation.scheduled_at + ) n_jobs = returns.get("n_jobs") if returns else 0 click.secho( f"Automation {automation.id} ('{automation.name}') queued {n_jobs} {automation.type} job(s) for asset {automation.asset_id}.", diff --git a/flexmeasures/cli/tests/test_automations.py b/flexmeasures/cli/tests/test_automations.py index 337af62d27..506f9f47a2 100644 --- a/flexmeasures/cli/tests/test_automations.py +++ b/flexmeasures/cli/tests/test_automations.py @@ -2,6 +2,7 @@ import json import pytest +import yaml import pytz from types import SimpleNamespace @@ -1124,6 +1125,351 @@ class FakeJob: assert calls["kwargs"]["end"] - start == timedelta(hours=12) +def test_prepare_report_parameters(app): + """Report start/end resolve per run: from Pandas offsets, or defaulting to the last cron period.""" + import pandas as pd + + from flexmeasures.data.services.automations import prepare_report_parameters + + now = pd.Timestamp("2026-07-11T14:00:00+02:00") + + # default: the last cron period (hourly cron -> the previous hour) + message = prepare_report_parameters({}, "0 * * * *", "Europe/Amsterdam", now=now) + assert pd.Timestamp(message["start"]) == now - pd.Timedelta(hours=1) + assert pd.Timestamp(message["end"]) == now + + # The fallback cron period is interpreted in the automation timezone, + # and ends at the claimed run rather than at a delayed runner's wall time. + scheduled_at = datetime(2026, 1, 1, 16, 0, tzinfo=timezone.utc) + message = prepare_report_parameters( + {}, + "0 1 * * *", + "Asia/Seoul", + now=datetime(2026, 1, 2, 0, 30, tzinfo=timezone.utc), + scheduled_at=scheduled_at, + ) + assert pd.Timestamp(message["start"]) == pd.Timestamp("2025-12-31T16:00:00+00:00") + assert pd.Timestamp(message["end"]) == pd.Timestamp(scheduled_at) + + # A cron run in Amsterdam's spring gap is canonicalized to 03:00, + # while its report starts at the prior day's real 02:30 run. + spring_run = datetime(2026, 3, 29, 1, 0, tzinfo=timezone.utc) + message = prepare_report_parameters( + {}, + "30 2 * * *", + "Europe/Amsterdam", + scheduled_at=spring_run, + ) + assert pd.Timestamp(message["start"]) == pd.Timestamp("2026-03-28T01:30:00+00:00") + assert pd.Timestamp(message["end"]) == pd.Timestamp(spring_run) + + # with a known actual last run, the window starts there instead + app.redis_connection.set("automation-last-run:1234", "2026-07-11T09:30:00+02:00") + try: + message = prepare_report_parameters( + {}, "0 * * * *", "Europe/Amsterdam", now=now, automation_id=1234 + ) + assert pd.Timestamp(message["start"]) == pd.Timestamp( + "2026-07-11T09:30:00+02:00" + ) + assert pd.Timestamp(message["end"]) == now + # an unknown automation id still falls back to the last cron period + message = prepare_report_parameters( + {}, "0 * * * *", "Europe/Amsterdam", now=now, automation_id=5678 + ) + assert pd.Timestamp(message["start"]) == now - pd.Timedelta(hours=1) + finally: + app.redis_connection.delete("automation-last-run:1234") + + # Offsets are applied to the run time on the automation's own clock, so "DB" is midnight in its timezone. + # 14:00 in Amsterdam is 21:00 in Seoul, where the previous day ran from midnight to midnight Seoul time, + # which in Amsterdam is 17:00 to 17:00, rather than Amsterdam's own midnight. + message = prepare_report_parameters( + {"start-offset": "-1D,DB", "end-offset": "DB"}, + "0 1 * * *", + "Asia/Seoul", + now=now, + ) + assert pd.Timestamp(message["start"]) == pd.Timestamp("2026-07-10T00:00:00+09:00") + assert pd.Timestamp(message["end"]) == pd.Timestamp("2026-07-11T00:00:00+09:00") + assert "start-offset" not in message and "end-offset" not in message + + +def test_report_coverage_cannot_move_backwards(app, clean_redis): + """An older report finishing later may not reopen already covered periods.""" + from flexmeasures.data.services.automations import ( + get_automation_last_run, + record_automation_run, + ) + + later_end = datetime(2026, 1, 3, tzinfo=timezone.utc) + older_end = datetime(2026, 1, 2, tzinfo=timezone.utc) + + assert record_automation_run(42, later_end) is True + assert record_automation_run(42, older_end) is False + assert get_automation_last_run(42) == later_end + + +def _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + parameters_extra=None, + asset_id=1, +): + """CLI input for a report automation using a simple PandasReporter aggregation.""" + reporter_config = dict( + required_input=[{"name": "sensor_1"}, {"name": "sensor_2"}], + required_output=[{"name": "df_agg"}], + transformations=[ + dict( + df_input="sensor_1", + method="add", + args=["@sensor_2"], + df_output="df_agg", + ), + dict(method="resample_events", args=["2h"]), + ], + ) + parameters = dict( + input=[ + dict(name="sensor_1", sensor=sensor1_id), + dict(name="sensor_2", sensor=sensor2_id), + ], + output=[dict(name="df_agg", sensor=report_sensor_id)], + **(parameters_extra or {}), + ) + config_file = tmp_path / "reporter_config.yml" + config_file.write_text(yaml.dump(reporter_config)) + parameters_file = tmp_path / "parameters.yml" + parameters_file.write_text(yaml.dump(parameters)) + return [ + "--asset", str(asset_id), + "--name", "Aggregation report", + "--cron", "0 1 * * *", + "--type", "reporting", + "--reporter", "PandasReporter", + "--config", str(config_file), + "--parameters", str(parameters_file), + ] # fmt: skip + + +def test_add_report_automation(app, fresh_db, setup_dummy_data, tmp_path): + """Create a reports automation; the reporter config lands on a data source.""" + from flexmeasures.cli.data_add import add_automation + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data + from flexmeasures.data.models.time_series import Sensor + + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + runner = app.test_cli_runner() + result = runner.invoke( + add_automation, + _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + parameters_extra={"start-offset": "-1D,DB", "end-offset": "DB"}, + asset_id=report_sensor.generic_asset_id, + ), + ) + assert "Successfully created" in result.output, result.output + automation = fresh_db.session.execute(select(Automation)).scalar_one() + assert automation.type == "reporting" + assert automation.generator is not None + assert automation.generator.model == "PandasReporter" + assert automation.parameters["start-offset"] == "-1D,DB" + + # a reports automation without a reporter is rejected + result = runner.invoke( + add_automation, + [ + "--asset", "1", + "--name", "No reporter", + "--cron", "0 1 * * *", + "--type", "reporting", + ], + ) # fmt: skip + assert result.exit_code != 0 + assert "reporter is required" in result.output + + # invalid time offsets are rejected (they would otherwise be silently skipped at run time) + result = runner.invoke( + add_automation, + _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + parameters_extra={ + "start-offset": "P1D,DB" + }, # ISO duration, not a Pandas offset + ), + ) + assert result.exit_code != 0 + assert "Invalid start-offset" in result.output + + +@pytest.mark.parametrize( + "fixed_timing", + [ + {"start": "2023-04-10T00:00:00+00:00"}, + {"end": "2023-04-10T10:00:00+00:00"}, + {"start-offset": "-1D,DB", "end": "2023-04-10T10:00:00+00:00"}, + ], +) +def test_report_automation_refuses_a_fixed_period( + app, fresh_db, setup_dummy_data, tmp_path, fixed_timing +): + """A report automation may not fix its window, or every run would report on the same period.""" + from flexmeasures.cli.data_add import add_automation + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data + result = app.test_cli_runner().invoke( + add_automation, + _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + parameters_extra=fixed_timing, + ), + ) + assert result.exit_code != 0 + assert "every run would then report on the same period" in result.output + assert fresh_db.session.scalars(select(Automation)).first() is None + + +def test_run_report_automation( + app, fresh_db, setup_dummy_data, clean_redis, tmp_path, freeze_server_now +): + """A due reports automation queues a reporting job; a worker computes and saves the report.""" + from flexmeasures.cli.data_add import add_automation + from flexmeasures.cli.jobs import run_automations + from flexmeasures.data.models.time_series import Sensor + from flexmeasures.utils.job_utils import work_on_rq + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + runner = app.test_cli_runner() + cli_input = _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + # report on today so far, which, with the clock frozen below, is the dummy data's day + parameters_extra={"start-offset": "DB"}, + asset_id=report_sensor.generic_asset_id, + ) + cli_input[cli_input.index("0 1 * * *")] = "* * * * *" # due every minute + cli_input += ["--timezone", "UTC"] + # the dummy data lives in April 2023 + freeze_server_now(datetime(2023, 4, 10, 10, 0, 30, tzinfo=timezone.utc)) + result = runner.invoke(add_automation, cli_input) + assert "Successfully created" in result.output, result.output + automation = fresh_db.session.execute(select(Automation)).scalar_one() + + result = runner.invoke(run_automations) + assert result.exit_code == 0, result.output + assert "queued 1 reporting job(s)" in result.output, result.output + + # the queued job recorded how it was created + jobs = app.queues["reporting"].jobs + assert len(jobs) == 1 + assert jobs[0].meta["trigger"] == { + "origin": "automation", + "automation_id": automation.id, + } + + # the covered-until anchor is only recorded once the job succeeds + assert not app.redis_connection.get(f"automation-last-run:{automation.id}") + + # process the job and check the report got saved + work_on_rq(app.queues["reporting"]) + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + stored_report = report_sensor.search_beliefs( + event_starts_after="2023-04-10T00:00:00+00:00", + event_ends_before="2023-04-10T10:00:00+00:00", + ) + assert (stored_report.values.T == [1, 2 + 3, 4 + 5, 6 + 7, 8 + 9]).all() + + # the successful job recorded the end of the report window as covered + import pandas as pd + + covered_until = app.redis_connection.get(f"automation-last-run:{automation.id}") + assert covered_until is not None + assert pd.Timestamp(covered_until.decode()) == pd.Timestamp( + "2023-04-10T10:00:00+00:00" + ) + + +def test_report_automation_refuses_a_sensor_nobody_checked( + app, + fresh_db, + setup_dummy_data, + clean_redis, + tmp_path, + freeze_server_now, + mocker, +): + """A reporter returning results for a sensor its automation did not declare is refused, before anything is recorded. + + The automation's output sensors were checked against its creator's permissions when it was created, + but the reporter decides at run time which sensors it returns results for. + Here it returns results for one of its input sensors, which was only ever checked for read access. + """ + import pandas as pd + + from flexmeasures.cli.data_add import add_automation + from flexmeasures.cli.jobs import run_automations + from flexmeasures.data.models.reporting.pandas_reporter import PandasReporter + from flexmeasures.data.models.time_series import Sensor + from flexmeasures.data.services.reporting import ( + ReportWritesUncheckedSensor, + run_report_job, + ) + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + cli_input = _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + parameters_extra={"start-offset": "DB"}, + asset_id=report_sensor.generic_asset_id, + ) + cli_input[cli_input.index("0 1 * * *")] = "* * * * *" # due every minute + cli_input += ["--timezone", "UTC"] + freeze_server_now(datetime(2023, 4, 10, 10, 0, 30, tzinfo=timezone.utc)) + runner = app.test_cli_runner() + result = runner.invoke(add_automation, cli_input) + assert "Successfully created" in result.output, result.output + automation = fresh_db.session.execute(select(Automation)).scalar_one() + result = runner.invoke(run_automations) + assert "queued 1 reporting job(s)" in result.output, result.output + job = app.queues["reporting"].jobs[0] + + input_sensor = fresh_db.session.get(Sensor, sensor1_id) + mocker.patch.object( + PandasReporter, + "compute", + return_value=[ + {"name": "df_agg", "sensor": input_sensor, "data": pd.DataFrame()} + ], + ) + mocker.patch( + "flexmeasures.data.services.reporting.get_current_job", return_value=job + ) + with pytest.raises(ReportWritesUncheckedSensor, match=str(sensor1_id)): + run_report_job(**job.kwargs) + + # a refused report covers nothing, so the next run still starts where the last successful one ended + assert not app.redis_connection.get(f"automation-last-run:{automation.id}") + + def test_run_automations( app, fresh_db, setup_dummy_data, clean_redis, freeze_server_now ): @@ -1249,7 +1595,7 @@ def test_failed_automation_attempt_is_not_retried(app, clean_redis, mocker): ) mocker.patch("flexmeasures.cli.jobs.claim_due_automation", return_value=True) - def queue_then_fail(_automation): + def queue_then_fail(_automation, **_kwargs): app.queues["forecasting"].enqueue("flexmeasures.utils.time_utils.server_now") raise RuntimeError("failed after queueing") diff --git a/flexmeasures/data/models/automations.py b/flexmeasures/data/models/automations.py index 459389ad0a..54a96de65f 100644 --- a/flexmeasures/data/models/automations.py +++ b/flexmeasures/data/models/automations.py @@ -50,19 +50,24 @@ class Automation(db.Model, AuthModelMixin): The recurrence is defined by a cron string. Every automation has a data generator, linked through a data source: a forecaster and its configuration for a forecast automation, - and a scheduler and the flex config it computes under for a schedule automation. - A forecast automation's generator is chosen when it is created. + a scheduler and the flex config it computes under for a schedule automation, + and a reporter and its configuration for a report automation. + A forecast automation's generator is chosen when it is created, and so is a report automation's. A schedule automation's is assembled from the trigger message and what its asset stores, so the runner puts it together afresh on every run. """ __tablename__ = "automation" - SUPPORTED_TYPES = ["forecasting", "scheduling"] # later also "reporting" + SUPPORTED_TYPES = ["forecasting", "scheduling", "reporting"] # What one result of each type is called, for messages that talk about a single result, # such as the parameters an automation of that type computes with. - RESULT_NOUNS = {"forecasting": "forecast", "scheduling": "schedule"} + RESULT_NOUNS = { + "forecasting": "forecast", + "scheduling": "schedule", + "reporting": "report", + } id = db.Column(db.Integer, autoincrement=True, primary_key=True) created_at = db.Column( diff --git a/flexmeasures/data/schemas/automations.py b/flexmeasures/data/schemas/automations.py index 3e007bcd63..4a070c9641 100644 --- a/flexmeasures/data/schemas/automations.py +++ b/flexmeasures/data/schemas/automations.py @@ -92,9 +92,12 @@ class AutomationCreationSchema(Schema): parameters = fields.Dict(keys=fields.Str(), load_default=dict) generator_class = fields.Str( data_key="data-generator", - load_default="TrainPredictPipeline", + load_default=None, + allow_none=True, metadata={ - "description": "Class of the data generator that computes this automation's results, reported back as the automation's `source`. Only a forecast automation chooses one; a schedule automation's generator follows from the asset and the flex config.", + "description": "Class of the data generator that computes this automation's results, reported back as the automation's `source`." + " A forecast automation defaults to TrainPredictPipeline, a report automation has to name its reporter (such as PandasReporter)," + " and a schedule automation's generator follows from the asset and the flex config.", "example": "TrainPredictPipeline", }, ) @@ -102,7 +105,7 @@ class AutomationCreationSchema(Schema): keys=fields.Str(), load_default=dict, metadata={ - "description": "Configuration stored on the data generator, as opposed to the `parameters` it runs with. Only used by a forecast automation.", + "description": "Configuration stored on the data generator, as opposed to the `parameters` it runs with. Used by a forecast or report automation.", "example": {}, }, ) diff --git a/flexmeasures/data/schemas/tests/test_reporting.py b/flexmeasures/data/schemas/tests/test_reporting.py index c669609d6d..25dde8983e 100644 --- a/flexmeasures/data/schemas/tests/test_reporting.py +++ b/flexmeasures/data/schemas/tests/test_reporting.py @@ -202,6 +202,22 @@ def test_profit_reporter_config_schema(config, is_valid, db, app, setup_dummy_se }, True, ), + ( # missing required input + { + "output": [{"sensor": 3}], + "start": start, + "end": end, + }, + False, + ), + ( # missing required output + { + "input": [{"sensor": 4}], + "start": start, + "end": end, + }, + False, + ), ( # wrong output unit { "input": [{"sensor": 4}], # unit: MW diff --git a/flexmeasures/data/scripts/data_gen.py b/flexmeasures/data/scripts/data_gen.py index 047e95039d..73efdedcd9 100644 --- a/flexmeasures/data/scripts/data_gen.py +++ b/flexmeasures/data/scripts/data_gen.py @@ -468,6 +468,7 @@ def depopulate_prognoses( if not sensor: num_forecasting_jobs_deleted = app.queues["forecasting"].empty() num_scheduling_jobs_deleted = app.queues["scheduling"].empty() + num_reporting_jobs_deleted = app.queues["reporting"].empty() # Clear all forecasts (data with positive horizon) query = delete(TimedBelief).filter(TimedBelief.belief_horizon > timedelta(hours=0)) @@ -480,6 +481,7 @@ def depopulate_prognoses( if not sensor: click.echo("Deleted %d Forecast Jobs" % num_forecasting_jobs_deleted) click.echo("Deleted %d Schedule Jobs" % num_scheduling_jobs_deleted) + click.echo("Deleted %d Report Jobs" % num_reporting_jobs_deleted) click.echo("Deleted %d forecasts (ex-ante beliefs)" % num_forecasts_deleted) diff --git a/flexmeasures/data/services/automations.py b/flexmeasures/data/services/automations.py index 04fc142d27..d71a697d32 100644 --- a/flexmeasures/data/services/automations.py +++ b/flexmeasures/data/services/automations.py @@ -14,12 +14,13 @@ from croniter import croniter from croniter.croniter import CroniterError import isodate +import pandas as pd from isodate.isoerror import ISO8601Error from flask import current_app from marshmallow import ValidationError from sqlalchemy import select, update -from flexmeasures import Forecaster +from flexmeasures import Forecaster, Reporter from flexmeasures.data import db from flexmeasures.data.models.automations import ( Automation, @@ -36,7 +37,7 @@ check_sensor_access, resolve_data_generator_sensors, ) -from flexmeasures.utils.time_utils import server_now +from flexmeasures.utils.time_utils import apply_offset_chain, server_now @dataclass(frozen=True) @@ -59,7 +60,7 @@ class DueAutomation: # Extend it whenever a flex-model or flex-context field starts naming somewhere results are recorded. # A field this list misses is not left unchecked so much as checked for the wrong thing: # the sensor is read as an input, so its creator needs only read access where recording data calls for create-children access. -# A schedule job created by an automation is therefore held to the sensors predicted here (see `_sensors_this_job_may_record_on`), +# A schedule job created by an automation is therefore held to the sensors predicted here (see `sensors_automation_job_may_record_on`), # so that drift shows up as a refusal rather than a quiet downgrade. OUTPUT_SENSOR_FIELDS = ( "consumption", @@ -452,7 +453,7 @@ def resolve_schedule_automation_sensors( def resolve_automation_sensors(automation: Automation) -> dict[str, list[Sensor]]: """Work out which sensors an automation reads from and writes to on each run. - Forecast sensors are derived from the data generator, + Forecast and report sensors are derived from the data generator, while schedule sensors are derived from the same prepared trigger message used to queue the scheduling job. Raises `AutomationSensorsUnknown` if that cannot be done, e.g. because a forecast automation has no data generator, because its generator is not registered in this FlexMeasures instance, @@ -474,9 +475,17 @@ def resolve_automation_sensors(automation: Automation) -> dict[str, list[Sensor] ) try: data_generator = automation.generator.data_generator + parameters = dict(automation.parameters or {}) + if automation.type == "reporting": + parameters = prepare_report_parameters( + parameters, + automation.cronstr, + automation.timezone, + automation_id=automation.id, + ) return resolve_data_generator_sensors( data_generator, - data_generator._parameters_schema.load(dict(automation.parameters or {})), + data_generator._parameters_schema.load(parameters), ) except (NotImplementedError, ValidationError) as e: raise AutomationSensorsUnknown( @@ -499,14 +508,49 @@ def get_automation_sensors(automation: Automation) -> dict[str, list[Sensor]]: return {"input_sensors": [], "output_sensors": []} +def sensors_automation_job_may_record_on(rq_job) -> set[int] | None: + """Return the sensor ids an automation-triggered job was cleared to record on, or None if it is not one. + + An automation's output sensors are checked against its creator's permissions when the automation is created, + which is the only moment a user is present. + A data generator decides at run time which sensors it returns results for, though, + so a job holds it to the sensors that were checked, turning a write on anything else into a refusal. + For a schedule this matters because its output sensors are predicted from the fields that name them (see `OUTPUT_SENSOR_FIELDS`), + so a sensor named by some other field was only ever checked for read access, as the prediction reads it as an input. + For a report, the output sensors are named in its parameters, but the reporter is free to return results for others. + + Returns None where there is nothing to hold the job to: a job that is not an automation's, + or an automation deleted since the job was queued. + An automation whose sensors cannot be determined returns an empty set instead, which permits nothing: + a guard that cannot work out what is allowed should not conclude that everything is. + """ + trigger = (rq_job.meta.get("trigger") if rq_job else None) or {} + if trigger.get("origin") != "automation": + return None + automation_id = trigger.get("automation_id") + if automation_id is None: + return None + + automation = db.session.get(Automation, automation_id) + if automation is None: + return None + try: + sensors = resolve_automation_sensors(automation)["output_sensors"] + except AutomationSensorsUnknown as exc: + current_app.logger.error( + f"Cannot check which sensors automation {automation_id} may record on, so it records nothing: {exc}" + ) + return set() + return {sensor.id for sensor in sensors} + + def get_automations_feeding_sensor(sensor: Sensor) -> list[Automation]: """Find the automations that write data to the given sensor. - Only automations on the sensor's own asset or on one of its ancestors are - considered, as an automation may only write to its asset's subtree - (see `validate_forecast_output_scope`). Working out the output sensors requires - setting up each candidate's data generator, so this keeps the work proportional - to the number of automations that could feed this sensor. + Only automations on the sensor's own asset or on one of its ancestors are considered, + as an automation may only write to its asset's subtree (see `validate_automation_output_scope`). + Working out the output sensors requires setting up each candidate's data generator, + so this keeps the work proportional to the number of automations that could feed this sensor. Note that this does not filter by permission: callers showing these to a user should check read access on each automation (e.g. with `user_can_read`). @@ -556,6 +600,167 @@ def prepare_schedule_trigger_message(parameters: dict, asset_id: int) -> dict: return message +def validate_offset_chain(offset_chain: str): + """Raise a ValueError on any offset that apply_offset_chain would silently skip. + + Valid offsets are Pandas offset strings, plus "DB" (day begin) and "HB" (hour begin). + """ + from pandas.tseries.frequencies import to_offset + + for offset in str(offset_chain).split(","): + offset = offset.strip() + if offset.lower() in ("db", "hb"): + continue + try: + to_offset(offset) + except ValueError: + raise ValueError( + f"'{offset}' is not a valid Pandas offset string (nor 'DB'/'HB')." + ) + + +def _last_run_redis_key(automation_id: int) -> str: + return f"automation-last-run:{automation_id}" + + +def record_automation_run(automation_id: int, now: datetime | None = None) -> bool: + """Remember (in Redis) until when a report automation's reports reach, so its next default window starts there. + + The reporting job records the end of its report window once it has succeeded (see run_report_job), + so a failed report job does not leave a permanent gap in the reported periods. + Only report automations use this: forecasts and schedules do not continue from where the previous run ended. + """ + from redis.exceptions import WatchError + + if now is None: + now = server_now() + candidate = floor_to_minute(now) + key = _last_run_redis_key(automation_id) + connection = current_app.redis_connection + while True: + with connection.pipeline() as pipeline: + try: + pipeline.watch(key) + value = pipeline.get(key) + if value: + if isinstance(value, bytes): + value = value.decode() + try: + current = floor_to_minute(datetime.fromisoformat(value)) + except ValueError: + current = None + if current is not None and current >= candidate: + pipeline.unwatch() + return False + pipeline.multi() + pipeline.set(key, candidate.isoformat()) + pipeline.execute() + return True + except WatchError: + # Another worker updated the coverage after our read. + # Re-read it, and only advance from the new value. + continue + + +def get_automation_last_run(automation_id: int) -> datetime | None: + """Until when this automation's work is covered, if known (the record lives in Redis).""" + from flask import current_app + + value = current_app.redis_connection.get(_last_run_redis_key(automation_id)) + if not value: + return None + if isinstance(value, bytes): + value = value.decode() + try: + return datetime.fromisoformat(value) + except ValueError: + return None + + +def prepare_report_parameters( + parameters: dict, + cronstr: str, + automation_timezone: str, + now: datetime | None = None, + automation_id: int | None = None, + scheduled_at: datetime | None = None, +) -> dict: + """Complete stored report parameters into a message for the ReporterParametersSchema. + + The (required) start and end of the report are resolved on each run: + + - "start-offset" and "end-offset" fields hold comma-separated Pandas offsets, such as "-1D,DB" for the start of the previous day, + applied to the run time in the automation's timezone. + - Without offsets, the window runs since the end of the automation's last successfully covered window, + falling back to the last cron period, from the previous cron fire time until the run time, when none is known, such as on the first run. + """ + message = dict(parameters) + if scheduled_at is None: + scheduled_at = now if now is not None else server_now() + scheduled_at = floor_to_minute(scheduled_at) + + # Offsets are applied on the automation's own clock, the one its cron string is read in, + # so that "DB" means midnight where the user who set up the automation expects it. + tz = ZoneInfo(automation_timezone) + now = scheduled_at.astimezone(tz) + + # A report automation cannot fix its window (see `_prepare_report_automation`), so offsets apply to the run time. + start_offset = message.pop("start-offset", None) + end_offset = message.pop("end-offset", None) + start = ( + apply_offset_chain(pd.Timestamp(now), start_offset) + if start_offset is not None + else None + ) + end = ( + apply_offset_chain(pd.Timestamp(now), end_offset) + if end_offset is not None + else None + ) + + # Default to the window since the last covered window's end, + # falling back to the last cron period, from the previous cron fire time until the run time. + if start is None: + last_run = ( + get_automation_last_run(automation_id) + if automation_id is not None + else None + ) + if last_run is not None: + start = last_run + else: + nominal_scheduled_at = _as_nominal_wall_time(scheduled_at.astimezone(tz)) + previous_nominal = croniter(cronstr, nominal_scheduled_at).get_prev( + datetime + ) + start = _canonical_run_time(previous_nominal, tz) + # A skipped wall time can canonicalize to the first valid instant after the gap, which may be the current run. + # Step back once more, so the first report still covers a non-empty cron period. + if start >= scheduled_at: + previous_nominal = croniter(cronstr, previous_nominal).get_prev( + datetime + ) + start = _canonical_run_time(previous_nominal, tz) + if end is None: + end = now + + message["start"] = pd.Timestamp(start).isoformat() + message["end"] = pd.Timestamp(end).isoformat() + return message + + +def _relevant_sensor_ids(automation: Automation, parameter_values: list) -> set[int]: + """The asset's sensor ids, plus any (castable) sensor ids among the given parameter values.""" + sensor_ids = {sensor.id for sensor in automation.asset.sensors} + for value in parameter_values: + if value is not None: + try: + sensor_ids.add(int(value)) + except (TypeError, ValueError): + pass + return sensor_ids + + def resolve_schedule_generator(asset_id: int, parameters: dict) -> DataSource: """The data source describing the scheduler a schedule automation runs, and the flex config it runs with. @@ -613,6 +818,7 @@ def get_automation_job_stats(automation: Automation) -> dict[str, int]: Note that jobs in Redis have a limited TTL, so this only counts fairly recent jobs. """ # Determine the job cache entries to scan. + parameters = automation.parameters or {} if automation.type == "scheduling": # Scheduling jobs are cached under the asset (multi-device wrap-up jobs) # and under individual sensors (per-device jobs). @@ -622,17 +828,25 @@ def get_automation_job_stats(automation: Automation) -> dict[str, int]: for asset in assets for sensor in asset.sensors ] + elif automation.type == "reporting": + # Reporting jobs are cached under their output sensor(s), + # which may belong to a different asset than the automation's own asset. + sensor_ids = _relevant_sensor_ids( + automation, + [ + output.get("sensor") + for output in parameters.get("output", []) or [] + if isinstance(output, dict) + ], + ) + cache_refs = [(sensor_id, "reporting", "sensor") for sensor_id in sensor_ids] else: # Forecasting jobs are cached under the forecast target sensor(s), # which may belong to a different asset than the automation's own asset. - sensor_ids = {sensor.id for sensor in automation.asset.sensors} - for key in ("sensor", "sensor-to-save"): - value = (automation.parameters or {}).get(key) - if value is not None: - try: - sensor_ids.add(int(value)) - except (TypeError, ValueError): - pass + sensor_ids = _relevant_sensor_ids( + automation, + [parameters.get("sensor"), parameters.get("sensor-to-save")], + ) cache_refs = [(sensor_id, "forecasting", "sensor") for sensor_id in sensor_ids] counts: dict[str, int] = {} @@ -648,6 +862,84 @@ def get_automation_job_stats(automation: Automation) -> dict[str, int]: return counts +def _prepare_forecast_automation( + asset, parameters: dict, generator_class: str | None, config: dict | None, source +) -> tuple[Forecaster, dict, list[str]]: + """Validate forecast automation parameters and set up the forecaster, without creating a data source.""" + from flexmeasures.data.models.time_series import Sensor + from flexmeasures.data.schemas.forecasting.pipeline import ( + ForecasterParametersSchema, + ) + from flexmeasures.data.services.data_sources import get_data_generator + + warnings = [] + deserialized_parameters = ForecasterParametersSchema().load(parameters) + sensor = deserialized_parameters.get("sensor") + if isinstance(sensor, Sensor) and sensor.generic_asset_id != asset.id: + warnings.append( + f"The sensor to forecast ({sensor.id}) does not belong to asset {asset.id}." + ) + forecaster = get_data_generator( + source=source, + model=generator_class or "TrainPredictPipeline", + config=config or {}, + save_config=True, + data_generator_type=Forecaster, + ) + if forecaster is None: + raise ValueError(f"Could not set up forecaster '{generator_class}'.") + return forecaster, deserialized_parameters, warnings + + +def _prepare_report_automation( + parameters: dict, + cronstr: str, + automation_timezone: str, + generator_class: str | None, + config: dict | None, + source, +) -> tuple[Reporter, dict, list[str]]: + """Validate report automation parameters without creating a data source.""" + from marshmallow import ValidationError + + from flexmeasures.data.services.data_sources import get_data_generator + + warnings: list[str] = [] + if generator_class is None and source is None: + raise ValidationError( + "A reporter is required for report automations (e.g. PandasReporter)." + ) + # An automation runs again and again, so a fixed period would have it report on the same period every time. + fixed_fields = [field for field in ("start", "end") if field in parameters] + if fixed_fields: + raise ValidationError( + f"A report automation cannot fix {' or '.join(repr(field) for field in fixed_fields)}, as every run would then report on the same period." + " Use 'start-offset' and 'end-offset' (Pandas offsets, applied to the run time in the automation's timezone)," + " or leave the timing out to report on the period since the last successful report." + ) + for offset_field in ("start-offset", "end-offset"): + if offset_field in parameters: + try: + validate_offset_chain(parameters[offset_field]) + except ValueError as e: + raise ValidationError(f"Invalid {offset_field}: {e}") + reporter = get_data_generator( + source=source, + model=generator_class, + config=config or {}, + save_config=True, + data_generator_type=Reporter, + ) + if reporter is None: + raise ValueError(f"Could not set up reporter '{generator_class}'.") + # Validate with the chosen reporter's own parameters schema, + # which may extend the base ReporterParametersSchema. + deserialized_parameters = reporter._parameters_schema.load( + prepare_report_parameters(parameters, cronstr, automation_timezone) + ) + return reporter, deserialized_parameters, warnings + + def create_automation( asset, name: str, @@ -656,7 +948,7 @@ def create_automation( automation_type: str = "forecasting", active: bool = True, parameters: dict | None = None, - generator_class: str = "TrainPredictPipeline", + generator_class: str | None = None, config: dict | None = None, source=None, origin: str = "API", @@ -664,7 +956,7 @@ def create_automation( ) -> tuple[Automation, list[str]]: """Create an automation (not committed yet), validating its parameters by type. - For forecasts, the forecaster config is stored on a data source. + For forecasts and reports, the data generator config is stored on a data source. An audit log record is added to the asset. :param check_permissions: whether to require that the current user may read the sensors that the automation reads from, @@ -672,43 +964,29 @@ def create_automation( Set this for automations created by a user (through the API or the UI); the CLI runs without a user, and is trusted. :raises marshmallow.ValidationError: if the parameters are invalid. - :raises ValueError: if the forecaster cannot be set up. + :raises ValueError: if the data generator cannot be set up. :raises werkzeug.exceptions.Forbidden: if a sensor is not accessible to the user. :returns: the automation and a list of warnings. """ from marshmallow import ValidationError from flexmeasures.data.models.audit_log import AssetAuditLog - from flexmeasures.data.models.time_series import Sensor parameters = parameters or {} + timezone = timezone or get_default_automation_timezone(asset) warnings: list[str] = [] generator_id = None - forecaster = None + data_generator = None input_sensors: list[Sensor] = [] output_sensors: list[Sensor] = [] - forecast_output_sensor: Sensor | None = None if automation_type == "forecasting": - from flexmeasures.data.schemas.forecasting.pipeline import ( - ForecasterParametersSchema, - ) - from flexmeasures.data.services.data_sources import get_data_generator - - deserialized_parameters = ForecasterParametersSchema().load(parameters) - sensor = deserialized_parameters.get("sensor") - if isinstance(sensor, Sensor) and sensor.generic_asset_id != asset.id: - warnings.append( - f"The sensor to forecast ({sensor.id}) does not belong to asset {asset.id}." + forecaster, deserialized_parameters, forecast_warnings = ( + _prepare_forecast_automation( + asset, parameters, generator_class, config, source ) - forecaster = get_data_generator( - source=source, - model=generator_class, - config=config or {}, - save_config=True, - data_generator_type=Forecaster, ) - if forecaster is None: - raise ValueError(f"Could not set up forecaster '{generator_class}'.") + warnings.extend(forecast_warnings) + data_generator = forecaster # A forecast reads the history of the sensor to forecast, plus its regressors, # and records the forecast on the sensor to save to (the same sensor by default). @@ -718,7 +996,6 @@ def create_automation( ) input_sensors = forecast_sensors["input_sensors"] output_sensors = forecast_sensors["output_sensors"] - forecast_output_sensor = output_sensors[0] if output_sensors else None elif automation_type == "scheduling": from flexmeasures.utils import flexmeasures_inflection from flexmeasures.data.schemas.scheduling import ( @@ -749,6 +1026,17 @@ def create_automation( "The schedule 'start' is fixed, so each run will compute the same period." " Omit 'start' to schedule from the run time instead." ) + elif automation_type == "reporting": + reporter, deserialized_parameters, report_warnings = _prepare_report_automation( + parameters, cronstr, timezone, generator_class, config, source + ) + warnings.extend(report_warnings) + data_generator = reporter + report_sensors = resolve_data_generator_sensors( + reporter, deserialized_parameters + ) + input_sensors = report_sensors["input_sensors"] + output_sensors = report_sensors["output_sensors"] else: raise ValidationError( f"Automation type '{automation_type}' is not supported (supported types: {Automation.SUPPORTED_TYPES})." @@ -759,13 +1047,14 @@ def create_automation( # Only once the sensors are known to be the user's to involve do we say anything about them, # so that this does not reveal where a sensor sits to someone who may not read it. - if forecast_output_sensor is not None: - validate_forecast_output_scope(asset.id, forecast_output_sensor) + if automation_type in ("forecasting", "reporting"): + for output_sensor in output_sensors: + validate_automation_output_scope(asset.id, output_sensor, automation_type) - if forecaster is not None: - # Look up or create the data source storing the forecaster config only now that the automation is going ahead, + if data_generator is not None: + # Look up or create the data source storing the generator config only now that the automation is going ahead, # so that a refused request leaves nothing behind, whatever the caller does with the session afterwards. - generator = forecaster.data_source + generator = data_generator.data_source db.session.flush() generator_id = generator.id elif automation_type == "scheduling": @@ -784,7 +1073,7 @@ def create_automation( generator_id=generator_id, parameters=parameters, ) - automation_fields["timezone"] = timezone or get_default_automation_timezone(asset) + automation_fields["timezone"] = timezone automation = Automation(**automation_fields) db.session.add(automation) db.session.flush() @@ -875,16 +1164,20 @@ def get_forecast_output_sensor(parameters: dict[str, Any]) -> Sensor: return sensor -def validate_forecast_output_scope(asset_id: int, output_sensor: Sensor) -> None: - """Require forecast output on the automation asset or a descendant.""" +def validate_automation_output_scope( + asset_id: int, output_sensor: Sensor, automation_type: str +) -> None: + """Require generated output on the automation asset or a descendant.""" if not asset_is_in_subtree(asset_id, output_sensor.generic_asset_id): raise ValueError( - f"Forecast automation output sensor {output_sensor.id} must belong to asset " + f"{automation_type.capitalize()} automation output sensor {output_sensor.id} must belong to asset " f"{asset_id} or one of its descendants." ) -def run_automation(automation: Automation) -> dict[str, Any] | None: +def run_automation( + automation: Automation, scheduled_at: datetime | None = None +) -> dict[str, Any] | None: """Queue the jobs for one run of an automation. :returns: a dict like {"job_id": , "n_jobs": }. @@ -893,6 +1186,10 @@ def run_automation(automation: Automation) -> dict[str, Any] | None: return _run_forecast_automation(automation) elif automation.type == "scheduling": return _run_schedule_automation(automation) + elif automation.type == "reporting": + # The reporting job records how far the reports reach once it succeeds (see run_report_job), + # so a failed job leaves no gap for the next run to skip over. + return _run_report_automation(automation, scheduled_at=scheduled_at) raise NotImplementedError( f"Automations of type '{automation.type}' cannot be run yet." ) @@ -911,13 +1208,48 @@ def _run_forecast_automation(automation: Automation) -> dict[str, Any] | None: f"Data source {automation.generator_id} of automation {automation.id} does not store a Forecaster." ) output_sensor = get_forecast_output_sensor(automation.parameters or {}) - validate_forecast_output_scope(automation.asset_id, output_sensor) + validate_automation_output_scope( + automation.asset_id, output_sensor, automation.type + ) # Wipe any parameter state the copy inherited from a previous run. forecaster._parameters = None forecaster.set_job_trigger("automation", automation_id=automation.id) return forecaster.compute(as_job=True, parameters=dict(automation.parameters)) +def _run_report_automation( + automation: Automation, scheduled_at: datetime | None = None +) -> dict[str, Any] | None: + if automation.generator is None: + raise ValueError( + f"Automation {automation.id} has no data generator to run (generator_id is not set)." + ) + reporter = automation.generator.data_generator + if not isinstance(reporter, Reporter): + raise ValueError( + f"Data source {automation.generator_id} of automation {automation.id} does not store a Reporter." + ) + parameters = prepare_report_parameters( + dict(automation.parameters), + automation.cronstr, + automation.timezone, + automation_id=automation.id, + scheduled_at=scheduled_at, + ) + report_sensors = resolve_data_generator_sensors( + reporter, reporter._parameters_schema.load(parameters) + ) + for output_sensor in report_sensors["output_sensors"]: + validate_automation_output_scope( + automation.asset_id, output_sensor, automation.type + ) + # The data generator instance is cached on the data source, which may be shared by several automations, + # so wipe any parameter state from a previous run. + reporter._parameters = None + reporter.set_job_trigger("automation", automation_id=automation.id) + return reporter.compute(as_job=True, parameters=parameters) + + def _run_schedule_automation(automation: Automation) -> dict[str, Any]: from flexmeasures.data.schemas.scheduling import AssetTriggerSchema from flexmeasures.data.services.scheduling import ( diff --git a/flexmeasures/data/services/reporting.py b/flexmeasures/data/services/reporting.py index e2b4350c32..086f3a4ec2 100644 --- a/flexmeasures/data/services/reporting.py +++ b/flexmeasures/data/services/reporting.py @@ -2,10 +2,11 @@ from __future__ import annotations -from datetime import timedelta +from datetime import datetime, timedelta from typing import TYPE_CHECKING from flask import current_app +from rq import get_current_job from rq.job import Job from flexmeasures.data import db @@ -83,8 +84,17 @@ def _count_persistable_values(data) -> int: return len(data.dropna(subset=["event_value"])) +class ReportWritesUncheckedSensor(PermissionError): + """Raised when a reporter returns results for a sensor that nobody's permissions were checked against.""" + + def run_report_job(data_source_id: int, parameters: dict) -> list[dict]: - """Compute and store a report in a reporting worker.""" + """Compute and store a report in a reporting worker. + + If the report was triggered by an automation, the end of the report window is recorded upon success, + so the automation's next default window starts where this one ended. + A failed report job therefore leaves no permanent gap in the reported periods. + """ from flexmeasures.data.models.data_sources import DataSource from flexmeasures.data.models.reporting import Reporter @@ -96,6 +106,30 @@ def run_report_job(data_source_id: int, parameters: dict) -> list[dict]: raise ValueError(f"Data source {data_source_id} does not store a Reporter.") reporter._parameters = None results = reporter.compute(parameters=parameters) + + # An automation's job may only record on the sensors its creator was checked against. + # Judge the whole set before writing any of it. + from flexmeasures.data.services.automations import ( + sensors_automation_job_may_record_on, + ) + + rq_job = get_current_job() + permitted_output_sensor_ids = sensors_automation_job_may_record_on(rq_job) + if permitted_output_sensor_ids is not None: + refused = sorted( + { + result["sensor"].id + for result in results + if result["sensor"].id not in permitted_output_sensor_ids + } + ) + if refused: + raise ReportWritesUncheckedSensor( + f"This report would record data on sensor(s) {', '.join(str(i) for i in refused)}," + f" which are not among the sensors automation {rq_job.meta['trigger']['automation_id']}" + " was checked against when it was created." + ) + saved = [] for result in results: n_rows = _count_persistable_values(result["data"]) @@ -116,4 +150,18 @@ def run_report_job(data_source_id: int, parameters: dict) -> list[dict]: source, summary, ) + + # The job's trigger says whether an automation created it, as it does for the guard above. + automation_id = ( + rq_job.meta["trigger"]["automation_id"] + if permitted_output_sensor_ids is not None + else None + ) + if automation_id is not None and parameters.get("end"): + from flexmeasures.data.services.automations import record_automation_run + + record_automation_run( + automation_id, now=datetime.fromisoformat(parameters["end"]) + ) + return saved diff --git a/flexmeasures/data/services/scheduling.py b/flexmeasures/data/services/scheduling.py index dc257f9b13..470ae3ec5d 100644 --- a/flexmeasures/data/services/scheduling.py +++ b/flexmeasures/data/services/scheduling.py @@ -914,47 +914,6 @@ class ScheduleWritesUncheckedSensor(PermissionError): """Raised when a scheduler returns results for a sensor that nobody's permissions were checked against.""" -def _sensors_this_job_may_record_on(rq_job) -> set[int] | None: - """Return the sensor ids an automation-triggered job was cleared to record on, or None if it is not one. - - An automation's output sensors are checked against its creator's permissions when the automation is created, - which is the only moment a user is present. - Those sensors are predicted from the fields that name them (see `OUTPUT_SENSOR_FIELDS`), - so a scheduler which returned results for a sensor named by some other field would record data on a sensor that was only ever checked for read access, - as the prediction reads it as an input. - Holding the scheduler to the prediction turns that silent downgrade into a refusal. - - Returns None where there is nothing to hold the job to: a job that is not an automation's, - or an automation deleted since the job was queued. - An automation whose sensors cannot be determined returns an empty set instead, which permits nothing: - a guard that cannot work out what is allowed should not conclude that everything is. - """ - trigger = (rq_job.meta.get("trigger") if rq_job else None) or {} - if trigger.get("origin") != "automation": - return None - automation_id = trigger.get("automation_id") - if automation_id is None: - return None - - from flexmeasures.data.models.automations import Automation - from flexmeasures.data.services.automations import ( - AutomationSensorsUnknown, - resolve_automation_sensors, - ) - - automation = db.session.get(Automation, automation_id) - if automation is None: - return None - try: - sensors = resolve_automation_sensors(automation)["output_sensors"] - except AutomationSensorsUnknown as exc: - current_app.logger.error( - f"Cannot check which sensors automation {automation_id} may record on, so it records nothing: {exc}" - ) - return set() - return {sensor.id for sensor in sensors} - - def make_schedule( # noqa: C901 sensor_id: int | None = None, start: datetime | None = None, @@ -1085,7 +1044,11 @@ def make_schedule( # noqa: C901 rq_job.save_meta() # Save any result that specifies a sensor to save it to - permitted_output_sensor_ids = _sensors_this_job_may_record_on(rq_job) + from flexmeasures.data.services.automations import ( + sensors_automation_job_may_record_on, + ) + + permitted_output_sensor_ids = sensors_automation_job_may_record_on(rq_job) if permitted_output_sensor_ids is not None: # Judge the whole set before writing any of it. # The job's transaction would roll an interrupted write back, since `save_to_db` only flushes, diff --git a/flexmeasures/data/tests/test_automations_fresh_db.py b/flexmeasures/data/tests/test_automations_fresh_db.py index 2c953ec4e4..7189e3701a 100644 --- a/flexmeasures/data/tests/test_automations_fresh_db.py +++ b/flexmeasures/data/tests/test_automations_fresh_db.py @@ -88,6 +88,21 @@ def test_automation_requires_generator(fresh_db, automation_with_generator): fresh_db.session.commit() +def test_report_automation_requires_generator(fresh_db, automation_with_generator): + forecast_automation, _ = automation_with_generator + report_automation = Automation( + asset=forecast_automation.asset, + type="reporting", + name="generator-free report", + cronstr="0 1 * * *", + parameters={}, + ) + fresh_db.session.add(report_automation) + + with pytest.raises(IntegrityError): + fresh_db.session.commit() + + def test_schedule_automation_generator_describes_its_scheduler_and_config( fresh_db, automation_with_generator ): @@ -458,18 +473,20 @@ def test_a_job_that_is_not_an_automations_is_held_to_nothing(app, fresh_db, mock A schedule triggered through the API or the CLI has its sensors checked against the requester at trigger time, so there is nothing for this guard to add there. """ - from flexmeasures.data.services.scheduling import _sensors_this_job_may_record_on + from flexmeasures.data.services.automations import ( + sensors_automation_job_may_record_on, + ) - assert _sensors_this_job_may_record_on(None) is None + assert sensors_automation_job_may_record_on(None) is None api_job = mocker.Mock() api_job.meta = {"trigger": {"origin": "API"}} - assert _sensors_this_job_may_record_on(api_job) is None + assert sensors_automation_job_may_record_on(api_job) is None # An automation deleted since its job was queued leaves nothing to hold the job to. gone = mocker.Mock() gone.meta = {"trigger": {"origin": "automation", "automation_id": 999999}} - assert _sensors_this_job_may_record_on(gone) is None + assert sensors_automation_job_may_record_on(gone) is None def test_an_automation_whose_sensors_are_unknown_records_nothing( @@ -480,7 +497,9 @@ def test_an_automation_whose_sensors_are_unknown_records_nothing( The alternative, proceeding unchecked, is what the run-time check exists to stop. """ from flexmeasures.data.services.automations import AutomationSensorsUnknown - from flexmeasures.data.services.scheduling import _sensors_this_job_may_record_on + from flexmeasures.data.services.automations import ( + sensors_automation_job_may_record_on, + ) battery = add_battery_assets_fresh_db["Test battery"] message = message_for_trigger_schedule() @@ -502,4 +521,4 @@ def test_an_automation_whose_sensors_are_unknown_records_nothing( job = mocker.Mock() job.meta = {"trigger": {"origin": "automation", "automation_id": automation.id}} - assert _sensors_this_job_may_record_on(job) == set() + assert sensors_automation_job_may_record_on(job) == set() diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 84e0970d0b..a3bd4bb002 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -6731,7 +6731,8 @@ "default": "forecasting", "enum": [ "forecasting", - "scheduling" + "scheduling", + "reporting" ] }, "name": { @@ -6760,14 +6761,17 @@ "additionalProperties": {} }, "data-generator": { - "type": "string", - "default": "TrainPredictPipeline", - "description": "Class of the data generator that computes this automation's results, reported back as the automation's `source`. Only a forecast automation chooses one; a schedule automation's generator follows from the asset and the flex config.", + "type": [ + "string", + "null" + ], + "default": null, + "description": "Class of the data generator that computes this automation's results, reported back as the automation's `source`. A forecast automation defaults to TrainPredictPipeline, a report automation has to name its reporter (such as PandasReporter), and a schedule automation's generator follows from the asset and the flex config.", "example": "TrainPredictPipeline" }, "config": { "type": "object", - "description": "Configuration stored on the data generator, as opposed to the `parameters` it runs with. Only used by a forecast automation.", + "description": "Configuration stored on the data generator, as opposed to the `parameters` it runs with. Used by a forecast or report automation.", "example": {}, "additionalProperties": {} } diff --git a/flexmeasures/ui/templates/assets/asset_automations.html b/flexmeasures/ui/templates/assets/asset_automations.html index 540ea7a256..54a13d92f3 100644 --- a/flexmeasures/ui/templates/assets/asset_automations.html +++ b/flexmeasures/ui/templates/assets/asset_automations.html @@ -77,7 +77,7 @@

Automations of {{ asset.name }} @@ -117,6 +117,7 @@

New automation for {{ asset.name }}
@@ -125,15 +126,26 @@
New automation for {{ asset.name }}
Use five fields: minute, hour, day of month, month, day of week. 0 6 * * * runs daily at 06:00; */15 * * * * runs every 15 minutes. Times use the timezone below.
- + -
The local clock used by the schedule. For example, Europe/Amsterdam keeps 06:00 at 06:00 locally when daylight saving changes. Defaults to this asset's timezone.
+
The local clock the recurrence is read in. For example, Europe/Amsterdam keeps 06:00 at 06:00 locally when daylight saving changes. Defaults to this asset's timezone.
+
+
+ + +
+ Forecaster class (defaults to TrainPredictPipeline) or reporter class (required for type reporting, e.g. PandasReporter). Not used for scheduling. +
+
+
+ +
- Forecast parameters (for type forecasting) or a schedule trigger message (for type scheduling). + Forecast parameters (for type forecasting), a schedule trigger message (for type scheduling), or report parameters (for type reporting).
@@ -170,7 +182,7 @@
Edit automation
Five fields: minute, hour, day of month, month, day of week. 0 6 * * * runs daily at 06:00 in the timezone below. Changing the schedule recalculates the next eligible run.
- +
The cron time follows this local clock, including daylight-saving changes. For example, choose Europe/Amsterdam to run at 06:00 Amsterdam time year-round.
@@ -197,7 +209,7 @@
Edit automation
@@ -211,6 +223,11 @@
Edit automation
+
+
+
+
+
@@ -424,7 +441,7 @@
Recently created jobs
{ data: "created-at", title: "Created At", render: renderSortable }, { data: "active", title: "Active", render: renderSortable }, { data: "recurrence", title: "Recurrence", render: renderSortable }, - { data: "timezone", title: "Schedule timezone" }, + { data: "timezone", title: "Recurrence timezone" }, { data: "next-run", title: "Next run (local)", render: renderSortable }, { data: "jobs", title: "Jobs", orderable: false }, { data: "details", title: "", className: "text-center align-middle", orderable: false }, @@ -460,7 +477,7 @@
Recently created jobs
method: "GET", success: function (res) { res.automations.forEach(automation => automationsById.set(automation.id, automation)); - for (const automationType of ["forecasting", "scheduling"]) { + for (const automationType of ["forecasting", "scheduling", "reporting"]) { makeAutomationsTable( automationType, res.automations.filter(automation => automation.type === automationType), @@ -468,7 +485,8 @@
Recently created jobs
} }, error: function (xhr) { - for (const automationType of ["forecasting", "scheduling"]) { + console.error("Error fetching automations:", xhr); + for (const automationType of ["forecasting", "scheduling", "reporting"]) { makeAutomationsTable(automationType, []); $(`#automationsTable-${automationType}`).hide(); } @@ -554,6 +572,16 @@
Recently created jobs
return; } } + let config = {}; + const configText = $("#automationConfig").val().trim(); + if (configText) { + try { + config = JSON.parse(configText); + } catch (e) { + $("#newAutomationErr").removeClass("d-none").text("The data generator config is not valid JSON."); + return; + } + } $.ajax({ url: `/api/v3_0/assets/${assetId}/automations`, method: "POST", @@ -564,6 +592,8 @@
Recently created jobs
cron: $("#automationCron").val(), timezone: $("#automationTimezone").val(), active: $("#automationActive").is(":checked"), + "data-generator": $("#automationGenerator").val().trim() || null, + config: config, parameters: parameters, }), success: () => location.reload(), diff --git a/flexmeasures/ui/tests/test_asset_crud.py b/flexmeasures/ui/tests/test_asset_crud.py index 2f947c406c..c70c355d39 100644 --- a/flexmeasures/ui/tests/test_asset_crud.py +++ b/flexmeasures/ui/tests/test_asset_crud.py @@ -80,8 +80,10 @@ def test_asset_page(db, client, setup_assets, as_prosumer_user1, view): assert "Automations of".encode() in asset_page.data assert "Forecasts".encode() in asset_page.data assert "Schedules".encode() in asset_page.data + assert "Reports".encode() in asset_page.data assert b'id="automationsTable-forecasting"' in asset_page.data assert b'id="automationsTable-scheduling"' in asset_page.data + assert b'id="automationsTable-reporting"' in asset_page.data assert b"automation.type === automationType" in asset_page.data assert b"No ${automationType} automations" in asset_page.data assert b'id="automations_err"' in asset_page.data @@ -89,7 +91,7 @@ def test_asset_page(db, client, setup_assets, as_prosumer_user1, view): # NB the automations listing is now one table per automation type, so there is no single #automationsTable to hide. assert b"`#automationsTable-${automationType}`" in asset_page.data assert b"columns.adjust();" in asset_page.data - assert b'title: "Schedule timezone"' in asset_page.data + assert b'title: "Recurrence timezone"' in asset_page.data assert b'title: "Next run (local)"' in asset_page.data assert b"timeZone: automation.timezone" in asset_page.data assert b'"next-run": nextRun(automation)' in asset_page.data @@ -118,7 +120,7 @@ def test_automations_page_manager_can_set_timezones(client, setup_assets, as_adm b"Use five fields: minute, hour, day of month, month, day of week." in response.data ) - assert b"The local clock used by the schedule." in response.data + assert b"The local clock the recurrence is read in." in response.data assert b'timezone: $("#automationTimezone").val()' in response.data assert b'timezone: $("#editAutomationTimezone").val()' in response.data