diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 9e6fd4c6fb..f8784aaaa1 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -94,7 +94,7 @@ Automations arrived over several pull requests. This is what each of them contri * Schedules as well as forecasts: a schedule automation stores what the schedule trigger endpoint accepts, and schedules from each run's own time [see `PR #2293 `_] * A scheduler's data source now also records the flex config the scheduler computed under, so a schedule can be traced back to the configuration that produced it, and a schedule automation points at such a data source, the way a forecast automation points at its forecaster's [see `PR #2464 `_] * A single automation can now be run on demand, from the CLI (``flexmeasures jobs run-automation``), the API (``POST /assets//automations//trigger``) and the asset's *Automations* page (a *Run now* button), which is useful to try out a new automation, to re-run one after fixing what made it fail, or to refresh its results after late input data arrived [see `PR #2460 `_] -* Automations can be created, edited and deleted in the UI and through new API endpoints (``[POST|PATCH|DELETE] /assets/(id)/automations``), by whoever may add data under the asset, with their recurrence expressed in a selectable IANA timezone, and only involving sensors they can access themselves (read access to the sensors an automation reads, and permission to record data on the sensors it writes to). The *Automations* page shows when each active automation is due to run next, as a clock time in the automation's own timezone, so a recurrence no longer has to be read back from its cron string; the same time is available as ``next_run`` on the automations API endpoints, and is null while an automation is inactive. It is the next scheduled clock time, so it excludes catch-up work still pending, and it follows the dispatcher's daylight-saving rules, taking the first fold of a repeated local time and the first valid minute after a skipped one. Each row's *Run now*, *Edit*, *Activate*/*Deactivate* and *Delete* controls are collected into a single *Actions* menu, leaving *Details* beside it, so a long listing carries two controls per row instead of five [see `PR #2294 `_ and `PR #2563 `_] +* Automations can be created, edited and deleted in the UI and through new API endpoints (``[POST|PATCH|DELETE] /assets/(id)/automations``), by whoever may add data under the asset, with their recurrence expressed in a selectable IANA timezone, and only involving sensors they can access themselves (read access to the sensors an automation reads, and permission to record data on the sensors it writes to). The *Automations* page shows when each active automation is due to run next, as a clock time in the automation's own timezone, so a recurrence no longer has to be read back from its cron string; the same time is available as ``next_run`` on the automations API endpoints, and is null while an automation is inactive. It is the next scheduled clock time, so it excludes catch-up work still pending, and it follows the dispatcher's daylight-saving rules, taking the first fold of a repeated local time and the first valid minute after a skipped one. Each row's *Run now*, *Edit*, *Copy*, *Activate*/*Deactivate* and *Delete* controls are collected into a single *Actions* menu, leaving *Details* beside it, so a long listing carries two controls per row instead of five. *Copy* opens the creation form filled in from that automation, and leaves the copy inactive [see `PR #2294 `_, `PR #2563 `_ and `PR #2591 `_] * An automation's output sensors are checked against its creator's permissions when the automation is created, and the schedules it computes are held to exactly those sensors: a scheduler that returns results for any other sensor is refused, rather than recording on a sensor that was only ever checked for read access [see `PR #2536 `_] * Reports as well as forecasts and schedules: a report automation stores report parameters, with its reporter and the reporter's configuration on a data source, and reports on a period resolved afresh on each run, either from ``start-offset`` and ``end-offset`` applied to the run time in the automation's timezone, or since the last successful report ended, while a fixed ``start`` or ``end`` is refused; a report job records only on the sensors the automation was checked against [see `PR #2297 `_] * Every automation times its runs the same way: a fixed ``start``, ``end`` or ``prior`` in its parameters is refused, as every run would share that moment, and two of ``start-offset``, ``end-offset`` and ``duration`` describe the period each run covers instead, with the offsets applied to the time the run was due on the automation's own clock, so that, for instance, a schedule automation can plan the whole of the next day [see `PR #2551 `_] diff --git a/documentation/features/automations.rst b/documentation/features/automations.rst index d27f4df6bb..0f0b5c28ca 100644 --- a/documentation/features/automations.rst +++ b/documentation/features/automations.rst @@ -298,6 +298,10 @@ An automation's *Info* panel shows the sensors it reads from and writes to, link It also summarizes the automation's recent runs and their outcomes (see :ref:`automation_runs`). Conversely, a sensor's page lists the automations that write data to it. +Each row's *Actions* menu also offers *Copy*, which opens the creation form with that automation's type, recurrence, timezone and parameters filled in, and its data source already selected. +It is a starting point for a variation on an existing automation, such as the same forecast on a different recurrence, rather than a command of its own: nothing is created until you submit the form, and you can change anything in it first. +The copy is created inactive, so that it does not start running while you are still working on it, and it is named after the original with a ``(copy)`` suffix. + .. _automation_cursor: Appendix: how the runner decides what is due diff --git a/flexmeasures/ui/templates/assets/asset_automations.html b/flexmeasures/ui/templates/assets/asset_automations.html index 26cb9b2a4f..3ddbb9285e 100644 --- a/flexmeasures/ui/templates/assets/asset_automations.html +++ b/flexmeasures/ui/templates/assets/asset_automations.html @@ -145,7 +145,7 @@

{% if user_can_manage_automations %}
- +
@@ -334,6 +334,8 @@
Edit automation
const userCanRunAutomations = {{ user_can_create_children | tojson }}; const automationTypes = {{ automation_types | tojson }}; const automationsById = new Map(); + // What the details endpoint said about each automation, kept so that copying one does not have to ask again. + const automationDetailsById = new Map(); let includeChildAssets = {{ 'true' if include_child_assets else 'false' }}; // How often the listing catches up with the server, while the tab is in front and nothing is open. const REFRESH_INTERVAL_MS = 60000; @@ -476,6 +478,68 @@
Edit automation
setGeneratorFieldsEnabled(true); } + /* A forecast and a report each run a data generator the creator chooses and configures; + a schedule automation's generator follows from the asset and its flex config instead. + The rule sits here, rather than in the ready callback, because filling the form in from an existing automation applies it too. */ + function typeChoosesGenerator() { + return $("#automationType").val() !== "scheduling"; + } + + function showGeneratorFields() { + $(".chooses-generator").toggle(typeChoosesGenerator()); + } + + /* Empty the creation form, so that the next automation starts from the defaults the form was rendered with. + The form is shared with the Copy action, which leaves the previous automation's settings in it, + so opening it blank has to clear them rather than trust that they were never there. */ + function resetNewAutomationForm() { + document.getElementById("newAutomationForm").reset(); + // Resetting restores the type the form was rendered with, and the generator fields follow whichever type that is. + clearSelectedSource(); + showGeneratorFields(); + $("#newAutomationErr").addClass("d-none").text(""); + } + + function prefillNewAutomationForm(values) { + resetNewAutomationForm(); + $("#automationName").val(values.name); + $("#automationType").val(values.type); + // The type decides whether a data generator can be named at all, so the fields follow it before anything is filled in. + showGeneratorFields(); + $("#automationCron").val(values.cron); + $("#automationTimezone").val(values.timezone); + $("#automationParameters").val(values.parameters); + $("#automationActive").prop("checked", values.active); + } + + // The name column allows 80 characters, and so does the API, so a long name loses its tail rather than the suffix that marks the copy. + const MAX_AUTOMATION_NAME_LENGTH = 80; + const AUTOMATION_COPY_SUFFIX = " (copy)"; + + function copiedAutomationName(name) { + const room = MAX_AUTOMATION_NAME_LENGTH - AUTOMATION_COPY_SUFFIX.length; + return `${String(name ?? "").slice(0, room)}${AUTOMATION_COPY_SUFFIX}`; + } + + /* What the creation form should hold to recreate an automation on this same asset. + The copy starts inactive, so that a variation being worked on does not start running while it is still being edited, + and so that two automations writing the same results never appear by a single click. + The sensors the parameters name stay valid, as the copy stays on this asset, so they are carried over as they are stored. + A forecast or report automation reuses the original's data source, which already stores the data generator and its configuration; + a schedule automation resolves its own source on every run, so it names none. */ + function automationCopyValues(automation, details) { + const sourceId = details?.source?.id; + return { + name: copiedAutomationName(automation.name), + type: automation.type, + cron: automation["cron"], + timezone: automation.timezone, + parameters: JSON.stringify(details?.parameters ?? {}, null, 4), + active: false, + sourceId: SOURCE_TYPE_PER_AUTOMATION_TYPE[automation.type] && sourceId ? sourceId : null, + }; + } + // Run this automation once, now, on top of its recurring runs. function runButton(automation) { if (!userCanRunAutomations) { @@ -492,6 +556,8 @@
Edit automation
` : ""; const manageItems = userCanManage ? `
    • +
    • @@ -628,6 +694,9 @@
    Automation '${esc(automation.name)}' (ID: ${automation.i url: automationUrl(automationId), method: "GET", success: function (res) { + // Copying an automation reads its parameters and its data source from here, rather than asking for them again. + automationDetailsById.set(automationId, res); + const jobStats = res["job-stats"] || {}; $(`#automation-jobs-${automationId}`).text(jobStatsText(jobStats)); @@ -907,6 +976,62 @@
    Recently created jobs bootstrap.Modal.getOrCreateInstance(document.getElementById("editAutomationModal")).show(); }); + // Open the creation form filled in from an existing automation, to create a variation on it. + function copyAutomationFrom(automation, details) { + const values = automationCopyValues(automation, details); + prefillNewAutomationForm(values); + bootstrap.Modal.getOrCreateInstance(document.getElementById("newAutomationModal")).show(); + if (values.sourceId) { + // The form is already open while this loads, so a slow read of the source does not hold up the rest of the copy. + $.ajax({ + url: `/api/v3_0/sources/${values.sourceId}`, + method: "GET", + success: (source) => showSelectedSource(source), + error: function (xhr) { + // Without the source, the copy would fall back to a default data generator rather than the original's, so say so plainly. + $("#newAutomationErr").removeClass("d-none").text( + `Could not read data source ${values.sourceId}, which this automation computes under:` + + ` ${automationAjaxErrorMessage(xhr)}.` + + " Name a data generator and its configuration below, or the copy will use the default one." + ); + }, + }); + } + } + + $(document).on("click", ".automation-copy", function () { + const automationId = Number($(this).data("id")); + const automation = automationsById.get(automationId); + if (!automation) { + showToast("Could not find the automation to copy. Refresh the page to try again.", "error"); + return; + } + const details = automationDetailsById.get(automationId); + if (details) { + copyAutomationFrom(automation, details); + return; + } + /* The page asks about one automation only when its panel is opened (see `loadAutomationInfo`), + so a copy reads the parameters and the data source of this one now. */ + $.ajax({ + url: automationUrl(automationId), + method: "GET", + success: function (res) { + automationDetailsById.set(automationId, res); + copyAutomationFrom(automation, res); + }, + error: function (xhr) { + showToast( + `Could not read this automation to copy it: ${automationAjaxErrorMessage(xhr)}.`, + "error" + ); + }, + }); + }); + + // Opening the form from the New automation button starts from the defaults, not from whatever a copy left in it. + $("#newAutomationButton").on("click", () => resetNewAutomationForm()); + $("#editAutomationForm").submit(function (event) { event.preventDefault(); const automationId = $("#editAutomationId").val(); @@ -954,14 +1079,6 @@
    Recently created jobs }); }); - /* A forecast and a report each run a data generator the creator chooses and configures; - a schedule automation's generator follows from the asset and its flex config instead. */ - function typeChoosesGenerator() { - return $("#automationType").val() !== "scheduling"; - } - function showGeneratorFields() { - $(".chooses-generator").toggle(typeChoosesGenerator()); - } $("#automationType").on("change", function () { showGeneratorFields(); // The sources offered are of the automation's own type, so a selection made for another type no longer applies. diff --git a/flexmeasures/ui/tests/js/test_automation_actions.py b/flexmeasures/ui/tests/js/test_automation_actions.py index 0b6409e76a..2bb030e9fd 100644 --- a/flexmeasures/ui/tests/js/test_automation_actions.py +++ b/flexmeasures/ui/tests/js/test_automation_actions.py @@ -74,7 +74,7 @@ def test_automation_actions_are_grouped_and_permission_gated(assert_js): recurrence_description: "At 06:00", next_run: "2026-09-15T04:00:00+00:00", }}; const row = manager.AutomationRow(automation); - check("one menu holds all four controls", ["run-automation", "automation-edit", "automation-toggle", "automation-delete"] + check("one menu holds all five controls", ["run-automation", "automation-edit", "automation-copy", "automation-toggle", "automation-delete"] .every(name => row.actions.includes(name)), row.actions); check("run now is no longer mixed with info", !row.info.includes("run-automation"), row.info); check("info stays its own button", row.info.includes("automation-info"), row.info); @@ -84,8 +84,8 @@ def test_automation_actions_are_grouped_and_permission_gated(assert_js): const toggle = holder.querySelector(".automation-actions .dropdown-toggle"); check("the row offers a single Actions toggle", toggle !== null && holder.querySelectorAll(".automation-actions > .btn").length === 1, row.actions); - check("the four actions sit in its menu as items", - holder.querySelectorAll(".automation-actions .dropdown-menu .dropdown-item").length === 4, + check("the five actions sit in its menu as items", + holder.querySelectorAll(".automation-actions .dropdown-menu .dropdown-item").length === 5, row.actions); const inactive = manager.AutomationRow({{...automation, active: false, next_run: null}}); check("inactive action says Activate", inactive.actions.includes("Activate") && !inactive.actions.includes("Deactivate"), inactive.actions); diff --git a/flexmeasures/ui/tests/js/test_automation_copy.py b/flexmeasures/ui/tests/js/test_automation_copy.py new file mode 100644 index 0000000000..64cfff6b42 --- /dev/null +++ b/flexmeasures/ui/tests/js/test_automation_copy.py @@ -0,0 +1,218 @@ +"""Browser checks for copying an automation into the automation page's creation form.""" + +import json +import re + +import pytest + +# The page's script is rendered by one helper, which asserts that it leaves no Jinja value behind. +# Sharing it keeps these checks from breaking whenever the page reads another value. +from test_automation_actions import TEMPLATE, automation_script + + +@pytest.fixture(scope="module", autouse=True) +def setup_ui_test_data(): + """The inline-script checks do not need the UI suite's database fixtures.""" + + +def copy_script(can_manage: bool = True) -> str: + """Render the page's inline JavaScript, as someone who may copy an automation sees it.""" + return automation_script(can_manage=can_manage, can_run=True) + + +def creation_form_html() -> str: + """The creation form as the page renders it, so the checks run against the real fields.""" + match = re.search( + r'(
    .*?
    )', TEMPLATE.read_text(), re.DOTALL + ) + assert match is not None + return match.group(1).replace("{{ asset.timezone }}", "Europe/Amsterdam") + + +# A stand-in for the handful of jQuery calls the form helpers make, backed by the real DOM. +JQUERY_STUB = """ + window.$ = function (selector) { + const nodes = typeof selector === "string" ? [...document.querySelectorAll(selector)] : []; + const self = { + ready: () => {}, + on: () => self, + val: function (value) { + if (value === undefined) return nodes.length ? nodes[0].value : undefined; + nodes.forEach(node => { node.value = value; }); + return self; + }, + prop: function (name, value) { + nodes.forEach(node => { node[name] = value; }); + return self; + }, + is: (what) => what === ":checked" && nodes.length > 0 && nodes[0].checked, + text: function (value) { + nodes.forEach(node => { node.textContent = value; }); + return self; + }, + empty: function () { + nodes.forEach(node => { node.innerHTML = ""; }); + return self; + }, + addClass: function (name) { + nodes.forEach(node => node.classList.add(name)); + return self; + }, + removeClass: function (name) { + nodes.forEach(node => node.classList.remove(name)); + return self; + }, + toggle: function (on) { + nodes.forEach(node => { node.style.display = on ? "" : "none"; }); + return self; + }, + toggleClass: function (name, on) { + nodes.forEach(node => node.classList.toggle(name, on)); + return self; + }, + }; + return self; + }; +""" + + +def test_a_copy_carries_the_settings_over_and_starts_inactive(assert_js): + """A copy recreates the original on the same asset, except that it does not start running.""" + assert_js(f""" + window.$ = () => ({{ready: () => {{}}}}); + const page = new Function({json.dumps(copy_script())} + + "\\nreturn {{ automationCopyValues, copiedAutomationName }};")(); + + const automation = {{ + id: 7, name: "Campus forecast", type: "forecasting", active: true, + "cron": "0 6 * * *", timezone: "Europe/Amsterdam", + }}; + const details = {{parameters: {{sensor: 2092, "start-offset": "1D,DB"}}, source: {{id: 6, description: "Seita's forecaster"}}}}; + const values = page.automationCopyValues(automation, details); + + eq("the copy is marked as one in its name", values.name, "Campus forecast (copy)"); + eq("the recurrence is carried over", values.cron, "0 6 * * *"); + eq("and so is the timezone it is read in", values.timezone, "Europe/Amsterdam"); + eq("the type is carried over", values.type, "forecasting"); + eq("the parameters are carried over as stored", JSON.parse(values.parameters), details.parameters); + eq("a copy of an active automation still starts inactive", values.active, false); + eq("a forecast copy reuses the original's data source", values.sourceId, 6); + """) + + +def test_a_copy_only_reuses_a_data_source_where_one_can_be_named(assert_js): + """A schedule automation resolves its source on every run, so a copy of one must not name the original's.""" + assert_js(f""" + window.$ = () => ({{ready: () => {{}}}}); + const page = new Function({json.dumps(copy_script())} + + "\\nreturn {{ automationCopyValues }};")(); + + const source = {{id: 6, description: "Seita's generator"}}; + const scheduleCopy = page.automationCopyValues( + {{id: 8, name: "Battery schedule", type: "scheduling", "cron": "0 * * * *", timezone: "UTC"}}, + {{parameters: {{}}, source: source}}); + eq("a schedule copy names no data source", scheduleCopy.sourceId, null); + + const reportCopy = page.automationCopyValues( + {{id: 9, name: "Weekly report", type: "reporting", "cron": "0 0 * * 1", timezone: "UTC"}}, + {{parameters: {{}}, source: source}}); + eq("a report copy reuses the original's data source", reportCopy.sourceId, 6); + + const sourceless = page.automationCopyValues( + {{id: 10, name: "Orphan", type: "forecasting", "cron": "0 0 * * *", timezone: "UTC"}}, + {{parameters: {{}}}}); + eq("an automation whose details carry no source names none", sourceless.sourceId, null); + eq("and its parameters still default to an empty object", JSON.parse(sourceless.parameters), {{}}); + """) + + +def test_a_long_name_keeps_the_suffix_that_marks_the_copy(assert_js): + """The name field and the API both stop at 80 characters, so the tail goes rather than the suffix.""" + assert_js(f""" + window.$ = () => ({{ready: () => {{}}}}); + const page = new Function({json.dumps(copy_script())} + + "\\nreturn {{ copiedAutomationName }};")(); + + const long = page.copiedAutomationName("x".repeat(120)); + eq("a long name is cut to what the field holds", long.length, 80); + check("and still says it is a copy", long.endsWith(" (copy)"), long); + eq("a short name is left alone apart from the suffix", + page.copiedAutomationName("Campus forecast"), "Campus forecast (copy)"); + """) + + +def test_opening_a_blank_form_clears_what_a_copy_left_in_it(assert_js): + """The creation form is shared with the Copy action, so New automation has to clear it rather than inherit a copy.""" + assert_js(f""" + // Appended rather than assigned to the body, which holds the harness's own results element. + const holder = document.createElement("div"); + holder.innerHTML = {json.dumps(creation_form_html())}; + document.body.appendChild(holder); + {JQUERY_STUB} + const page = new Function({json.dumps(copy_script())} + + "\\nreturn {{ prefillNewAutomationForm, resetNewAutomationForm, automationCopyValues }};")(); + + const values = page.automationCopyValues( + {{id: 7, name: "Campus forecast", type: "reporting", "cron": "30 7 * * *", timezone: "Africa/Tunis"}}, + {{parameters: {{sensor: 2092}}, source: {{id: 6}}}}); + page.prefillNewAutomationForm(values); + + eq("the copy fills the name in", document.getElementById("automationName").value, "Campus forecast (copy)"); + eq("and the recurrence", document.getElementById("automationCron").value, "30 7 * * *"); + eq("and the timezone", document.getElementById("automationTimezone").value, "Africa/Tunis"); + eq("and the type", document.getElementById("automationType").value, "reporting"); + eq("and the parameters", JSON.parse(document.getElementById("automationParameters").value), {{sensor: 2092}}); + eq("and leaves the copy inactive", document.getElementById("automationActive").checked, false); + + const generatorFields = () => [...document.querySelectorAll(".chooses-generator")]; + check("a report copy can still name a data generator", + generatorFields().length > 0 && generatorFields().every(field => field.style.display !== "none"), + "hidden for a type that chooses its own generator"); + + // A schedule automation works its generator out from the asset on every run, so a copy of one may not name it. + page.prefillNewAutomationForm(page.automationCopyValues( + {{id: 8, name: "Battery schedule", type: "scheduling", "cron": "0 * * * *", timezone: "UTC"}}, + {{parameters: {{}}, source: {{id: 6}}}})); + check("a schedule copy hides the data generator fields", + generatorFields().every(field => field.style.display === "none"), + "still shown for a schedule automation"); + + page.resetNewAutomationForm(); + + eq("opening a blank form clears the copied name", document.getElementById("automationName").value, ""); + eq("and the copied recurrence", document.getElementById("automationCron").value, ""); + eq("and the copied parameters", document.getElementById("automationParameters").value, ""); + eq("and restores the asset's own timezone", + document.getElementById("automationTimezone").value, "Europe/Amsterdam"); + eq("and the default type", document.getElementById("automationType").value, "forecasting"); + eq("and a new automation is active again, as the form is rendered", + document.getElementById("automationActive").checked, true); + check("and the data generator fields are shown again, after a schedule copy hid them", + generatorFields().every(field => field.style.display !== "none"), "still hidden"); + check("and the data generator fields are usable again, whatever the copy did to them", + !document.getElementById("automationGenerator").disabled + && !document.getElementById("automationConfig").disabled, "still disabled"); + """) + + +def test_copy_is_offered_to_managers_only(assert_js): + """Copying creates an automation, so it sits behind the same permission as creating one.""" + manager_script = copy_script(can_manage=True) + viewer_script = copy_script(can_manage=False) + assert_js(f""" + window.$ = () => ({{ready: () => {{}}}}); + const manager = new Function({json.dumps(manager_script)} + "\\nreturn {{ AutomationRow }};")(); + const viewer = new Function({json.dumps(viewer_script)} + "\\nreturn {{ AutomationRow }};")(); + const automation = {{ + id: 7, name: "Campus forecast", type: "forecasting", active: true, + "created-at": null, "cron": "0 6 * * *", timezone: "Europe/Amsterdam", + "recurrence-description": "At 06:00", "next-run": null, + }}; + const managerRow = manager.AutomationRow(automation); + check("a manager is offered Copy", managerRow.actions.includes("automation-copy"), managerRow.actions); + check("and it carries the id to copy", managerRow.actions.includes('class="dropdown-item automation-copy" data-id="7"'), + managerRow.actions); + const viewerRow = viewer.AutomationRow(automation); + check("someone who may not manage automations is not offered Copy", + !viewerRow.actions.includes("automation-copy"), viewerRow.actions); + """)