Automations UI polish: edit action and batch job stats - #2299
Draft
Flix6x wants to merge 9 commits into
Draft
Conversation
- The automations page gets an Edit action (name and cron string), using the existing PATCH endpoint. - The list endpoint now includes per-automation job stats, computed in a single pass over the relevant job caches, instead of the UI firing one detail request per automation on page load; the Details modal loads its contents lazily, when first opened. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Documentation build overview
144 files changed ·
|
…s-ui-polish # Conflicts: # flexmeasures/data/services/automations.py
Context: PR #2299 was based on an earlier revision of the report automation stack and conflicted with the newer timezone, activation, report sensor, and error-handling behavior. Change: Merge the current report automation branch while retaining the batched job-stat listing and lazy detail loading. The reconciled editor continues to update name, recurrence, timezone, and activation state. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: The batched automation listing failed with HTTP 500 on Redis timeouts and schedule statistics only scanned sensors named in stored parameters. A minimal schedule can inherit device sensors from descendant assets, so its child jobs were omitted. Change: Treat Redis client failures as unavailable statistics while returning the automation list with a stable public error. Scan scheduling cache entries for the complete asset subtree once per listing, while continuing to deduplicate jobs cached under multiple entities. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: The existing coverage proved only duplicate suppression for one schedule job and assumed Redis failures always used the service wrapper exception. It did not exercise a distinct descendant job or a real Redis client timeout. Change: Verify that distinct child-sensor schedule jobs are counted while duplicate cache references remain deduplicated. Verify that a Redis timeout leaves the list response usable, clears statistics, exposes a safe message, and does not leak connection details. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: PR #2299 changes the automation list contract and the UI interaction model, but the branch had no user-facing or API changelog entry for those additions. Change: Document the batched job counts, lazy detail loading, complete recurrence editor, and Redis-unavailable response behavior in the main and API changelogs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…s-ui-polish Context: - Picking up the review changes from #2290 and the automations documentation page as it now stands. Change: - Kept this branch's wording for the automations listing endpoint, which describes its batched job counts, with the cursor no longer called a scheduling cursor. - Kept both ancestor and descendant query helpers in the automations service imports. - Dropped the API changelog entry for timezone and cursor, folded into the endpoint entry upstream, and kept this branch's job_stats entry. - Moved the automation management endpoints from v3.0-32 to v3.0-33, so all of this unreleased automation work is described in one version section. Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x
added a commit
that referenced
this pull request
Sep 1, 2026
Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl>
…to HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x
added a commit
that referenced
this pull request
Sep 1, 2026
* feat: add Automation data model Automations are recurring tasks (for now: computing forecasts) defined per asset. The recurrence is defined by a cron string, and the work to be done is defined by a data generator (linked through a data source) together with the parameters to call it with. Includes a migration for the new table, and new dependencies on croniter (cron matching/validation) and cron-descriptor (natural-language recurrence descriptions). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: CLI commands to manage and run automations - `flexmeasures add automation` creates an automation (active by default), validating the forecast parameters with the forecast parameter schema and storing the forecaster config on a data source. - `flexmeasures edit automation` edits the name, recurrence (cron string) or activation status. - `flexmeasures delete automation` deletes an automation. - All three record their events in the asset's audit log. - `flexmeasures jobs run-automations` queues jobs for all automations due this minute (to be run once per minute, e.g. via cron), with a Redis-based guard against duplicate runs within the same minute. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: record on forecasting jobs how they were created Data generators can now be told how their queued jobs got triggered (via the CLI, the API or an automation), and the train-predict pipeline stores this on the jobs as meta data. The asset's status page shows it in a new 'Created Via' column of the jobs table. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: API endpoints to list an asset's automations GET /api/v3_0/assets/<id>/automations lists the automations defined on an asset (without generator and parameters details). GET /api/v3_0/assets/<id>/automations/<automation_id> additionally provides the parameters, data generator info and counts of recently created jobs per job status. Both are documented in the OpenAPI specs. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * feat: UI page listing an asset's automations /assets/<id>/automations shows the asset's automations in a tabbed view (schedules and reports tabs are prepared but deactivated), with per-row details (parameters, data generator, job counts) loaded asynchronously into a modal. The page is linked in the breadcrumbs dropdown and links to the status page, where recent jobs are listed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: cover automations CLI, API and UI Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: document automations Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * docs: changelog entry for automations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: render cron descriptions in 24-hour format regardless of locale CI runners have no locale set (POSIX), which made cron-descriptor render 'At 06:00' while dev environments with an en_US-style locale rendered 'At 06:00 AM'. Request 24-hour format explicitly so the description is deterministic across environments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pxkeq64jtENY7fiWjwUsVS * fix: address code review findings for automations - Escape automation names (and other user-controlled strings) in the Automations page and the status page's jobs table, closing two stored HTML/script injection sinks. - Wipe parameter state on the (possibly shared) cached data generator before each automation run, so automations sharing a generator data source don't pollute each other's runs. - Count automation job stats under the forecast target sensor(s) from the automation's parameters, which may belong to a different asset. - Release the per-minute Redis guard when a run fails, so a retry within the same minute can still queue jobs. - Return 404 (as documented) for nonexistent automation ids on the detail endpoint, and check permissions on the asset, so automation ids can no longer be enumerated across accounts via 403-vs-422 differences. - Use ondelete=SET NULL for the generator FK: deleting a data source no longer silently deletes automations. - Delegate Automation ACL to the asset's ACL instead of duplicating it. - Extract the config/parameters assembly shared by `add forecasts` and `add automation` into a helper (which no longer drops falsy config values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * fix: address code review findings for automations (remaining files) Completes the previous commit, whose staged files were dropped by an interrupted pre-commit run: template escaping, shared-generator state reset, job stats under target sensors, Redis guard release on failure, 404 for nonexistent automations, SET NULL generator FK, ACL delegation, and the shared CLI config/parameters assembly helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * test: assert on the cron validation failure without pinning click's message format PR #2303 makes click report the validation message rather than the offending value, which changes the exact wording of this error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX * data/schemas: restrict automations to five-field cron Reject cron expressions with seconds, year fields, or aliases because the automation runner executes once per minute. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/schemas/tests: cover automation cron field count Test valid five-field expressions and reject unsupported seconds, year, and alias formats. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/jobs: retain automation guard after queueing failure Keep the per-minute Redis guard after failures because an attempt may already have queued some forecast jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover partial automation queue failure Verify that retrying a failed partial queueing attempt does not create duplicate jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli: normalize YAML forecasting option files Convert YAML dates to ISO strings, accept empty files, and report non-object config or parameter files as usage errors. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * cli/tests: cover automation YAML option files Test YAML dates and timestamps, empty files, and invalid top-level list values for automation options. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/services: redact inaccessible automation provenance Hide automation names and IDs from asset job responses when the current user cannot read the source automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/v3_0/tests: cover automation provenance authorization Verify inaccessible automation provenance is redacted while authorized callers still receive the full identity. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/assets: distinguish automation load failures Show a persistent API error instead of presenting failed automation requests as an empty list. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * ui/tests: cover automation load error state Check that the automations page renders the warning target and hides the table when loading fails. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/docs: preserve standalone asterisks in RST conversion Avoid interpreting cron wildcard asterisks as RST italic markup when generating OpenAPI documentation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * utils/tests: cover RST cron wildcard conversion Verify cron wildcards remain unchanged while ordinary italic markup is still converted. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs/forecasting: clarify automation execution contract Document five-field cron expressions, at-most-once queueing attempts, and the automations API endpoint. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation API and runner contract Record the automation endpoints, authorization-aware provenance, and five-field runner behavior. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * api/docs: show job creation provenance Include the created_via field in the asset jobs OpenAPI example. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: keep forecast CLI stub compatible with job provenance Add the trigger method required by the forecasting CLI to the regressor parsing test double. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: require valid automation generators Require every automation to reference a data generator and prevent deleting a data source while an automation still depends on it, matching the retention policy for belief and annotation sources. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover automation generator retention Verify referenced generators cannot be deleted, generator references cannot be cleared, and automation API fixtures always use valid generators. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: constrain forecast automation outputs Allow forecast output only on the automation asset or its descendants and revalidate that relationship before every scheduled run, including explicit sensor-to-save targets. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover forecast automation output scope Cover same-asset, child, grandchild, ancestor, and unrelated output targets, explicit sensor-to-save behavior, and runtime revalidation after an asset is moved. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain forecast automation ownership rules Document output-sensor scope, runtime relationship checks, and the requirement to retain a generator while its automation exists. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix: merge automation and main migration heads Join the automation and main Alembic branches so installations have a single database upgrade target. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: let data generators report their input and output sensors Context: - Review of #2290 asked for a data generator property listing the sensors it reads from and writes to, so that automations can link to those sensors (and, later, check the creating user's permissions on them) Change: - Added input_sensors and output_sensors to DataGenerator (empty by default), implemented for Forecaster from its regressors and target sensor - Added the same properties to Automation, resolved from its data generator configured with the automation's own parameters - Added get_automations_feeding_sensor to look up automations by output sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models/forecasting: only announce a pipeline run when actually running it Context: - 'flexmeasures jobs run-automations' logged 'Starting Train-Predict Pipeline' for every automation, while it only queues the cycles as jobs Change: - Log that line at debug level when running with as_job, where the workers running the cycles log their own start Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: default the automation recurrence to daily, and reject options that --source already determines Context: - Review of #2290: --cron should not be required, and --forecaster/--config are redundant with --source, whose data generator attributes already hold both (get_data_generator silently ignores them when a source is given) Change: - Default --cron to '0 0 * * *' (daily at midnight) - Abort when --source is combined with --forecaster, --config or any of the forecaster configuration options, naming the conflicting options Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: report an automation's input and output sensors Context: - The automation details modal should link to the sensors an automation feeds Change: - Added input_sensors and output_sensors (id and name each) to GET /assets/<id>/automations/<automation_id> Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: add an endpoint for one data source Context: - The sensor page should be able to show the full record of a data source, including the attributes in which data generators store their configuration Change: - Added GET /sources/<id>, with the same access rules as listing sources - Let _serialize_source optionally include the attributes and unset fields Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs Context: - The specs are generated from the endpoint docstrings by a pre-commit hook Change: - Regenerated after adding the data source endpoint and the automation's input and output sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: link an automation's details to its sensors, and make the listing sortable Context: - Review of #2290: the details modal should link to the sensors an automation feeds, with its data source pre-selected there, and the listing was not sortable Change: - Show the input and output sensors in the details modal, linking to /sensors/<id>?source=<generator id> - Enabled ordering, sorting the rendered columns on separate values (the ISO timestamp, the activation status and the cron string), newest first Signed-off-by: F.N. Claessen <felix@seita.nl> * ui: show a sensor's data source record and the automations feeding it Context: - Review of #2290: the sensor page should be able to show all details of a data source, and list the automations that write data to the sensor Change: - Added an info button next to the source selector, opening a modal with the full data source record - Pre-select the source given in the source query parameter, so that links from an automation land on its own source - List the automations feeding the sensor (those the user may read), linking to the automations page of their asset - Added user_can_read to the UI's permission helpers Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the automation and data source review follow-ups Context: - New behaviour from the #2290 review needs regression coverage Change: - CLI: the daily default recurrence, the --source conflict, and an automation's input and output sensors - API: an automation without a data generator reports no sensors; the new data source endpoint, its access rules and its 404 - UI: the source query parameter reaches the page, and automations feeding a sensor are listed on it Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: describe the automation and data source follow-ups Context: - The #2290 review changed user-facing CLI, API and UI behaviour Change: - Documented the daily default recurrence and reusing a forecaster via --source - Documented the links between automations and the sensors they feed - Added API change log entries for the automations and data source endpoints - Extended the changelog entry of #2290 Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: regenerate the OpenAPI specs after merging Context: - The merge combined endpoint docstring changes from both sides Change: - Regenerated the specs Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: only reject configuration options that were actually given with --source Context: - The guard added on this branch compared against the assembled config, which always holds the schema defaults of the list-valued options, so any use of --source was rejected Change: - Detect the conflicting options from click's parameter sources, so --source on its own works again while explicitly given configuration options still abort - Name the conflicting options in the error message Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: only consider automations that could feed a sensor Context: - Listing the automations feeding a sensor sets up a data generator per candidate, which does not need to happen for every automation in the database Change: - Narrowed the candidates to automations on the sensor's asset or an ancestor, which is where an automation writing to it must live Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the merged automation behaviour Context: - Automations now always have a data generator, and the --source guard also covers the forecaster configuration options Change: - Assert the sensors an automation with a generator reads from and writes to - Cover a configuration option conflicting with --source Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: keep mypy happy about click 8 attributes types-Flask pins types-click 7.1, whose stubs shadow the inline types that click ships itself. Those stubs predate ParameterSource and Context.get_parameter_source, both added in click 8.0, so mypy rejected the --source conflict detection and pre-commit failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * cli: keep the automation help focused on the automation `add automation` reuses the forecast schemas, so Click rendered every forecaster and pipeline option in its help, burying the options that describe the automation itself. Accept those options still, but hide them, and let the parameters file supply a field the schema requires, so --sensor no longer has to be repeated on the command line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/models: count a source-filtered regressor as an input sensor SensorIdOrReferenceField deserializes a regressor that filters on sources into a SensorReference rather than a Sensor, which _resolve_sensors skipped, so those regressors were missing from a data generator's input sensors. The source filters only narrow down which beliefs are read from a sensor, not which sensor is involved, so the wrapped sensor counts as an input just like a plain sensor ID does. This matters beyond the sensor links in the UI: the input and output sensors are meant to carry the access checks for automations administered through the API, where a missing input sensor means a missing permission check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * data/services: do not report no sensors when an automation's sensors are unknown Working out an automation's sensors could fail for several reasons, and every one of them was reported as no sensors at all. That is fine for the sensor links in the UI, but the same answer is meant to carry the access checks for automations administered through the API, where no sensors reads as nothing to check, so a broken automation would pass every check on the sensors it involves. Split the two uses: resolve_automation_sensors raises AutomationSensorsUnknown, while get_automation_sensors keeps reporting none for display. The broad exception handler is narrowed to the failures that can actually occur here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Feat automation timezones catchup (#2396) * feat: add timezone-aware automation catch-up Store each automation's IANA timezone and a durable UTC scheduling watermark. Canonicalize daylight-saving transitions, coalesce missed forecasts, and claim occurrences before queueing while retaining the existing at-most-once failure policy. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test: cover timezone-aware automation scheduling Exercise timezone defaults and validation, independent timezone evaluation, normal and missed occurrences, DST gaps and folds, durable cursor claims, inactive automations, API/UI exposure, and the existing Redis failure guard. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: explain automation timezone and catch-up semantics Document per-automation timezone snapshots, scheduling watermarks, downtime coalescing, daylight-saving behavior, reconfiguration boundaries, and the at-most-once retry limitation. Refresh the published automation API examples. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * changelog: record automation timezone and catch-up support Announce the new CLI options, additive automation response fields, persistent scheduling progress, and DST-aware catch-up behavior for issue #2392. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * docs: link automation catch-up changelog to PR Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> * fix(data/schemas): reject cron expressions without dates Validate that a syntactically correct five-field recurrence can produce an actual calendar occurrence, preventing impossible dates such as February 31 from entering the automation table through either CLI or API input. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(data/services): isolate invalid recurrences and stale claims Keep one legacy or corrupted recurrence from aborting global discovery, and make occurrence claiming an atomic comparison against the active state, recurrence, timezone, and cursor observed by the runner so concurrent edits cannot queue stale work. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * fix(api/v3_0): protect automation sensor details Require read access to every resolved input and output sensor before returning full automation details, ensuring the derived metadata and raw parameters cannot reveal cross-organisation sensor information to an asset reader. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(cli): cover impossible recurrence input Exercise the CLI validation path with a syntactically valid recurrence that can never match, while updating the runner fixture to carry the scheduling snapshot required by atomic occurrence claims. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(data/services): cover resilient automation claims Verify that corrupt recurrences are isolated and that deactivation, deletion, recurrence edits, timezone edits, or cursor movement after discovery prevent a stale claim, while non-execution name edits remain harmless. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * test(api/v3_0): cover private automation dependencies Build an automation with a supplier-owned regressor and confirm that a plain member who may read the automation asset receives a generic denial without the inaccessible sensor name. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> * data/models: name an automation's cursor after what it points at Context: - Review of #2396 asked what the "scheduling cursor" is, how an automation is "watermarked" (watermarks do not update), and why the field is needed at all. - "scheduling" also collides with FlexMeasures' scheduling machinery (the "scheduling" queue, StorageScheduler), which this field has nothing to do with. - The feature is unreleased, so the column, the API field and the migration can still be renamed without a compatibility burden. Change: - Renamed Automation.scheduling_cursor to Automation.cursor, and get_initial_scheduling_cursor to get_initial_cursor. As a column on the automation table, it reads as an automation's cursor without further qualification, like the neighbouring timezone column. - Replaced the "watermark" wording everywhere with what the field holds: the scheduled time of the most recent run the automation committed to, advanced just before queueing, and therefore not a record of success. - Said "run" instead of "occurrence" throughout the automation code, matching the vocabulary already used for run time, run-automations and the automation-run guard key. - Explained in the migration why both columns are added nullable and backfilled before NOT NULL, and why the backfilled cursor is one minute before the upgrade. - Regenerated the OpenAPI specs. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: follow the automation cursor rename Context: - Automation.scheduling_cursor became Automation.cursor, and automation "occurrences" became "runs". Change: - Updated the field name in the automation fixtures and assertions, and the UI assertion on the "Cursor (UTC)" heading. - Renamed the coalescing and spring-forward test cases to speak of runs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain the automation cursor, and say "run" instead of "occurrence" Context: - Review of #2396 found "the cursor is a watermark", "migration watermark", "the cursor is committed" and "durable run records" unclear, and asked why the cursor is needed at all. - The docs already spoke of a run time, run records and run-automations, so "occurrence" was a second word for the same thing. Change: - Introduced the cursor by the problem it solves: the runner is a stateless once-a-minute command, so it needs a durable record of how far each automation has got. - Stated what it holds, that it advances before queueing (so it is not a record of success), and that keeping one moving timestamp instead of a record per run is what produces the catch-up and concurrency behaviour described below it. - Replaced the "migration watermark" sentence with what an upgrade actually does to existing automations. - Linked issue #2393 where the docs referred to "durable run records". - Applied the suggested wording for the "add automation" command summary. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: give the automation API changes their own version section Context: - Review of #2290 asked to move the automation API entries to a new v3.0-33 section, and noted that a revision to a CLI command introduced in the same version does not warrant its own entry. Change: - Moved the automation and data source entries from v3.0-32 to a new "v3.0-33 | September 1, 2026" section. - Folded the timezone and cursor entry into the entry introducing the automation endpoints, applying the same reasoning as for the CLI changelog, and described the cursor in terms of the run it points at. - Folded the --timezone and catch-up entry into the two CLI entries introducing the commands it revises. - Folded the #2396 entry in the main changelog into the #2290 entry it refines, listing both PRs. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: restore the v3.0-32 underline to full length Context: - This branch had shortened the underline of "v3.0-32 | August 11, 2026" from 26 to 24 characters, one short of the 25-character title, which makes docutils warn that the title underline is too short. Change: - Set the underline to exactly the title length. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/services: address review findings on the automations service Context: - Reviewing #2290 turned up three issues in how the automations service handles shared state and asset trees. Change: - run_automation now works on a copy of the data generator, like resolve_automation_sensors already did. The generator is cached on the data source, which several automations may share, so setting the job trigger on the shared instance would attribute jobs to the wrong automation as soon as anything runs concurrently. - Moved the upward tree walk to asset_and_ancestor_ids in data/queries/generic_assets, and expressed asset_is_in_subtree in terms of it, so the two copies of that walk introduced by this branch became one. - Added get_automations_involving_sensor, which considers every automation rather than only those on the sensor's asset and its ancestors, because a regressor may live anywhere in the tree. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/models: index the automation asset foreign key Context: - PostgreSQL does not index a foreign key by itself, and automations are looked up by asset on an asset's automations page and when finding the automations that feed a sensor. - Reviewing #2290 also showed that a bool would be read as a sensor ID, as bool is a subclass of int. Change: - Added an index on automation.asset_id, in a new migration rather than in the migration that creates the table, so a database that already ran that one still gets the index. - Excluded bools from the integer branch of DataGenerator._resolve_sensors. Signed-off-by: F.N. Claessen <felix@seita.nl> * api/v3_0: work out an automation's sensors once when they cannot be resolved Context: - On the error path, the automation details endpoint called resolve_automation_sensors and then get_automation_sensors, which calls resolve_automation_sensors again and swallows the error, so a broken automation set up its data generator and loaded its parameters twice. Change: - Log the reason and fall back to empty sensor lists directly, which is what the second call amounted to. Signed-off-by: F.N. Claessen <felix@seita.nl> * cli: warn which automations a sensor deletion would break Context: - An automation refers to its sensors by ID inside its parameters, which no foreign key protects. Deleting such a sensor left the automation looking healthy while failing on its next run, with the reason visible only in the runner's output. - A data source is protected from this by a foreign key, so the sensors were the remaining gap. Change: - flexmeasures delete sensor now lists the automations that read from or write to each sensor before asking for confirmation. The deletion is still allowed, as the host may well intend it. Signed-off-by: F.N. Claessen <felix@seita.nl> * tests: cover the sensor deletion warning, and stop depending on caplog Context: - test_invalid_cron_does_not_hide_other_due_automations failed whenever an earlier test in the session had built an app: creating one reconfigures logging and replaces the root handlers, after which pytest's caplog captures nothing. Reproduced with utils/tests/test_job_utils.py::test_app_queues_use_custom_global_and_queue_job_timeout running first. - The condition is pre-existing and hits any test that reads caplog afterwards, including data/tests/test_utils.py::test_schema_mismatch_log_record_is_deduplicated on main, which already uses caplog.at_level. So at_level is not a workaround: the handler is gone, not merely filtered. - The test's behavioural assertion passed throughout; only the log assertion failed. Change: - Assert on the logger itself rather than on caplog, which makes the test independent of what ran before it. - Cover that deleting a sensor names the automations using it, including a regressor-only sensor, which get_automations_feeding_sensor does not find. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs/changelog: record the sensor deletion warning Context: - flexmeasures delete sensor now warns which automations use a sensor. Change: - Added a CLI changelog entry, as delete sensor is a pre-existing command rather than one introduced in this version. - Folded the behaviour into the automations entry in the main changelog, which already covers this feature. Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: give automations their own page Context: - Review of #2290 found the automations section sitting under forecasting, although most of it will apply to scheduling and reporting automations too. - The same review found the CLI example silent about what it automates, --forecaster and --config referred to before being introduced, the daylight-saving-time rules reading as a developer's note in the main body, and the pointer to issue #2393 reading as a todo. Change: - Moved the section to features/automations.rst, split into creating, running and viewing automations, and left a pointer in features/forecasting.rst. Wrote it in terms of automations in general, mentioning forecasts as today's only type. - Made the example pass --type forecasts explicitly, which is the option the follow-up PRs use to distinguish schedules and reports. - Introduced --forecaster and --config before the sentence that says --source makes them unnecessary. - Moved the cursor and daylight-saving-time rules to an appendix, marked as bookkeeping you do not need in order to use automations. - Dropped the sentence pointing at issue #2393, keeping the statement that a failed attempt is not retried, which is the part users need. Signed-off-by: F.N. Claessen <felix@seita.nl> * data/migrations: index the automation asset FK without adding a revision Context: - The separate revision added for this index branched off 9f2b6e1d4a73, but so does c63896a97a8e on the branches stacked on top of this one. Merging this branch down therefore left two alembic heads, and `flexmeasures db upgrade` fails on multiple heads. That broke the Docker build job on #2293, #2294, #2297 and #2299, while this PR itself stayed green with its single head. - Adding the index in its own revision was meant to spare a database that had already run 9f2b6e1d4a73. Breaking the upgrade on four stacked PRs is the greater harm, so that trade-off no longer holds. Change: - Folded the index into 9f2b6e1d4a73, which every branch in the stack shares, and dropped the separate revision. No branch gains a head. - Anyone whose database already ran 9f2b6e1d4a73 will not have the index; recreate the database or add the index by hand. Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com> Co-authored-by: Mohamed Belhsan Hmida <149331360+BelhsanHmida@users.noreply.github.com>
…to HEAD Signed-off-by: F.N. Claessen <felix@seita.nl>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-ups on the automations UI (stacked on #2297):
GET /assets/<id>/automationsnow includes per-automationjob_stats, computed in a single pass over the relevant job caches (deduped cache refs across all the asset's automations, jobs grouped by their trigger's automation id). The UI fills the Jobs column from the list response instead of firing one detail request per automation on page load, and the Details modal loads its contents lazily when first opened. This resolves the O(automations × sensors × jobs) Redis fanout flagged in the Automations - first roundtrip for forecasts #2290 review.redis_connection_err, like the jobs endpoint.How to test
Open
/assets/<id>/automationsas an account admin: the Jobs column fills immediately from one request; Edit lets you rename/re-schedule.Tests:
pytest flexmeasures/api/v3_0/tests/test_automations_api.py flexmeasures/cli/tests/test_automations.py flexmeasures/ui/tests/test_asset_crud.py🤖 Generated with Claude Code
https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX