Skip to content

Event-loop storm on large installs: single-coordinator fan-out to >10k entities trips HA 2026.7's new event-queue guard #670

Description

@tykeal

Summary

On a large parent/child deployment, Home Assistant Core repeatedly raises its endless-loop guard roughly every 15–30 seconds:

homeassistant.exceptions.HomeAssistantError: Event state_reported not fired: more than 10000 events were queued by event listeners while dispatching a single event; event listeners are likely firing events in an endless loop

The traceback surfaces in custom_components/keymaster/switch.py at _handle_coordinator_updateasync_write_ha_state() (lines 209 and 278), but that switch is the victim, not the cause — it is simply the entity that happens to be writing state when the queued-event count crosses 10,000. The real cause is a self-perpetuating parent→child sync + coordinator fan-out loop.

Environment

  • keymaster: current main
  • Locks: 1 Akuvox X916 parent; 13 children (2 Akuvox incl. an A08S, 11 Schlage WiFi). 70 code slots per device (~980 keymaster entities total). Single parent → 13-child fan-out.
  • Companion: local-akuvox v1.0.0+ / pylocal-akuvox 1.x, and 10 Rental Control integrations each driving 4 slots on the X916 parent.
  • Most children inherit only parent slots 1–10, their 4 RC-managed slots, and one backup slot; remaining slots are set to override parent (empty).
  • Symptom appeared when the Akuvox stack moved to pylocal-akuvox 1.x (see "Why it appeared" below).

Root cause

Every coordinator refresh runs parent→child sync for all locks:

  • _async_update_data calls _sync_child_locks for every lock on each pass — coordinator.py:1942-1943.
  • _update_child_code_slots copies parent slot attributes onto each inherited (non-overridden) child slot unconditionally (coordinator.py:2400-2420), then computes pin_mismatch (coordinator.py:2429-2431).
  • On any pin_mismatch or Synced.OUT_OF_SYNC, it calls set_pin_on_lock / clear_pin_from_lock and sets self._quick_refresh = Truecoordinator.py:2463-2483 (_quick_refresh at :2469).
  • set_pin_on_lock calls self.async_set_updated_data(dict(self.kmlocks)) twice per invocation (coordinator.py:1574 and :1598, plus the failure path at :1587); clear_pin_from_lock does the same (coordinator.py:1656, :1679).
  • Each async_set_updated_data synchronously notifies all ~980 coordinator entities, every one of which writes HA state in its _handle_coordinator_update (e.g. switch.py:188-209/:276-278, text.py, sensor.py, binary_sensor.py, number.py, datetime.py, time.py). HA emits state_reported even when nothing changed.

With many persistently-mismatched inherited slots across 13 children, a single sync pass produces dozens of async_set_updated_data calls × ~980 entities → well over 10,000 queued events in one dispatch → HA's loop guard fires.

It also self-perpetuates: because _quick_refresh = True is set on every mismatch, _schedule_quick_refresh_if_needed reschedules a full refresh QUICK_REFRESH_SECONDS later (15s — const.py:28). If any child slot never reaches Synced.SYNCED, the mismatch persists, so the loop re-fires every ~15–30s indefinitely — matching the observed cadence.

Secondary amplifier

Rental Control slot writes to the parent trigger the same parent→child fan-out on top of the periodic quick-refresh churn, compounding the event volume.

Why it appeared with pylocal-akuvox 1.x

Before 1.x, the X916 parent's user-code APIs were not supported, so parent slots carried no real PIN data and the parent→child PIN-sync path was effectively a no-op. pylocal-akuvox 1.x marks the X916's user APIs as supported, so keymaster now reads real PIN data on the parent and attempts to push it down to all 13 children on every refresh. Any child that does not read back an exact match stays perpetually OUT_OF_SYNC/mismatched, which drives the endless re-push + quick-refresh loop. This is a keymaster fan-out/re-entrancy problem that the Akuvox capability change merely exposed — not an Akuvox integration bug.

Impact / risk

  • The integration currently still functions, but Core logs this error every ~15–30s (log flooding).
  • HA drops the guarded event dispatch, so state updates can be missed/delayed, with elevated CPU from ~980-entity fan-outs firing several times per pass.
  • Risk of cascading instability as slot/device counts grow, and potential to trip HA watchdogs under load.

Suggested fix direction (not prescriptive)

  • Add change-detection guards before async_set_updated_data and before entity state writes / _set_property_value — skip notifying when nothing actually changed.
  • Coalesce the parent→child sync pass into a single coordinator notification at the end, instead of 2× per changed slot × N slots.
  • Sync only inherited child slots whose relevant attributes actually changed; avoid re-issuing set_pin_on_lock/clear_pin_from_lock for slots already in the desired state.
  • Investigate why affected child slots never reach Synced.SYNCED (perpetual pin_mismatch) — a provider read-back that can never equal the written value would keep the loop alive regardless of the above.
  • Consider debouncing/skipping no-op quick refreshes when the prior pass produced no real change.

To confirm on the reporting system

Debug logs for a few affected slots around the storm, specifically the [_sync_child_locks] ... mismatch=True lines showing parent vs child PIN values, to verify whether a specific provider read-back is causing the perpetual mismatch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions