Unhandled ObjectDisposedException on host shutdown when a dream cycle is scheduled or in flight

[rockfordlhotka]

Summary

When the host shuts down while DreamService has a cycle scheduled or in flight, an
ObjectDisposedException escapes as an unhandled exception rather than being caught and
logged. It happens on a normal SIGTERM (docker compose restart / up -d --force-recreate),
not just on abnormal termination.

Observed on roughly eight consecutive container restarts today, i.e. every restart where a
dream cycle was scheduled.

Unhandled exception. System.ObjectDisposedException: The CancellationTokenSource has been disposed.

On one occasion it was accompanied by:

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, ...)

Impact is limited — the process is exiting anyway — but it makes clean shutdown indistinguishable
from a crash in logs and monitoring, and it obscures any real shutdown fault that occurs alongside it.

Likely mechanism

Two ordering hazards, both plausible sources. I have not isolated which one fires (or whether both do).

1. Slot acquisition happens outside DreamService.DreamAsync's try.

AgentWorkSerializer.Dispose() disposes _preemptCts. TryAcquireForScheduledAsync then reads
_preemptCts.Token, which throws ObjectDisposedException once disposal has happened:

// AgentWorkSerializer.cs
lock (_preemptLock)
{
    preemptToken = _preemptCts.Token;   // throws if _preemptCts already disposed
}

DreamAsync calls this before its try block opens, so the exception is not covered by the
existing catch (OperationCanceledException) / catch (Exception) handlers:

// DreamService.cs — slot acquired outside the try
var slot = await _workSerializer.TryAcquireForScheduledAsync(CancellationToken.None);
if (slot is null) { ... return; }

LoadDirectives(initialLoad: false);
_logger.LogInformation("DreamService: dream cycle starting");

try   // <- everything below is protected; the acquisition above is not
{
    ...
}
catch (Exception ex) { _logger.LogError(ex, "DreamService: dream cycle failed"); }
finally { await slot.DisposeAsync(); }

2. ArmNextCronTimer can touch an already-disposed Timer.

OnTimerTickAsync calls ArmNextCronTimer() after DreamAsync() returns, which calls
_timer?.Change(...). If DreamService.Dispose() has already run, Change throws on the disposed
timer.

Why shutdown does not prevent either. StopAsync only disarms the timer; it does not wait for a
callback that is already running:

public Task StopAsync(CancellationToken cancellationToken)
{
    _timer?.Change(Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan);
    return Task.CompletedTask;
}

public void Dispose() => _timer?.Dispose();

So an in-flight cycle can outlive StopAsync and reach for objects the DI container has since
disposed. The timer callback is also fire-and-forget, so nothing observes the resulting task:

_timer = new Timer(
    state => { _ = OnTimerTickAsync(); },   // task discarded
    null,
    _options.InitialDelay,
    Timeout.InfiniteTimeSpan);

Reproduction

1. Configure a dream schedule so a cycle is due soon — e.g. Dream__CronSchedule=*/5 * * * *
   with Dream__InitialDelay=00:01:00.
2. Start the agent and let it register the timer.
3. docker compose restart agent (or up -d --force-recreate agent) around the time a cycle
   is due, or while one is running.
4. The unhandled ObjectDisposedException appears in the container log during teardown.

A short InitialDelay plus a frequent cron makes this hit essentially every restart.

Suggested fixes

Any of these would remove the unhandled exception; the first two look like the substantive ones.

• Make shutdown wait for an in-flight cycle. Hold a CancellationTokenSource for the service,
  cancel it in StopAsync, and await the running cycle (or use Timer.DisposeAsync(), which
  completes after callbacks finish) so the cycle cannot outlive the objects it depends on.
• Bring slot acquisition inside the guarded region in DreamAsync, so a disposed serializer
  is logged as a failed cycle rather than escaping. A disposed AgentWorkSerializer could also
  return null from TryAcquireForScheduledAsync — semantically "no slot available", which
  callers already handle — instead of throwing.
• Observe the timer task rather than discarding it, so anything thrown from OnTimerTickAsync
  is logged instead of becoming an unobserved/unhandled exception.
• Guard ArmNextCronTimer against a disposed timer (catch ObjectDisposedException, or check
  a _stopping flag set by StopAsync).

Environment

• .NET 10, RockBot.Host.DreamService / RockBot.Host.AgentWorkSerializer
• Reproduced in the deploy/docker-compose stack

[rockfordlhotka]

Still reproducing a month on, and not specific to docker compose — it happens on the Kubernetes deployment too.

Noticed while deploying agent 0.14.35 (PR #551) on 2026-09-03. Loki over the last 14 days:

{namespace="rockbot"} |= "ObjectDisposedException"

20 occurrences across 20 distinct agent pods, spanning 2026-08-28 → 2026-09-03 — i.e. effectively one per agent pod shutdown, matching the "every restart" observation in the original report. Every line is the same:

Unhandled exception. System.ObjectDisposedException: The CancellationTokenSource has been disposed.

Two from this deploy specifically, which bracket it nicely:

Time (UTC)	Pod	Build
2026-09-03 19:18:14	rockbot-agent-6c95c948b6-rwmmj	0.14.34, shutting down for the upgrade
2026-09-03 19:25:07	rockbot-agent-8444c5f996-g5qd4	0.14.35, shutting down on a later rollout restart

So it is unaffected by anything in #551, and reproduces on both builds.

One data point against the "only when a cycle is scheduled in the initial-delay window" reading: the 8444c5f996 pod had been up ~7 minutes and had already completed a full dream cycle (dream cycle complete — 27 deleted, 8 saved) about 90 seconds before shutdown, while 6c95c948b6 had been up ~14 hours. Both still threw. That is consistent with hazard 1 in the original analysis (slot acquisition outside the try, or more generally a disposed _preemptCts being read during shutdown) rather than with an in-flight cycle being the trigger — though I did not isolate it.

Practical cost is as described: clean shutdown is indistinguishable from a crash in Loki, which is exactly what made me stop and spend time ruling it out as a regression mid-deploy.

🤖 Generated with Claude Code

https://claude.ai/code/session_018GsCZ7jPi8GLu14CoDovvP