Pre-warm pool can silently die: stuck instances inflate `unallocated_instances_count` and block refill

[pmelab]

Summary

The pre-warm pool refill mechanism can silently stop working when unallocated instances get stuck in non-terminal intermediate states (e.g. post-create-running, deploy-running). They count toward unallocated_instances_count, so the pool reports as "full" even though no instance is actually claimable.

We observed this in the dev environment for store_app 5ebb8540-fbaa-47ff-bba5-b34e16dd475b (amazeeClaw - DEV (us3)):

target_unallocated_app_instances = 5
unallocated_instances_count       = 5
status breakdown of those 5: post-create-running × 5

All 5 instances have been stuck at post-create-running since 2026-04-10 / 2026-04-11 — ~41 days. Every claim against this store_app during that window has had to fall through to the full create path (~5-6 min) instead of using a pre-warmed instance (~30s claim). On MOAD this manifests as the AmazeeClaw E2E test repeatedly timing out, because MOAD's provisioning UI has a 5-minute timeout.

Root cause

Two pieces of logic disagree on what "unallocated" means:

getUnallocatedInstancesCountAttribute in app/Models/PolydockStoreApp.php:252-257 counts every instance with user_group_id IS NULL, regardless of status:

public function getUnallocatedInstancesCountAttribute(): int
{
    return $this->instances()
        ->whereNull('user_group_id')
        ->count();
}

needs_more_unallocated_instances (line 264) compares that count to the target and gates EnsureUnallocatedAppInstancesJob, so as long as the count is at or above target, no refill is dispatched.

The claim query in app/Models/UserGroup.php:145-150 only picks instances at status RUNNING_HEALTHY_UNCLAIMED:

PolydockAppInstance::where('polydock_store_app_id', $storeApp->id)
    ->whereNull('user_group_id')
    ->whereNull('allocation_lock')
    ->where('status', PolydockAppInstanceStatus::RUNNING_HEALTHY_UNCLAIMED)
    ->limit(1)
    ->update(['allocation_lock' => $allocationLock, 'user_group_id' => $userGroup->id]);

Result: instances stuck at post-create-running, pending-deploy, deploy-running, post-create-failed, etc. inflate the count but contribute zero capacity. The "stale refresh" path (removableUnallocatedInstancesQuery, line 305) only matches RUNNING_HEALTHY_UNCLAIMED, so stuck instances are never refreshed either. Nothing in polydock-engine times out an instance that stops progressing inside the create pipeline.

Immediate workaround (per dev env, run today)

1. Mark the stuck instances as failed so they stop inflating the count:

   UPDATE polydock_app_instances
   SET status = 'post-create-failed',
       status_message = 'Manually marked failed — stuck post-create-running since April 2026'
   WHERE id IN (394, 396, 398, 400, 402);

2. Wait ≤5s for the supervisord poll worker (polydock:poll-unallocated-instances) to dispatch EnsureUnallocatedAppInstancesJob, or trigger it manually:

   php artisan polydock:dispatch-ensure-unallocated-app-instances-job
   

post-create-failed is excluded from unallocated_instances_count if the count is changed to status-aware (see fix options below). Until the model changes, marking them failed at least makes the situation visible in admin.

Suggested fixes (one or both)

A. Make the count status-aware. getUnallocatedInstancesCountAttribute should only count instances that are either currently claimable or actively progressing toward claimable. A simple version: exclude any *-failed status, and exclude any non-*-failed instance that hasn't updated in N minutes (configurable, e.g. 30). That alone restores self-healing — once a stuck instance ages out, the count drops below target, refill kicks in.

B. Add a stuck-detector job. A periodic command that finds instances at non-terminal intermediate statuses with updated_at older than a threshold, transitions them to the corresponding *-failed status (or a new *-stuck status), and logs/alerts. Wire it into the supervisord loop or routes/console.php.

Either approach would have surfaced this within an hour rather than 41 days.

Verification

After applying the workaround, run:

SELECT status, COUNT(*) FROM polydock_app_instances
WHERE polydock_store_app_id = 12 AND user_group_id IS NULL
GROUP BY status;

You should see new instances cycling through pre-create → … → running-healthy-unclaimed within ~6 minutes per instance, with the pool eventually settling at 5 ready instances.

Impact / context

• Customer-facing path (MOAD AmazeeClaw subscription flow) silently degraded for >40 days on dev.
• Production stores have higher target_unallocated_app_instances and more traffic, so the same failure mode would be more disruptive there — recommend checking prod store_apps for the same condition (SELECT status, COUNT(*) … WHERE user_group_id IS NULL GROUP BY polydock_store_app_id, status looking for non-running-healthy-unclaimed rows in the pool).

[dan2k3k4]

There's a "refresh warm environment every X days" so I've enabled that for the app instances that we're testing.

I've also created this PR which might help:
#145