Skip to content

fix(e2e): wait for fresh runners endpoint#61

Open
casey-brooks wants to merge 1 commit into
mainfrom
noa/issue-58-e2e-fix
Open

fix(e2e): wait for fresh runners endpoint#61
casey-brooks wants to merge 1 commit into
mainfrom
noa/issue-58-e2e-fix

Conversation

@casey-brooks

Copy link
Copy Markdown
Contributor

Summary

  • Fix runners source-deploy readiness so E2E waits for a fresh source pod ready marker instead of stale logs.
  • Wait for synced migration files before go run, preventing the observed missing github.com/agynio/runners/migrations package crash.
  • Wait for ready service endpoints before starting E2E and add failure diagnostics for pods, service/endpoints, logs, describes, and /tmp/runners-devspace.log.

Follow-up to #59 after it was merged.

Test & lint summary

  • buf generate buf.build/agynio/api --include-imports --path agynio/api/agents/v1 --path agynio/api/runner/v1 --path agynio/api/runners/v1 --path agynio/api/identity/v1 --path agynio/api/authorization/v1 --path agynio/api/ziti_management/v1 --path agynio/api/notifications/v1: completed successfully
  • go test ./...: passed 1 / failed 0 / skipped 0
  • go vet ./...: passed with no errors
  • devspace run-pipeline --help: passed config load and listed diagnostics-runners

@casey-brooks

Copy link
Copy Markdown
Contributor Author

Root cause and fix details

I inspected E2E run 26501969016 artifacts/logs. /tmp/runners-devspace.log was not included in the downloaded artifact, but the uploaded suite diagnostics contained enough runner pod logs to identify the cause.

Concrete cause: wait-runners matched a stale runners: ready log from the pre-existing deployed image and returned immediately. E2E then started while the DevSpace source pod was not serving. The source pod later crashed before listening with:

internal/db/migrate.go:9:2: no required module provides package github.com/agynio/runners/migrations

That happened because the source-sync wait only checked for go.mod, buf files, and cmd/runners/main.go; go run could start before the embedded migrations package was synced.

This PR fixes that deterministically by requiring a fresh per-run ready marker, waiting for migration files before go run, and waiting for ready service endpoints before E2E starts. It also adds failure diagnostics for pods, service/endpoints, endpoint slices, current/previous logs, pod describes, and /tmp/runners-devspace.log.

Test & lint summary

  • buf generate buf.build/agynio/api --include-imports --path agynio/api/agents/v1 --path agynio/api/runner/v1 --path agynio/api/runners/v1 --path agynio/api/identity/v1 --path agynio/api/authorization/v1 --path agynio/api/ziti_management/v1 --path agynio/api/notifications/v1: completed successfully
  • go test ./...: passed 1 / failed 0 / skipped 0
  • go vet ./...: passed with no errors
  • devspace run-pipeline --help: passed config load and listed diagnostics-runners

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant