Skip to content

ci(e2e): run the unlicensed e2e tests in every E2E job - #6017

Merged
adrians5j merged 1 commit into
nextfrom
ci/e2e-login-test-everywhere
Oct 15, 2026
Merged

adrians5j merged 1 commit into
nextfrom
ci/e2e-login-test-everywhere

Conversation

@adrians5j

@adrians5j adrians5j commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Only the WCP jobs ran anything in e2e/ after the installation test, and those jobs only run when someone in AI_E2E_USERS posts /e2e. For everyone else, and on every push to next, the login test never ran. Each job proved that a fresh project installs, and nothing more.

Every other E2E job now runs one more step after the installation test. That covers the AWS DDB and DDB+OS jobs and the standalone SQLite and Postgres jobs, both in /e2e and on push:

yarn e2e --exclude-tag wcp,ai,installation --pass-with-no-tests
  • These are the tests that need neither a license nor a model. Today that's login.e2e.ts, which signs in as the admin the installation test created and checks the dashboard loads.
  • ai is excluded along with wcp. These jobs have no ANTHROPIC_API_KEY, so a test tagged ai but not wcp would fail with MODEL_UNAVAILABLE.
  • --pass-with-no-tests keeps the step green if the filter ever selects nothing, instead of failing with NO_TESTS.
  • On failure the step uploads e2e/.e2e/ (report, screenshots, traces) as e2e-test-results-<variant>.
  • The WCP jobs are unchanged. Their yarn e2e --exclude-tag installation run already includes these tests.

The step lives in .github/workflows/wac/e2e/unlicensedE2e.ts, next to the installation test step. e2e/README.md describes the new CI order and notes that a test without wcp or ai now runs in every job.

Tested: against a freshly built standalone project on an empty SQLite database, the installation test passed, then this exact command ran the login test and it passed. The AWS jobs haven't run it yet; /e2e on this PR covers them.

Workspace: wby-next5 · 🧪 port admin install e2e

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Expanded end-to-end test coverage across AWS-backed and standalone database setups, including SQLite and Postgres.
    • Failed test runs now retain separate results for each database setup for up to seven days.
  • Documentation
    • Updated the E2E testing guide to describe the test coverage and which tests run across CI jobs.

Only the WCP jobs ran anything in e2e/ past the installation test, and
those only run for the people in AI_E2E_USERS. So the login test ran
for nobody else, on /e2e or on push.

Every other E2E job, AWS and standalone, /e2e and push, now runs
`yarn e2e --exclude-tag wcp,ai,installation --pass-with-no-tests`
after the installation test, and uploads e2e/.e2e/ if it fails. Today
that is the login test. The WCP jobs keep their own run, which already
covers these tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🚓 Slop Cop

✅ Nothing worth flagging. The diff looks consistent with the PR's stated intent and the code-style rules.

The PR coherently adds an unlicensed-e2e-test step to all non-WCP e2e jobs, matching its stated intent, with no integrity or style issues found.

Automated, non-blocking heads-up from an LLM. It can be wrong — use your judgment. Regenerates on every push.

@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: dfc8e0ef-99fb-4970-b3a2-ee568b49ee9a
📥 Commits

Reviewing files that changed from the base of the PR and between d2a84df and cdce0dc.

📒 Files selected for processing: 8
  • .github/workflows/pullRequestsCommandE2e.yml
  • .github/workflows/push.yml
  • .github/workflows/wac/e2e/createAwsJobs.ts
  • .github/workflows/wac/e2e/createStandaloneJobs.ts
  • .github/workflows/wac/e2e/index.ts
  • .github/workflows/wac/e2e/unlicensedE2e.ts
  • .github/workflows/wac/push.wac.ts
  • e2e/README.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

📝 Walkthrough

CI workflows now run a filtered E2E suite for AWS-backed and standalone database configurations. Failed runs upload database-specific result artifacts. The workflow documentation describes the updated test coverage.

Changes: Non-installation E2E coverage

Layer / File(s) Summary
Define and export unlicensed E2E steps
.github/workflows/wac/e2e/unlicensedE2e.ts, .github/workflows/wac/e2e/index.ts
Adds and exports a step factory that runs E2E tests excluding the wcp, ai, and installation tags. It uploads results on failure and ignores missing files.
Add steps to generated E2E jobs
.github/workflows/wac/e2e/createAwsJobs.ts, .github/workflows/wac/e2e/createStandaloneJobs.ts, .github/workflows/wac/push.wac.ts
AWS-backed and non-AI standalone jobs pass the working directory, admin URL, and storage-specific artifact name to the shared step factory. AI standalone jobs remain unchanged.
Add steps to explicit CI jobs
.github/workflows/pullRequestsCommandE2e.yml, .github/workflows/push.yml, e2e/README.md
Pull request and push jobs add filtered E2E runs and failure-only result uploads for DDB, DDB+OpenSearch, SQLite, and Postgres. The CI guidance describes this coverage.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Merge Risk: ⚪ Minimal · up to cdce0

The added CI test step uses supported runner options and has no identified merge blocker.

Pre-merge checks: Passed 5
Passed Passed checks: 5 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: running unlicensed E2E tests in every E2E job.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 5…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@adrians5j
adrians5j merged commit 6f58abe into next Oct 15, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant