Skip to content

LTS: report the failed bstackStep as the test failure; skip setSessionStatus executor on LTS - #66

Merged
shubhamkd merged 1 commit into
nightwatchjs:mainfrom
sanath-browserstack:LTAR-420-lts-step-failure
Oct 23, 2026
Merged

shubhamkd merged 1 commit into
nightwatchjs:mainfrom
sanath-browserstack:LTAR-420-lts-step-failure

Conversation

@sanath-browserstack

Copy link
Copy Markdown
Contributor

Summary

Two LTS-only fixes to the Nightwatch main path, both gated on helper.isLoadTestingSession() so non-LTS Automate customers see byte-identical uploads and unchanged session marking.

1. Report the failed bstackStep() as the test failure (src/testObservability.js, src/utils/helper.js)

LCNC specs throw their own Error from the step helper ([Step N: …] No selector matched after 30000ms. …). Nightwatch keeps that error out of testcase.commands, so sendTestRunEvent picked the first failed command instead — an internal selector probe (executeAsyncScript that hit script timeout) whose result is {status: -1, value: null}. The upload then carried:

result: 'failed', failure: [{backtrace: ['', '']}], failure_reason: null

TestHub had no error message or stack to cluster on, so unique errors were empty for the run, while the real message sat in meta.steps. Seen in production on a 100-VU LCNC run (plugin 3.11.4 from the session capabilities): 4 failed tests, all with an empty failure. When the probes fail cleanly (no failed command at all), the test was reported as passed although its step had failed.

Fix: on LTS runs, the last step in global.__bstack_steps with result: 'failed' and a non-empty failure becomes the test failure (result, failure, failure_reason, failure_type). The buffer is reset on every TestRunStarted (#60), so the step always belongs to the current test. New helper getLastFailedLtsStep(steps). With no failed step, the existing command-based failure is untouched.

2. Skip the setSessionStatus executor on LTS runs (nightwatch/globals.js)

The LTS grid runs with browserstackAutomation: false and does not intercept browserstack_executor. Chrome evaluates the string as JavaScript and it fails with javascript error: Unexpected token ':' (3 retries) after every test, passing or failing. On a failing test that line lands in the run's failure evidence and was being read as a script syntax error. Session status on LTS is owned by the LTS runner, so the call is skipped there.

Verification

Unit tests (npm test)

8 new cases, all passing (109 passing vs 101 on main):

  • getLastFailedLtsStep: missing/non-array buffer, no failed step, last failed step wins over its parent, failed step with an empty/non-string message is skipped.
  • sendTestRunEvent (LTS failed step): timed-out probe with an empty result → failure from the step; no failed command → failed, not passed; LTS run with no failed step → command-derived failure kept (AssertionError); non-LTS run → step buffer ignored.

The one failing test (getBuildName → "returns empty for empty options") fails identically on main — it asserts the checkout directory name.

End-to-end (real Nightwatch 3.16 + local chromedriver + Chrome 155, upload intercepted)

LCNC-style spec (steps pushed to global.__bstack_steps, executeAsyncScript selector probes, helper throws "No selector matched"):

Scenario Before After
Probes hit script timeout failed, failure ['', ''] failed, failure_reason = the step's "No selector matched…" message
Probes fail cleanly passed failed, same message
Passing test passed, 3× Unexpected token ':' passed, none
Non-LTS run — unchanged (executor still sent, same upload)

…nStatus executor on LTS

LCNC specs throw their own Error from the step helper, which Nightwatch keeps
out of the command list. The first failed command was then an internal
selector probe (executeAsyncScript "script timeout") whose result has no
message or stack, so TestRunFinished shipped failure.backtrace ['', ''] and
TestHub could not cluster unique errors. When no command failed at all, the
test was reported as passed. On LTS runs the last failed step in
global.__bstack_steps is now the authoritative failure.

The setSessionStatus browserstack_executor call is skipped on LTS runs: the
LTS grid does not intercept it, Chrome evaluates it as JS and it fails with
"Unexpected token ':'" after every test.
@shubhamkd
shubhamkd merged commit de30531 into nightwatchjs:main Oct 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants