Repository navigation
LTS: report the failed bstackStep as the test failure; skip setSessionStatus executor on LTS - #66
Merged
shubhamkd merged 1 commit intoOct 23, 2026
Conversation
…nStatus executor on LTS LCNC specs throw their own Error from the step helper, which Nightwatch keeps out of the command list. The first failed command was then an internal selector probe (executeAsyncScript "script timeout") whose result has no message or stack, so TestRunFinished shipped failure.backtrace ['', ''] and TestHub could not cluster unique errors. When no command failed at all, the test was reported as passed. On LTS runs the last failed step in global.__bstack_steps is now the authoritative failure. The setSessionStatus browserstack_executor call is skipped on LTS runs: the LTS grid does not intercept it, Chrome evaluates it as JS and it fails with "Unexpected token ':'" after every test.
sanath-browserstack
marked this pull request as ready for review
October 22, 2026 10:33
MihirR-BS
approved these changes
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two LTS-only fixes to the Nightwatch main path, both gated on
helper.isLoadTestingSession()so non-LTS Automate customers see byte-identical uploads and unchanged session marking.1. Report the failed
bstackStep()as the test failure (src/testObservability.js,src/utils/helper.js)LCNC specs throw their own
Errorfrom the step helper ([Step N: …] No selector matched after 30000ms. …). Nightwatch keeps that error out oftestcase.commands, sosendTestRunEventpicked the first failed command instead — an internal selector probe (executeAsyncScriptthat hitscript timeout) whose result is{status: -1, value: null}. The upload then carried:TestHub had no error message or stack to cluster on, so unique errors were empty for the run, while the real message sat in
meta.steps. Seen in production on a 100-VU LCNC run (plugin 3.11.4 from the session capabilities): 4 failed tests, all with an empty failure. When the probes fail cleanly (no failed command at all), the test was reported as passed although its step had failed.Fix: on LTS runs, the last step in
global.__bstack_stepswithresult: 'failed'and a non-emptyfailurebecomes the test failure (result,failure,failure_reason,failure_type). The buffer is reset on everyTestRunStarted(#60), so the step always belongs to the current test. New helpergetLastFailedLtsStep(steps). With no failed step, the existing command-based failure is untouched.2. Skip the
setSessionStatusexecutor on LTS runs (nightwatch/globals.js)The LTS grid runs with
browserstackAutomation: falseand does not interceptbrowserstack_executor. Chrome evaluates the string as JavaScript and it fails withjavascript error: Unexpected token ':'(3 retries) after every test, passing or failing. On a failing test that line lands in the run's failure evidence and was being read as a script syntax error. Session status on LTS is owned by the LTS runner, so the call is skipped there.Verification
Unit tests (
npm test)8 new cases, all passing (109 passing vs 101 on
main):getLastFailedLtsStep: missing/non-array buffer, no failed step, last failed step wins over its parent, failed step with an empty/non-string message is skipped.sendTestRunEvent(LTS failed step): timed-out probe with an empty result → failure from the step; no failed command →failed, notpassed; LTS run with no failed step → command-derived failure kept (AssertionError); non-LTS run → step buffer ignored.The one failing test (
getBuildName→ "returns empty for empty options") fails identically onmain— it asserts the checkout directory name.End-to-end (real Nightwatch 3.16 + local chromedriver + Chrome 155, upload intercepted)
LCNC-style spec (steps pushed to
global.__bstack_steps,executeAsyncScriptselector probes, helper throws "No selector matched"):script timeoutfailed, failure['', '']failed, failure_reason = the step's "No selector matched…" messagepassedfailed, same messagepassed, 3×Unexpected token ':'passed, none