CFE-3041: Made testall fail regular acceptance tests which log errors by aleksandrychev · Pull Request #6405 · cfengine/core

aleksandrychev · 2026-10-14T12:25:35Z

Tests that print Pass message but have logs errors will fail with FAIL (Error printed) message. Tests which are expected to log errors should use .error.cf or .error.pass.cf suffix. see: tests/acceptance/README

together:

CFE-3041: Fixed acceptance tests logging errors when reading files too early masterfiles#3225

Ticket: CFE-3041

aleksandrychev · 2026-10-14T12:25:57Z

@cf-bottom jenkins please

cf-bottom · 2026-10-14T12:31:42Z

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14799/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14799/

olehermanse

This cannot be done on its own - there are hundreds of tests that need to be reviewed and adjusted after this change in testall.

aleksandrychev · 2026-10-14T18:43:22Z

This cannot be done on its own - there are hundreds of tests that need to be reviewed and adjusted after this change in testall.

I see, there are quite a lot of them. So, we need to identify which ones are expected and should be renamed, and also find the ones that shouldn't be there. As a first step, I'll create a table listing each failure and the errors printed.

aleksandrychev · 2026-10-15T11:17:58Z

at least half of errors can be fixed easily with a fileexists guard.
for example 01_vars/02_functions/011.cf

error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'
error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)

the error comes from the test bundle:

vars:
    "nums" slist => readstringlist("$(G.testfile)", "NoComment", "\s+", 5, 100);

vars are evaluated early (during policy validation and pre-evaluation), before init has created the file, so the read function fails and logs an error. A later pass reads the file fine, so the test still passes.

it can be fixed by adding a fileexists guard.

nickanderson

@aleksandrychev The commit for Added fileexists() guards to file reads in acceptance tests is good. I think we could take it separately. Will continue looking into the other commits.

Looked through the rest and they all seem ok. Logged CFE-4752 which i found while reviewing.

nickanderson · 2026-10-21T16:39:11Z

    "lefti" int => "$(init.pair[1])";
    "rights" string => "$(init.pair[2])";
-    "rightr" real => "$(init.pair[2])";
+    "rightr" real => "$(init.pair[2])",


This one is a bit odd. Maybe the question is why does only real vars log an error fail to parse real number when it encounters an empty string. Seems that we should align the int and real logging behavior here. I created CFE-4752 for that.

nickanderson · 2026-10-21T16:59:43Z

+bundle common framework
+{
+  classes:
+    "framework_loaded" expression => isvariable("G.testdir");


Strange. I guess this can affect all cases where a body is used (on classes or vars promises) in the entry file or in some bundle before reaching the body in pre-eval.

nickanderson · 2026-10-21T19:18:34Z

+bundle common framework
+{
+  classes:
+    "framework_loaded" expression => isvariable("G.testdir");


Strange. I guess this can affect all cases where a body is used (on classes or vars promises) in the entry file or in some bundle before reaching the body in pre-eval.

aleksandrychev · 2026-10-22T13:20:10Z

@olehermanse your requested change blocks PR

olehermanse · 2026-10-22T13:22:11Z

@cf-bottom jenkins please

cf-bottom · 2026-10-22T13:23:22Z

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14855/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14855/

olehermanse · 2026-10-22T14:44:53Z

@aleksandrychev please look at why Ubuntu failed.

aleksandrychev · 2026-10-22T18:22:58Z

@aleksandrychev please look at why Ubuntu failed.

I will look, maybe this is environment issue or platform-specific, as it passes in GH runner (ubuntu 24) and also passed in docker container (same as GH runner uses)

nickanderson · 2026-10-22T21:45:46Z

@cf-bottom jenkins please

cf-bottom · 2026-10-22T21:48:20Z

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14858/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14858/

aleksandrychev · 2026-10-23T07:53:46Z

@nickanderson thanks for help here, now only diff_reporting_large_file.cf fails, looking at it

aleksandrychev · 2026-10-23T08:53:17Z

@cf-bottom jenkins please

cf-bottom · 2026-10-23T09:01:05Z

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14860/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14860/

aleksandrychev · 2026-10-23T10:56:27Z

@cf-bottom jenkins please

cf-bottom · 2026-10-23T10:58:23Z

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14861/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14861/

aleksandrychev · 2026-10-23T12:07:10Z

https://ci.cfengine.com/job/testing-pr/4299/label=PACKAGES_HUB_x86_64_linux_ubuntu_22/ greeen now, nice

aleksandrychev · 2026-10-26T09:23:55Z

win acceptances -

aleksandrychev · 2026-10-27T08:24:39Z

@cf-bottom jenkins please

cf-bottom · 2026-10-27T08:28:40Z

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14878/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14878/

nickanderson · 2026-10-28T02:50:02Z

@cf-bottom jenkins, please

cf-bottom · 2026-10-28T09:58:32Z

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14897/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14897/

olehermanse · 2026-10-28T17:15:30Z

@cf-bottom jenkins, please

cf-bottom · 2026-10-28T17:22:59Z

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14921/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14921/

Tests that print Pass message but have logs errors will fail with `FAIL (Error printed)` message. Tests which are expected to log errors should use `.error.cf` or `.error.pass.cf` suffix. see: `tests/acceptance/README` Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

These tests read files created by their own init bundle, but pre-evaluation calls the read functions before the files exist, logging "Could not examine file" errors which now fail regular tests. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Errors printed by the command were reported on lines starting with "error:", so testall mistook them for errors of the test itself. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

These tests read files and directories created by their own policy, but pre-evaluation calls the functions before they exist, logging errors which now fail regular tests. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The agent loads def.json from the directory of the input policy file, so tests in 00_basics/ tried to read this directory as their augments file and logged "Could not open JSON file" errors. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The test used the 'immediate' body from plucked.sub.cf before it was loaded during pre-evaluation, so the promise is now skipped until the test framework is loaded. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The test referred to a promise module which never existed and did not run its init and cleanup bundles. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The check bundle counted lines in files which do not exist until the test bundle has run, but the agent evaluates the functions of all promises before any bundle runs. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

wrong-bundle-type.cf printed the sub-agent output in its DEBUG report, so the sub-agent errors were mistaken for errors of the test, and ifelse.cf used invalid class expressions as false conditions. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

These tests intentionally exercise error paths, the reason is documented in the header of each test. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Ticket: CFE-3041 Signed-off-by: Nick Anderson <nick@cmdln.org>

Ticket: CFE-3041

The errorpass branch only checked for logged errors and the Pass message, so a test marked test_soft_fail or test_skip_unsupported was reported as FAIL (e.g. process_select_fncalls.error.pass.cf on Windows, which soft fails due to ENT-10217 and logs no errors there). Errorpass tests now go through the regular outcome handling and only differ in requiring at least one logged error on Pass.

Ticket: CFE-3041 Signed-off-by: Nick Anderson <nick@cmdln.org>

aleksandrychev · 2026-10-29T08:39:40Z

@cf-bottom jenkins please

cf-bottom · 2026-10-29T08:43:27Z

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14923/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14923/

aleksandrychev · 2026-10-29T09:39:12Z

@cf-bottom jenkins please

cf-bottom · 2026-10-29T09:44:02Z

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14925/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14925/

aleksandrychev requested review from craigcomstock and olehermanse October 14, 2026 12:25

olehermanse requested changes Oct 14, 2026

View reviewed changes

aleksandrychev marked this pull request as draft October 14, 2026 18:43

aleksandrychev marked this pull request as ready for review October 15, 2026 13:56

aleksandrychev requested a review from olehermanse October 15, 2026 13:56

olehermanse requested a review from nickanderson October 19, 2026 21:59

nickanderson reviewed Oct 21, 2026

View reviewed changes

nickanderson approved these changes Oct 21, 2026

View reviewed changes

craigcomstock approved these changes Oct 21, 2026

View reviewed changes

olehermanse approved these changes Oct 22, 2026

View reviewed changes

aleksandrychev marked this pull request as draft October 22, 2026 18:15

github-advanced-security Bot found potential problems Oct 23, 2026

View reviewed changes

Comment thread libpromises/evalfunction.c Fixed

aleksandrychev marked this pull request as ready for review October 26, 2026 11:57

nickanderson force-pushed the CFE-3041 branch from cdd5cac to a868166 Compare October 28, 2026 02:43

aleksandrychev and others added 16 commits October 28, 2026 12:24

Indented command output in dcs_passif_output() DEBUG report

5398b14

Errors printed by the command were reported on lines starting with "error:", so testall mistook them for errors of the test itself. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Fixed classic_evaluation_custom_promise_types.cf logging errors

f5197a4

The test referred to a promise module which never existed and did not run its init and cleanup bundles. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Renamed acceptance tests which log errors on purpose to .error.pass.cf

350760c

These tests intentionally exercise error paths, the reason is documented in the header of each test. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Fixed linting issues

22cab40

Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Fixed unsafe and check_syntax.cf acceptance tests logging errors

14691f5

Ticket: CFE-3041 Signed-off-by: Nick Anderson <nick@cmdln.org>

Fixed formatting of acceptance tests

5c6c043

Ticket: CFE-3041 Signed-off-by: Nick Anderson <nick@cmdln.org>

Fixed mapdata_json_pipe.cf logging errors

3e5e690

Ticket: CFE-3041

Fixed diff_mode_ownership.cf logging errors

4cb9088

Ticket: CFE-3041 Signed-off-by: Nick Anderson <nick@cmdln.org>

nickanderson force-pushed the CFE-3041 branch from a868166 to 4cb9088 Compare October 28, 2026 17:24

Conversation

aleksandrychev commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aleksandrychev commented Oct 14, 2026

Uh oh!

cf-bottom commented Oct 14, 2026

Uh oh!

olehermanse left a comment

Choose a reason for hiding this comment

Uh oh!

aleksandrychev commented Oct 14, 2026

Uh oh!

aleksandrychev commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickanderson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nickanderson Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

nickanderson Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

nickanderson Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

aleksandrychev commented Oct 22, 2026

Uh oh!

olehermanse commented Oct 22, 2026

Uh oh!

cf-bottom commented Oct 22, 2026

Uh oh!

olehermanse commented Oct 22, 2026

Uh oh!

aleksandrychev commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickanderson commented Oct 22, 2026

Uh oh!

cf-bottom commented Oct 22, 2026

Uh oh!

aleksandrychev commented Oct 23, 2026

Uh oh!

aleksandrychev commented Oct 23, 2026

Uh oh!

cf-bottom commented Oct 23, 2026

Uh oh!

aleksandrychev commented Oct 23, 2026

Uh oh!

cf-bottom commented Oct 23, 2026

Uh oh!

aleksandrychev commented Oct 23, 2026

Uh oh!

Uh oh!

aleksandrychev commented Oct 26, 2026

Uh oh!

aleksandrychev commented Oct 27, 2026

Uh oh!

cf-bottom commented Oct 27, 2026

Uh oh!

nickanderson commented Oct 28, 2026

Uh oh!

cf-bottom commented Oct 28, 2026

Uh oh!

olehermanse commented Oct 28, 2026

Uh oh!

cf-bottom commented Oct 28, 2026

Uh oh!

aleksandrychev commented Oct 29, 2026

Uh oh!

cf-bottom commented Oct 29, 2026

Uh oh!

aleksandrychev commented Oct 29, 2026

Uh oh!

cf-bottom commented Oct 29, 2026

Uh oh!

Reviewers

aleksandrychev commented Oct 14, 2026 •

edited

Loading

aleksandrychev commented Oct 15, 2026 •

edited

Loading

nickanderson left a comment •

edited

Loading

aleksandrychev commented Oct 22, 2026 •

edited

Loading