Added support for several different verbosity levels in our integration tests

[bieryAtFnal]

The changes are intended for the fddaq-v5.7.0 release.

Description

In discussions of possible improvements to our regression/integration tests, it was suggested to provide several different levels of verbosity. As described in a presentation at the 31-Mar-2026 SWI&T meeting, we currently have only two options available: one with very limited output and one with the full output. The proposal is to move to a model in which 6 different verbosity levels are supported (where the sixth one enables drunc debug messages).

Suggested instructions for testing the changes are provided below. Please note that the new default verbosity level is 3, so there will be less console output unless the full amount is specifically requested.

In addition to the verbosity-level changes, there are changes included in this PR that are not directly related, but are beneficial. These include:

• a reusable function to perform basic checks on the results of integration tests
  • this common function includes the check on the return code of the run_dunerc fixture, as before, and there is also now a check for warning messages that may have been caught during the setup of the tests. These latter messages were being silently ignored previously.
• support for a user-supplied substring that will trigger the printout of the full drunc output when it is encountered (This is only useful when only part of the drunc output is being displayed, of course)
• test-file-specific changes that will be described in the appropriate repository-based PRs

Here are the command-line options that control the verbosity level and the triggering of the full drunc output:

• for dunedaq_integtest_bundle.sh:
  • --verbosity <level> : requested level of console messages, in range 1-6, where 1 is least, 6 is DRUNC debug
  • --trigger-full-rc-output <phrase that will trigger the full printout of run control messages> : the phrase can be a Python regex, which can be useful in handling colorized text
• for calling pytest directly:
  • --integtest-verbosity <level> - controls the volume of messages that are printed out by the integration test infrastructure
  • --dunerc-fullprint-watch-string <string> - A phrase that, if found in run control messages, will trigger the printout of all RC messages

These changes are correlated with ones in the following repositories:

• Changes to support integration test verbosity levels daqsystemtest#312
• First round of changes to integtests for verbosity levels. dfmodules#485
• Updated the iceberg_real_hsi_test.py to include many of the elements … hsilibs#83
• Updated the listrev_test.py to include the changes that are part of t… listrev#120
• Updated the simple_transform_test to include the changes that are par… snbmodules#30
• Updated the change_rate_test.py to include the changes that are part … trigger#430

Here are suggested steps for testing:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260420_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/integtest_verbosity_level_ideas
git clone https://github.com/DUNE-DAQ/dfmodules.git -b kbiery/integtest_verbosity_level_ideas
git clone https://github.com/DUNE-DAQ/hsilibs.git -b kbiery/integtest_verbosity_level_ideas
git clone https://github.com/DUNE-DAQ/listrev.git -b kbiery/integtest_verbosity_level_ideas
git clone https://github.com/DUNE-DAQ/snbmodules.git -b kbiery/integtest_verbosity_level_ideas
git clone https://github.com/DUNE-DAQ/trigger.git -b kbiery/integtest_verbosity_level_ideas
cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/integtest_verbosity_level_ideas
cd ..

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh --verbosity 2 -r daqsystemtest

Type of change

• New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

[bieryAtFnal]

For reference, here is the current output from dunedaq_integtest_bundle.sh --help:

Usage:
dunedaq_integtest_bundle.sh [option(s)]

Options:
    -h, --help : prints out usage information
    -r <the list of repositories for which integtests will be run>
       - this can be the name of a single repo; it defaults to "daqsystemtest"
       - it can be a pipe-delimited string with a list of repos, e.g. 'dfmodules|trigger'
       - it can have the special value of "all" - integtests in all repos will be run
       - it can have the special value of "local" - integtests in locally-cloned repos will be run
    -k, --include <pipe-delimited string to select the tests that will be run ('egrep -i' match to test name)>
    -x, --exclude <pipe-delimited string to specify tests to be excluded ('egrep -i' match to test name)>
    -n <number of times to run each individual test, default=1>
    -N <number of times to run the full set of selected tests, default=1>
    --stop-on-failure : causes the script to stop when one of the integtests reports a failure
    --verbosity <level> : requested level of console messages, in range 1-6, where 1 is least, 6 is DRUNC debug
    --trigger-full-rc-output <phrase that will trigger the full printout of run control messages>
       - the phrase can be a Python regex, which can be useful in handling colorized text
    --concise-output : suppresses run control and DAQApp messages in order to focus on test results
       - this is equivalent to "--verbosity 1"
    --tmpdir <dir> : specifies a root directory to use for test output, e.g. a directory instead of '/tmp'
    --list-only : list the tests that match the requested patterns without running them
    --pytest-options <options> : string with one or more dunedaq-specific command-line options to pass to Pytest
       - available options include the following:
         --dunerc-path <path> : Path to DUNE run control. Default is to search in $PATH
         --skip-resource-checks : Whether to skip the node resource (CPU/Memory) checks for this test
         --process-manager-type <type> : The run control process manager type to use for this test, e.g. ssh-standalone
         --dunerc-option <option-name> <option-value> : Repeatable, run control arguments without leading dashes
             for example, --dunerc-option log-level debug
       - example: --pytest-options "--skip-resource-checks --process-manager-type ssh-standalone --dunerc-option no-override-logs"

(I expect this output to change moderately soon because I have additional features in mind. But, I wanted to provide a snapshot of what is available in this PR.)

[eflumerf]

Tests look good to me with no new parameters