tests: Make Tests.Cumulocity.Software Management.Software suite faster

[Bravo555]

Proposed changes

Tests.Cumulocity.Software Management.Software suite is our slowest test suite, running at over 10 minutes on the test runner, I thought to try to speed it up a bit to get some sense what kind of changes would be necessary in other suites to speed up the entire test suite. Currently it runs in about 30 minutes, I think it would be neat if it ran closer to 5-10 minutes.

This PR speeds up software test suite by:

• rewriting some tests that spawn new containers to not spawn + restore previous state
• removing some Sleep keywords that were useless because keywords after retry anyway

On my pc

before: 00:03:49.143
after: 00:01:33.541
diff: 00:02:16 (-59%)

on test runner

before: 00:10:41.672
(projected) after: 00:06:18

We're running the suites in parallel, so it will have very limited impact on total runtime, but after doing it I have a better idea of what type of changes would be required to speed up other tests. Apart from repeating these changes for other suites, one other speedup would be to try to replace retry mechanisms that always wait 2s with blocking mechanisms that return as soon as assertion succeeds. In some tests there are multiple of these kinds of checks, for example in test case Operation gets updated regardless of the order of the creation time we hit a 2s retry 4 times, so 8s in a test that takes 14s (on my machine), little over 50%, is spent waiting, so I expect it to also inflate runtime of other suites a bit.

Though speeding it up does take a lot of time, so next time I'll try to task other suites to an AI model.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
947	0	7	947	100	2h43m12.397813999s

[rina23q]

Nice experiment.

I think one problem of this test file is mixed of 3 level things:

1. testing against c8y with legacy software management
2. testing against c8y with advanced software management
3. testing locally, without using c8y

I agree that the previous setup, creating a test container per each test is time consuming and too much costs.
However, I sometimes think we put too many tests in one robot file. For this case, if we split the file into three (or at least two, covering point 1&2-c8y and point 3-local) files, maybe the tests would look short and easy to read.

[rina23q]

Having the c8y_SupportedSoftwareTypes fragment in ManagedObject triggers filtering out "Installed Software" list in the "Software" tab. To avoid an unexpected surprise, c8y.software_management.with_types is set to false by default.

I tested how to reset the fragment in c8y. We can send either JSON over MQTT or SmartREST.

• tedge mqtt pub c8y/inventory/managedObjects/update/${DEVICE_SN} '{"c8y_SupportedSoftwareTypes":null}'
• tedge mqtt pub c8y/s/us 143
  After the MQTT request is sent, if we ensure the fragment does not exist in MO, this is the clean enough state for this test.

In short, yes, we can remove this Setup by replacing it as I wrote above.

[rina23q]

(trivial) I think we can promote this to library level 🤔

[rina23q]

(note) This change makes most of tests run with c8y.software_management.with_types = true, which is against default. Previously, they were tested with the default (false) value. Eventually, this value doesn't affect tests unless it uses advanced software management.

[reubenmiller]

I think this change is problematic as you might get the message before it has been cleared by the mapper, so the static sleep was in-place to ensure the mapper is not clearing the message. Removing the timeout means that the tedge mqtt sub might see the previously manually posted message before the tedge-mapper reacts to it.