How do you use OpenHTF?

[glados-verma]

Contributors and users of OpenHTF, we OpenHTF maintainers are trying to better understand how everyone is using OpenHTF. I'm throwing together a quick template below, and will appreciate it if you can respond to this thread with whatever details you are able to share!

Template:

1. Do you use OpenHTF, or a fork of it? Can you link the fork?
2. How do you incorporate the code - PyPi, tarball download from GitHub, something else?
3. Do you use, or have your previously used, another (open source or commercial) test sequencer/executor framework?
4. Have you replaced or overridden any OpenHTF components with your own? If yes, which ones?
5. Name one OpenHTF future that you are curious about but haven't used yet?

Again, all questions are optional!

[mbenabda]

Do you use OpenHTF, or a fork of it? Can you link the fork?
Both. I use the Spintop-openhtf "fork" when the bench requires form inputs (i know, it's a silly reason, but with the web UI build not working, it was the easy route compared to integrating spintop's form-capable UserInput plug into upstream openhtf)

How do you incorporate the code - PyPi, tarball download from GitHub, something else?
PYPI

Do you use, or have your previously used, another (open source or commercial) test sequencer/executor framework?
Yes, but not in the context of hardware in the loop testing (JUnit, go test, pytest, jest, rspec, robot framework, cucumber, ...)

Have you replaced or overridden any OpenHTF components with your own? If yes, which ones?
No, but we've built a small LEDs based UI for our benches that is a websocket client to the exposed websocket server

Name one OpenHTF future that you are curious about but haven't used yet?
Monitors, diagnostics, history handlers

[lalten]

1. Do you use OpenHTF, or a fork of it? Can you link the fork?
We use a private fork with company-specific changes. We try to integrate any
upstream changes but it’s hard when upstream drops “internal changes” PRs with
thousands of lines of diff in a single commit.

2. How do you incorporate the code - PyPi, tarball download from GitHub, something else?
We consume a pip wheel built from the internal fork.

3. Do you use, or have you previously used, another (open source or commercial) test
sequencer/executor framework?
No

4. Have you replaced or overridden any OpenHTF components with your own? If yes, which
ones?

• We use a phase decorator that can catch and log known exceptions and return a
  specific return value (think FAIL_AND_CONTINUE) instead
• We’ve disabled the requirements for subtests to have unique names. It’s annoying -
  sometimes you need to do the same thing twice (power cycle a device, etc...).
• We override the UserInput plug to provide a richer terminal UI via
  python-prompt-toolkit (think editing with arrow keys, history, ...) and convert the web
  interface HTML prompt to text with w3m.

5. Name one OpenHTF feature that you are curious about but haven't used yet?

• We have barely touched on Phase Branches because there is seemingly a whole lot
  of boilerplate involved and the mechanics around diagnostics are generally
  unintuitive/convoluted.
• We don’t really make use of the possible separation of frontend and backend /
  multicast feature, everything is started on the same machine
• I’d love to know what the xmlrpclib is used for
• openhtf/util/data.py is something that we’ve not found a use for yet

General feedback:

• For a long time, it looked like the project was abandoned and unmaintained. It’s great to see
  that you’re putting some effort into getting the OpenHTF codebase into 2022.
• There are way too many ways to manage state in a test. The framework seems to encourage
  configuring everything ahead of time and preferring a stateless approach, but in practice our
  use cases tend to heavily involve e.g. only checking/doing something with a hardware
  component if it is in a certain state.
  • Some of the many approaches I’ve seen for dealing with this are
    • Tracking state in a plug, which lives for the duration of a Test
    • Using test_api.state
    • Diagnostics
    • Global state stores (CONF or our own)
    • test_api.get_measurements()
    • Checkpoints
  • Code quickly becomes convoluted due to the above varying methods. Moreover, the
    test records that are produced from test execution do not themselves reflect the
    many state relationships well and rather for the most part just contain a flattened list
    of phases.
  • I would be happy to see more mechanisms to simplify and work with conditional test
    execution logic.
• The web UI is buggy and it is hard to develop and create fixes due to severe lack of testing.
• Would love to see the typing fleshed out.
• Would love to see support for parallel execution of nodes.