The agent runs the containers of every device of a Linux or macOS server: the connector (the only point of contact with the device), STF, Appium and the artifacts uploader (with S3, see Recordings). The connector takes the device whatever its connection is, the STF and Appium containers always take adb, usbmuxd and WebDriverAgent from it.
| Device | Connection | Linux | macOS |
|---|---|---|---|
| Android phone, tablet, TV | usb |
adb server of the connector (USB passed in) | adb server of the host (a launch agent), the connector forwards the device to its own adb server |
| Android | wireless |
adb connect of the connector | adb connect of the connector |
| Android emulator (AVD, Genymotion, a container Android) started outside the farm | android-sim |
adb connect of the connector to the address of the emulator |
the same, an AVD of the Mac is host.docker.internal:5555 |
| iPhone, iPad, Apple TV HD | usb |
usbmuxd of the connector | usbmuxd of the host, the connector installs and starts WebDriverAgent over it |
| iPhone, iPad, Apple TV over Wi-Fi | wireless |
netmuxd of the connector at address (paired over USB first) |
usbmuxd of the host (paired with the Mac) |
| iOS, iPadOS, tvOS simulator | ios-sim |
- | the connector runs xcrun simctl of the host over ssh |
| any iOS device whose WebDriverAgent is started by another tool (e.g. Apple TV 4K with tvOS 17+) | proxy |
the connector relays WebDriverAgent of address |
the same |
iOS 17.0 - 17.4.0 has a tunnel over the USB of a Linux host only (go-ncm); with the other connections WebDriverAgent of such a device must be started outside the containers.
Feel free to support the development with a donation for the next improvements.
| Requirements | |
|---|---|
| Operating System | Ubuntu 22.04+ (on 25.10+ see sudo-rs) macOS 13 (Ventura)+, Apple Silicon or Intel |
| CPU | 8+ Cores |
| Memory | 32 Gb RAM |
| Free space | SSD 128Gb+ of free space |
-
Install docker: Docker Engine on Linux, Docker Desktop on macOS.
-
Linux: the user of the agent runs docker without sudo (the deploy pulls the images,
zebrunner.shandzebrunner-farmmanage the containers as this user):sudo usermod -aG docker $USER, then log in again. -
Install ansible 2.10+ (the playbooks use
ansible.builtinmodule names, tested with ansible-core 2.22, the version of CI intests/requirements-lint.txt), e.g.pip install ansible-core. -
python3 and curl:
./zebrunner.sh joinanddoctorneed them. They read and write the settings with PyYAML ofpython3or, without it there, of the python of ansible (ansible-playbook --versionshows it);ZEBRUNNER_PYTHONsets another one. -
macOS on Apple Silicon: install Rosetta 2 and let Docker Desktop use it: the image of appium is
linux/amd64only and runs much slower under the QEMU emulation of Docker Desktop without it:softwareupdate --install-rosetta --agree-to-license
then turn on Use Rosetta for x86_64/amd64 emulation on Apple Silicon in Docker Desktop (Settings, General) and restart it.
./zebrunner.sh setupand./zebrunner.sh statusreport it. -
macOS: install socat utility to share usbmuxd websocket into the device containers
valid path to socat binary should be
/usr/local/bin/socaton Apple Silicon Homebrew installs it to/opt/homebrew/bin/socat, link it:sudo ln -s /opt/homebrew/bin/socat /usr/local/bin/socat -
The server of mcloud is set up and started: the devices connect to it.
-
Clone mcloud-agent repository and execute setup procedure
git clone https://github.com/zebrunner/mcloud-agent.git cd mcloud-agent ./zebrunner.sh setup -
The setup ends with the numbered next steps: the invite of this host on the MCloud server,
./zebrunner.sh join, the devices,./zebrunner.sh doctor,./zebrunner.sh ansibleandzebrunner-farm start.
The local settings of the host are in config/, out of git; ./zebrunner.sh setup makes them from the examples (an existing file is kept, its copy goes to <file>.bak):
| File | What it is | Example |
|---|---|---|
config/settings.yml |
the settings of the host, the same for Linux and macOS: the connection to MCloud of ./zebrunner.sh join, S3, WebDriverAgent; mode 600, it has secrets |
config/settings.yml.original |
config/devices.yml |
the list of the devices (devices:) |
config/devices.yml.original (Linux), config/mac-devices.yml.original (macOS) |
The other files of the settings are in git, do not change them: git pull brings their new values. A line of config/settings.yml overrides any of them:
| File | What it is |
|---|---|
versions.yml |
the images of the devices and their versions |
roles/common/defaults/main.yml |
the values by default of both OS: the options of Appium (APPIUM_RELAXED_SECURITY, ...), RESTARTS_LIMIT, STF_DOCKER_NETWORK (infra: the docker network of the containers of the devices, the deploy and zebrunner-farm create it), the empty optional settings (STF_API_URL, STF_AUTH_TOKEN, S3_*, WDA_FILE, WDA_BUNDLEID, the supervision identity of the MDM MDM_SUPERVISION_P12*): a line of them removed from config/settings.yml is the value by default, the Developer Disk Images (DDI_*) |
roles/devices/defaults/main.yml, roles/mac-devices/defaults/main.yml |
the values by default of the OS: STF_PROVIDER_NAME; USBMUXD_SOCKET_ADDRESS, ADB_SERVER_SOCKET_ADDRESS of macOS |
The playbooks load them in this order: the values by default of the roles, versions.yml, config/settings.yml, config/devices.yml. The deploy renders the settings zebrunner-farm runs with into /usr/local/etc/zebrunner-farm.env: udev and launchd run it without the environment of the user, a change of config/ needs ./zebrunner.sh ansible (./zebrunner.sh doctor reports the settings that differ). ./zebrunner.sh backup keeps the two files of config/ in a private folder backup/bak_<time>_<random>/config/, restore puts them back; the code of the agent is not in a backup, it comes from git (git pull).
The earlier versions kept everything in roles/devices/vars/main.yml (Linux) or roles/mac-devices/vars/main.yml (macOS), and the images and the options of Appium in defaults/main.yml (now versions.yml and roles/common/defaults/main.yml). Such a vars/main.yml overrides config/, so the deploy and ./zebrunner.sh doctor refuse it. ./zebrunner.sh migrate moves them: it shows the changes of config/ (the secrets hidden) and asks first, then writes the settings that differ from the ones the deploy has already into config/settings.yml and the devices: into config/devices.yml (the previous files are kept as *.bak), and renames the files of the earlier version to *.migrated. With a local change of defaults/main.yml, git pull stops: keep a copy, drop the change, pull, then migrate the copy:
cp defaults/main.yml defaults.local.yml && git checkout defaults/main.yml && git pull
./zebrunner.sh migrate defaults.local.ymlEvery device host joins MCloud with its own token: the administrator of MCloud invites it, the host enrolls itself, no secret is copied by hand.
- On the MCloud server:
./zebrunner.sh node-invite <label> <address>. The label is the name of the host (a-z,0-9,-), the address is the one the MCloud server connects to: the hub registers the Appium nodes by it, stf-proxy takes the screens from it (in the network of the MCloud server the address there, see Device hosts in a private network; on the MCloud server itself its LAN address, see Devices on the MCloud server). It prints a join string, valid once and for a limited time; a new invite of the label replaces it. - On this host:
./zebrunner.sh joinasks for the join string (hidden; or on stdin: never in the command line, where it stays in the shell history) and enrolls the host:- it makes the CURVE key of the host if there is none and sends its public key, never the secret one;
- it talks to MCloud over HTTPS and trusts only the server whose TLS key is the pinned one of the join string (a self-signed certificate is fine, another key stops the enrollment);
- it writes the connection to MCloud into
config/settings.yml(mode 600, the previous settings are kept insettings.yml.bak-join-<time>).
./zebrunner.sh doctor,./zebrunner.sh ansibleandzebrunner-farm start(restartfor the existing containers).
The settings of the connection, written by join (do not change them by hand, a new join string replaces them):
| Setting | Value |
|---|---|
MCLOUD_URL, MCLOUD_TLS_PIN |
the API of MCloud on the address of the join string and the pin of its TLS key: ./zebrunner.sh doctor checks the enrollment with them |
MCLOUD_NODE_TOKEN |
the token of this host: appium registers in mcloud-grid with it (NODE_REGISTRATION_TOKEN of appium), STF of the devices authenticates to the storage and proves its devices with it (STF_NODE_TOKEN of mcloud-device) |
STF_PROVIDER_HOST |
the address of the invite |
PUBLIC_IP, PUBLIC_IP_PORT, PUBLIC_IP_PROTOCOL |
the public URL of MCloud: the devices upload the screenshots and download the apps there |
STF_PROVIDER_CONNECT_*, SELENIUM_HUB_HOST, SELENIUM_HUB_PORT |
the triproxies of STF and mcloud-grid on the address of the join string |
STF_ZMQ_CURVE_SERVER_PUBLIC_KEY, STF_ZMQ_CURVE_SECRET_KEY |
the CURVE keys of ZeroMQ: the one of the server and the one of this host, see ZeroMQ security |
REMOTE_CONNECT_HOST is set by hand when needed, see Device hosts in a private network.
Security:
- the token is of this host only: MCloud keeps its hash, binds it to the address of the host and to its devices, and revokes it alone (
./zebrunner.sh nodes revoke <label>of mcloud); config/settings.ymland the deployed settings ofzebrunner-farm(/usr/local/etc/zebrunner-farm.env) have the secrets of the host:config/settings.ymlis mode 600, the settings ofzebrunner-farmare readable by root and the user of the deploy only (Linux: mode 640, the group of the user; macOS: mode 600, the user); the scriptzebrunner-farmhas no secret and passes them to docker in its environment (-e NAMEwithout the value), not in its arguments that every local user sees (ps); the users of thedockergroup can read the variables of the containers anyway, they are root on the host;- a new join string (
node-inviteof the same label) replaces the token and the key of the host: run./zebrunner.sh joinagain to rotate them; ./zebrunner.sh doctorasks MCloud about this host (enrolled, active, its address, the devices STF knows and binds to it) and reports the settings readable by other users.
-
./zebrunner.sh ansiblerunsagent.ymlon both OS: its first play groups the hosts by their OS (os_Linux,os_Darwinofhosts), the next ones deploy the role of each OS. -
The images of the devices are pinned in
versions.yml:DEVICE_VERSION(mcloud-device),APPIUM_VERSION,IOS_CONNECTOR_VERSION,ANDROID_CONNECTOR_VERSIONandUPLOADER_VERSION. mcloud-device must be the one of the STF of mcloud (TAG_STFof its.env). Another version is a line ofconfig/settings.yml(see Settings). Ondevelopthey are<version>-develop-latest, the newest image of thedevelopof each repository (UPLOADER_VERSIONis a release, artifacts-uploader has no such image); the deploy pulls them again, a release pins its versions. -
[Optional] To compare images in the tests (the Appium images plugin) appium needs the image of its
appium-imagestage, the-imgpostfix of its version. It is not published: build it with appium (make build VERSION=3.1.1-img TARGET=appium-image) on every device host (the deploy keeps a local image it cannot pull) or push it to a registry of yours, and set it inconfig/settings.yml:APPIUM_IMAGE: registry.example.com/zebrunner/appium APPIUM_VERSION: 3.1.1-img
The published appium images: https://gallery.ecr.aws/zebrunner/appium
-
update
config/settings.ymlandconfig/devices.ymlfiles according to the obligatory/optional comments inside (see Settings).Register all whitelisted Android and iOS devices with their udids! The deploy validates the devices: obligatory settings, the characters of the ids, names and locations (letters, digits,
_,.,-), unique ids and names, the size ofmin_port-max_port(4 ports of an Android device besidesconnect_port), and host ports (appium_port,min_port-max_port,connect_port) not shared by devices.adb_portis not used on Linux;connect_port(optional, Android) is the remote debug port of STF. -
Run ansible-playbook script to download the required components (the connector of the platforms of the registered devices only) and set up udev rules:
./zebrunner.sh ansible
To deploy again without the download of the images, e.g. after a change of
config/devices.yml, use the command below. It runs the whole role of the OS (the checks,zebrunner-farmand its settings, the list of the devices, the udev rules and the systemd units) and keeps the running containers of the devices:zebrunner-farm restart <devices>creates them with the new settings../zebrunner.sh ansible devices
The arguments after
ansibleoransible devicesare passed to ansible-playbook, e.g.-vvv; a run that changes nothing (--check,--syntax-check,--list-tasks) does not offer to remove the running containers. The agent is deployed for the user who runs./zebrunner.sh(the inventoryhostsconnects locally, the files and the launch agents are of itsUSERandHOME), so--userhas no effect: run it as that user. The sudo password is asked interactively: do not pass it in the command line, where it stays in the shell history and is visible in the processes list../zebrunner.sh ansible devices -vvv
-
Running containers of the devices keep the settings and the images they were created with:
./zebrunner.sh ansiblelists them and offers to remove them first (zebrunner-farm down, Enter is yes), otherwise runzebrunner-farm restartafter the deploy. -
Run
zebrunner-farm startto create the containers of the devices connected before the deploy and of the wireless, emulator and proxy devices: udev starts the containers of a USB device on its next connection only. -
Devices management script is deployed to /usr/local/bin/zebrunner-farm (the same script on Linux and macOS), its settings to /usr/local/etc/zebrunner-farm.env (see Settings).
-
Udev rules with whitelisted devices are in /etc/udev/rules.d/90_mcloud.rules: a connection of a USB device starts the systemd unit
mcloud-device-restart@<id>, a disconnectionmcloud-device-down@<id>(/etc/systemd/system), which runzebrunner-farm restartordownfor it. udev only queues the job: its programs are killed after the event (180 s), the containers may need longer (the images are pulled). A disconnection cancels an unfinished restart and the other way round. -
The commands of one device (
zebrunner-farm start,stop,restart,downof the device or of a group with it, the units of udev) run one at a time (flock,/run/lock/mcloud-device-<id>.lock): one waits for up to 30 minutes for another, a group takes its devices one by one. -
Whitelisted devices properties are in /usr/local/bin/mcloud-devices.txt, rendered from
config/devices.yml: one device per line, its fieldskey=value(the keys ofconfig/devices.yml) separated by|; an empty value is the value by default (wda_file,wda_bundleid:WDA_FILE,WDA_BUNDLEIDof the settings). -
Usbmuxd service is stopped and masked (disabled).
-
Wireless, emulator and proxy devices have no udev rules:
zebrunner-farm startcreates their containers. -
Android emulators (
android-sim) are started outside the farm (e.g. an AVD, Genymotion, a container Android): setaddressto their adb address reachable from the docker network; appium getsDEVICE_CONNECTION=android-sim(adefaultdevice is a phone). -
[Optional] The containers of all the devices after a reboot:
./zebrunner.sh autostart enable, see Autostart. -
[Optional] Increased
fs.inotify.max_user_instancesto workaround #328sudo sysctl fs.inotify.max_user_instances=2048
-
update
config/settings.ymlandconfig/devices.ymlfiles according to the obligatory/optional comments inside (see Settings).Register all whitelisted Android and iOS devices (phones, tablets, TVs or simulators) with their udids and connections! The deploy validates the devices: obligatory settings, the characters of the ids, names and locations (letters, digits,
_,.,-), unique ids and names, the size ofmin_port-max_port(4 ports of an Android device besidesconnect_port), the connections, host ports (appium_port,min_port-max_port,connect_port,wda_portandmjpeg_portof the iOS simulators,adb_portof the USB Android devices) not shared by devices. -
Run ansible-playbook script to download the required components (the connector of the platforms of the registered devices only) and set up launchd agents:
./zebrunner.sh ansible
To deploy again without the download of the images, e.g. after a change of
config/devices.yml, use the command below. It runs the whole role of the OS (the checks,zebrunner-farmand its settings, the list of the devices, the launch agents) and keeps the running containers of the devices:zebrunner-farm restart <devices>creates them with the new settings../zebrunner.sh ansible devices
The arguments after
ansibleoransible devicesare passed to ansible-playbook, e.g.-vvv; a run that changes nothing (--check,--syntax-check,--list-tasks) does not offer to remove the running containers. The agent is deployed for the user who runs./zebrunner.sh(the inventoryhostsconnects locally, the files and the launch agents are of itsUSERandHOME), so--userhas no effect: run it as that user. The sudo password is asked interactively: do not pass it in the command line, where it stays in the shell history and is visible in the processes list../zebrunner.sh ansible devices -vvv
-
Devices management script is deployed to /usr/local/bin/zebrunner-farm (the same script on Linux and macOS), its settings to /usr/local/etc/zebrunner-farm.env (see Settings).
-
The containers run on the platform of the Mac, arm64 on Apple Silicon (the STF provider of an iOS simulator hangs under emulation, the other ones are slower), but appium: its image is
linux/amd64only (see appium), emulated by Docker Desktop with Rosetta 2 on Apple Silicon. -
Whitelisted devices properties are in /usr/local/bin/mcloud-devices.txt, rendered from
config/devices.yml: one device per line, its fieldskey=value(the keys ofconfig/devices.yml) separated by|; an empty value is the value by default (wda_file,wda_bundleid:WDA_FILE,WDA_BUNDLEIDof the settings). -
Deployed and loaded $HOME/Library/LaunchAgents/ZebrunnerUsbmuxd.plist to share usbmuxd into the device containers: socat on localhost of the Mac, on the port of
USBMUXD_SOCKET_ADDRESS(host.docker.internal:2222by default) the containers reach it with -
USB Android devices: deployed and loaded $HOME/Library/LaunchAgents/ZebrunnerAdb.plist, the adb server of the host (
brew install android-platform-tools) on the port ofADB_SERVER_SOCKET_ADDRESS(host.docker.internal:5037by default); the connectors take the devices from it, the containers of a USB Android device are created byzebrunner-farm start(the devices listener handles iOS only). -
iOS simulators (
ios-sim): the ssh key of the agent is generated intossh/id_ed25519of the agent folder and authorized in~/.ssh/authorized_keys(restrict), turn on Remote Login (System Settings, General, Sharing) for the user. Set the simulator build of WebDriverAgent (WDA_SIMULATOR_FILEorwda_fileof the device), e.g.WebDriverAgentRunner-Build-Sim-arm64.zipof appium/WebDriverAgent. -
Deployed $HOME/Library/LaunchAgents/ZebrunnerDevicesListener.plist to load and manage iOS devices connect/disconnect automatically: the devices listener (/usr/local/bin/zebrunner-device-listener and usbmuxd_watch) restarts the containers of a connected USB device and removes them on disconnect, devices reachable over Wi-Fi only are not handled by it.
-
Log rotation of the devices listener log is set up in /etc/newsyslog.d/zebrunner-mcloud.conf.
-
Run
zebrunner-farm startto create the containers of the connected devices and to load the devices listener (otherwise it is loaded on the next login only).
./zebrunner.sh without arguments (or --help) lists its commands, grouped by stage: set up, operate, maintain. Its output has the terminal style of every MCloud repository (patch/ui.sh, CONVENTIONS.md): colors on a terminal only (NO_COLOR, UI_COLOR=always or never), the Unicode glyphs with a UTF-8 locale (UI_ASCII=true for ASCII); without a terminal it is plain text. The commands:
setup- prepares the agent: the settings ofconfig/from the examples. It changes no shell profile:zebrunner.shknows its folder,zebrunner-farmhas it in its settings, sozebrunner-farm ansibledeploys again from any folder; theZEBRUNNER_MCLOUD_AGENT_DIRline of the earlier versions is removed from the profiles.ansible [devices] [arguments]- deploys the agent,deviceswithout the download of the images and keeping the running containers of the devices, the arguments are of ansible-playbook (see Run ansible playbook).status- shows the state of the agent components: tools, Rosetta 2 of an Apple Silicon Mac, deployed files, launch agents.backupandrestore- save the settings ofconfig/into thebackupfolder and restore them from it (see Settings); a backup of an earlier version (vars/main.yml,defaults/main.yml) goes toconfig/with./zebrunner.sh migrate <its files>.shutdown- removes the devices containers, the launch agents or udev rules, the deployed files, the settings, the ssh key of the simulators (and its line of~/.ssh/authorized_keys) and the environment variable (asks for confirmation).version- shows the versions of the agent components.doctor- checks the host, the settings, the deployment, the network to the MCloud server and the devices, and tells what to fix, see Doctor.join- enrolls this host in MCloud with the join string of./zebrunner.sh node-inviteof mcloud, see Node enrollment.migrate [files...]- moves the settings of an earlier version toconfig/, see Settings.autostart [status|enable|disable]- the containers of the devices after a reboot, off by default, see Autostart.
The deploy adds no autostart. After a reboot the USB devices are started by udev (Linux) and by the devices listener at the login (iOS of macOS); the containers of the other devices (wireless, emulators, proxy; the USB Android devices of macOS) stay stopped: zebrunner-farm start starts them. The autostart is a choice of the administrator of the host, off by default:
./zebrunner.sh autostartshows whether it is on../zebrunner.sh autostart enabletells what it adds, shows the content of the file and asks first (No by default); nothing starts at once:- Linux: the systemd unit
/etc/systemd/system/mcloud-farm.service(sudo,systemctl enable), it runszebrunner-farm start --quietonce after docker at the boot; its log:journalctl -u mcloud-farm; - macOS: the launch agent
~/Library/LaunchAgents/ZebrunnerFarmStart.plistof the user (no sudo), at the login it waits for Docker Desktop (10 minutes at most: let it start at the login) and runszebrunner-farm start --quietonce; its log:logs/autostart.logof the agent folder.
- Linux: the systemd unit
./zebrunner.sh autostart disabletells what it removes and asks first; the running containers keep running../zebrunner.sh shutdownremoves it too.
STF of MCloud accepts on its ZeroMQ ports only the enrolled device hosts: the CURVE key of every host is in the registry of MCloud. ./zebrunner.sh join makes the key of this host (STF_ZMQ_CURVE_SECRET_KEY, kept when it is set), sends its public key and writes the public key of the server (STF_ZMQ_CURVE_SERVER_PUBLIC_KEY); a revocation of the host on the MCloud server disconnects it. patch/curve_keygen.py makes the keys with Python only (X25519 of RFC 7748), the same as stf curve-keygen.
./zebrunner.sh doctor checks the agent and tells what to fix, it changes nothing and prints no secret:
- the host: docker, ansible, python3, lsusb (Linux), Rosetta 2 (Apple Silicon);
- the YAML of
config/settings.ymlandconfig/devices.yml: doctor reads the settings as the playbooks load them (the values by default of the roles,versions.yml,config/settings.yml), a file that is not valid YAML is a problem; - the settings of
config/settings.yml: the ones of./zebrunner.sh join, the formats of the token and of the CURVE keys, the file readable by other users, aroles/.../vars/main.ymlof an earlier version; the devices ofconfig/devices.ymlwith the checks of the deploy (unique ids and names, the obligatory settings, the characters of the ids, names and locations, the size of the port ranges, the host ports, the connections of the OS); - the enrollment: it asks MCloud about this host (
GET /api/v1/nodes/selfofMCLOUD_URLwith the token, over HTTPS to the pinned TLS key): enrolled and active (or unknown, revoked), its address in MCloud isSTF_PROVIDER_HOST, the devices STF knows of it, which of them are bound to it and connected; - the deployment: the settings of
zebrunner-farm(/usr/local/etc/zebrunner-farm.env) are the ones of this folder and the current ones, also the versions of the images (otherwise./zebrunner.sh ansible) and readable by this user only, the containers created with the currentSTF_PROVIDER_HOST(otherwisezebrunner-farm restart); - the network: the name resolution and the ports of
PUBLIC_IP, mcloud-grid and the ZeroMQ triproxies;STF_PROVIDER_HOSTis an address of this host, or a public address of a NAT while the MCloud server is in the network of this host (see Device hosts in a private network); MCloud on this machine and the devices using its ports (see Devices on the MCloud server); - the remote debug: with
REMOTE_CONNECT_HOSTthe adb relay of mcloud accepts the remote debug ports of the Android devices (printed); - the devices: their containers, the refused grid registrations of Appium and the refused CURVE key in the logs of the last 30 minutes.
The MCloud server checks a host too: ./zebrunner.sh nodes check <label> of mcloud shows its record, its devices and whether the server reaches their screens.
Every check is a line: a green, amber or red dot on a terminal (ok, warn, FAIL without colors), a muted circle (info) for a fact, and fix under a problem; the last lines count them (12 ok, 1 warn, 0 fail without colors) and tell what to do. It exits with 1 when a check fails.
Many device hosts without public addresses can work with one MCloud server of their network: the users reach the MCloud server only. The screens go from the browser to stf-proxy of mcloud (/d/<STF_PROVIDER_HOST>/...) and from it to the device host, the tests go to mcloud-grid and from it to the Appium node of the device host (http://<STF_PROVIDER_HOST>:<appium_port>); ZeroMQ, the storage and S3 are connections of the device host to the server.
- Every device host is invited with its address in the network (
./zebrunner.sh node-invite <label> <address>), whichjoinwrites as itsSTF_PROVIDER_HOST. PUBLIC_IPstays the name the users open MCloud with: the device host resolves it to the address of the server in the network (e.g. a line in/etc/hosts, the containers resolve the names through the resolver of the host);STF_PROVIDER_CONNECT_*,SELENIUM_HUB_HOSTandMCLOUD_URLare on the address of the join string.- A public address of the router as the address of the invite fails inside the network when the router has no NAT hairpin: mcloud-grid refuses the nodes (Appium logs the registration
500withNode http://... is not reachable from the hub) and the screens stay gray../zebrunner.sh doctorfinds it and prints the address to invite the host with. - The remote debug of STF (
adb connect <host>:<port>) of an Android device uses its remote debug port:connect_portof the device, the second port ofmin_port-max_portwithout it (the first one is the screen, which is never relayed). It goes through the adb relay of mcloud: setREMOTE_CONNECT_HOSTto the public name of MCloud and turn onADB_RELAY=truein.envof mcloud, the relay takes the remote debug ports of the devices of the enrolled hosts from STF;ADB_RELAY_PORTSof mcloud limits them to the range opened in its firewall, so keep theconnect_portof all the device hosts in one range, unique among the hosts (see mcloud). Without the relay only the users of the network reachadb connect <STF_PROVIDER_HOST>:<port>. ./zebrunner.sh doctorchecks the relay on the remote debug port of the first deployed Android device and prints the remote debug ports of this host. The remote debug is of the Android devices only.
The devices can be on the MCloud server itself (mcloud and mcloud-agent on one machine):
AGENTS_ADDRESSof.envof mcloud is the LAN address of the machine, so is the address of the invite (./zebrunner.sh node-invite <label> <LAN address>): not127.0.0.1orlocalhost, which the containers of mcloud cannot reach the devices with (MCloud refuses them);- the devices share the
infranetwork with mcloud and reach the published ports of MCloud on that address; - the ports of the devices (
appium_port,min_port-max_port,connect_port) are not the ports of MCloud: its HTTPS port (STF_PORT), 4446 (mcloud-grid), 7160, 7250, 7260, 7270 (the triproxies of STF) and the remote debug ports of the adb relay (ADB_RELAY_PORTSof mcloud);./zebrunner.sh doctorreports the collisions with the HTTPS port, the hub and the triproxies, not withADB_RELAY_PORTS, which it does not know: keep them out of the ranges of the devices of this machine.
The Appium node of a device registers in mcloud-grid with the token of this host (MCLOUD_NODE_TOKEN, passed to appium as NODE_REGISTRATION_TOKEN): the hub accepts it only when STF knows the token of an active host whose address is the host of the node url (STF_PROVIDER_HOST), checks it again every minute and sends it back to this node only with every request, see mcloud-grid. A revoked host is unregistered.
The docker network of the containers of the devices (STF_DOCKER_NETWORK, infra by default) is trusted: every container on it reaches the ports of the connectors without authentication, the adb server of every Android device and, of every iOS device, usbmuxd (2222), WebDriverAgent and MJPEG and the go-ios tunnel API and port (mcloud-ios-connector); attach only the containers of the farm (and of mcloud on the same machine) to it.
The Appium port of a device (appium_port) is published on the host for mcloud-grid. Appium of the device answers only the requests of the grid, which carry the token of this host, and the ones of its own container: the others get 401 (also Appium Inspector and the WebSockets of the sessions, e.g. the logcat broadcast; use the grid, /mcloud/wd/hub of mcloud).
APPIUM_RELAXED_SECURITY of roles/common/defaults/main.yml is true: every session, also through the grid, gets the insecure features of Appium: mobile: shell (adb shell of the device), the Appium log of the earlier sessions of the device (get_server_logs), the emulator console, the screen streaming, the download of chromedriver. Set it to false in config/settings.yml when the tests do not need them.
With S3 (S3_BUCKET and the other S3_* settings of config/settings.yml) appium records the video of every session and the uploader container of the device uploads it with the logs of the session to s3://<S3_BUCKET>[/<S3_TENANT>]/artifacts/test-sessions/<session id>/. Without S3_BUCKET appium records no video (RECORD_ARTIFACTS=false) and the device has no uploader: there is nothing to upload to, the videos filled the disk. Appium still keeps the log of every session in a folder of the volume of the logs of the device, the volume is removed with the containers (zebrunner-farm restart, down, a disconnection of a USB device). A change of S3 needs ./zebrunner.sh ansible and zebrunner-farm restart.
zebrunner-farm <command> [devices...] [-x|--exclude <devices>]... [-q|--quiet] [--batch|--rolling] [--json] manages the containers of the devices: status, start (creates the missing containers, starts the stopped ones, keeps the running ones), stop, restart (removes and creates them again) and down (removes them); zebrunner-farm mdm the MDM of a device.
-
restartof several devices has two strategies:--batch(the default,--strategy batch) removes the containers of all the devices one by one, then creates and starts them;--rolling(--strategy rolling) removes and creates the containers device by device, the other devices keep working meanwhile. The question of a terminal shows the strategy,sswitches it. -
The devices are udids or names, groups (
ios,android,usb,wireless,android-sim,ios-sim,proxy) or patterns of the udids, names and locations in quotes ('Samsung_*'), several ones separated by spaces or commas; all the devices without them. -
-x(-e,--exclude, also-x=<devices>) skips devices, groups or patterns, e.g.zebrunner-farm restart android -x Pixel_8_Pro -x 'Samsung_Galaxy_S2?'. -
status --jsonprints the devices and their containers as a JSON array for the monitoring, the messages go to stderr: per deviceid,name,os,connection,location,plugged_in(a USB device:trueorfalse,nullof the other connections) andcontainers(connector,stf,appium,uploader):exists, and of an existing onestatus(docker:running,exited, ...),health(healthy,unhealthy,starting,nullwithout a health check),started_at,uptime_seconds(of a running one) andrestarts, e.g.zebrunner-farm status --json android | jq -r '.[] | select(.containers.stf.health != "healthy") | .name'. -
In a terminal
start,stop,restartanddownof a group show the numbered devices with the state of their containers and ask: Enter oryruns the command,ncancels,f <devices>keeps only these devices of the list,x <devices>excludes them;<devices>are the numbers of the list, udids, names, groups or patterns.-q(--quiet) runs at once; a single device named exactly, udev, launchd and scripts without a terminal are not asked. -
Every device prints one line per container:
running(kept),started,created <id>or the error of docker; the containers after a failed one are not created. -
stop,restartanddownstop the containers of a device one by one and wait for them: appium first (up to 40 s: it finishes the recordings of the active sessions), then the uploader (up to 130 s: it uploads them, itsFINISH_TIMEOUTis 120 s), STF (30 s) and the connector (10 s);restartanddownremove a container after its stop and the volume of the logs after all of them. The stop of a device with the recordings to upload takes a few minutes, the macOS devices listener waits for its actions up to 30 minutes as the units of udev of Linux.
The MDM of the devices of MCloud (mcloud-mdm, MDM_ENABLED=true of mcloud, see MDM of mcloud): the agent of mcloud-mdm as the device owner of an Android device (no factory reset), the enrollment profile of a supervised iOS device. The host reaches it with the settings of ./zebrunner.sh join (MCLOUD_URL, MCLOUD_TLS_PIN, MCLOUD_NODE_TOKEN): MCloud checks the node token and lets its own MDM in, the host needs no setting of its own for Android. The public address of the site must be in MDM_SITE_ADDRESSES of mcloud.
./zebrunner.sh mdm <status|enroll|release|supervise> [devices...] [-x <devices>] [-q|--yes] [--uninstall] [--force] prepares the devices for the farm (zebrunner-farm keeps the work of the running devices). The devices are selected as by zebrunner-farm (udids, names, groups such as ios or android, patterns in quotes, -x to skip some; status without devices: all of them), shown at a terminal before a change, then handled one by one, each with the lock of zebrunner-farm:
statusasks the MDM only, the device is not touched: Androidknown,device_owner,last_sync_at,agent_version,managed,presets; iOSknown,enrolled,last_seen_at,product,os_version.enrollstops the containers of the device (a new device without containers is enrolled too), runs a one-shot container of the image of its connector with its USB device (Linux) or with the adb server and usbmuxd of the host (macOS), then starts the containers that ran again:- Android: the agent of the MDM installed, the device owner set (Android refuses it while the device has an account, e.g. Google, or a second user: remove them first), the agent set up with the answer of the MDM; the adb key of the farm (
mcloud-key-volume), no new prompt on the device; - iOS: the enrollment profile of the MDM installed with the supervision identity of the farm,
MDM_SUPERVISION_P12(a PKCS#12 file of the host) andMDM_SUPERVISION_P12_PASSWORDofconfig/settings.yml, without a prompt; the device must be supervised by that identity (supervise). A lost pairing is made again with the identity, the pairing records of the device stay in its volume<device>-lockdown.
- Android: the agent of the MDM installed, the device owner set (Android refuses it while the device has an account, e.g. Google, or a second user: remove them first), the agent set up with the answer of the MDM; the adb key of the farm (
supervise(iOS) erases the device and prepares it without a person at the device: Activation Lock (Find My) and a passcode are checked first (the device is not erased then), the erase, the activation, the supervision with the identity of the farm (MDM_SUPERVISION_ORG, every Setup Assistant pane skipped), Developer Mode, the Wi-Fi of the farm (MDM_WIFI_SSID,MDM_WIFI_PASSWORD,MDM_WIFI_ENCRYPTION: the MDM reaches the device over it), the enrollment; the containers of the device are created again (the device comes back on USB with another number), the connector installs WebDriverAgent again and continues the Setup Assistant (it shows "iPhone Partially Set Up" after the restart of Developer Mode: WebDriverAgent swipes Hello up and touches "Continue with Partial Setup" untilSetupDoneof the device) and checks that an app starts. A device supervised already is only enrolled (--forceerases it anyway). It asks for the worderaseat a terminal,--yeswithout one. On Linux the one-shot container gets the USB bus of the device (the device restarts on it with another number): the other iPhones of the bus stop meanwhile (their locks are held), their containers start again at the end. About 10 minutes a device; checked with iOS 18.3 and 26.2.1 on Linux.release: Android, the agent gives up the device owner (the restrictions of the MDM go away, no factory reset),--uninstallremoves the agent too; iOS, the MDM removes its enrollment profile over APNs (the device leaves the MDM with every profile of it once it is online, it stays supervised).
The node token, the setup of the agent and the passwords of the identity and of the Wi-Fi go to curl and to the one-shot containers on their stdin or in a folder of the process (700), never in the arguments of a process. adb alone cannot remove the agent of a device (it is no test-only app): the test users of the remote debug cannot release a device.
- Enable the Developer Option and USB Debugging for each Android device.
- Connect an Android device physically to a USB direct port or through the hub.
- For the 1st connection, trust the device by picking "always trust..." on the device.
- Android: enable Wireless debugging (or
adb tcpip 5555over USB), setconnection: wirelessandaddress(the IP, theidby default). - iOS on Linux: connect the device over USB once with
connection: usb(it is paired with the farm), allow its lockdown over Wi-Fi:docker exec device-<name>-<udid>-connector mcloud-ios wifi-connections <udid> on, then setconnection: wirelessandaddress(its IP, a DHCP reservation is recommended) and re-run./zebrunner.sh ansible devices. - iOS on macOS: pair the device with the Mac and enable "Connect via network" in Xcode, set
connection: wireless. - Apple TV 4K (tvOS 17+) pairs with a PIN over RemotePairing, which the containers do not support yet: start its WebDriverAgent with another tool and use
connection: proxywith itsaddress.
- Enable Settings -> Developer -> Enable UI Automation.
- Settings -> Safari -> Advanced -> Web Inspector.
- Enable Siri.
- Disable screen lock:
Phone and Tablet: Settings -> Lock screen -> Turn display off when inactive -> select Never
Apple TV: Settings -> General -> Sleep after -> Select never
You need an Apple Developer account to sign in and build WebDriverAgent.
-
Clone WebDriverAgent from source you prefer. We recommend our forked repository with performance improvements.
git clone https://github.com/zebrunner/WebDriverAgent.git
The fork also builds it from the command line, without the steps 2-8:
bash Scripts/zebrunner-build.sh --help(--sdk devicefor the devices,--sdk simfor the simulators, signed with--code-sign yes --team-id <team>). The installable.ipaof a device build comes with zebrunner/WebDriverAgent#17; an earlier device build is a.zipof the.app: put it intoPayload/as in the steps 6-8. -
Open WebDriverAgent.xcodeproj in Xcode.
-
Ensure a team is selected before building the application. To do this, go to Targets and select each target (one at a time). There should be a field for assigning team certificates to the target.
-
Remove your WebDriverAgent folder from DerivedData and run Clean build folder (just in case).
-
Build the application by selecting the WebDriverAgentRunner target and build for Generic iOS Device. Run Product -> Build for testing. This will create a Products/Debug-iphoneos in the specified project directory. Example: /Users/$USER/Library/Developer/Xcode/DerivedData/WebDriverAgent-dzxbpamuepiwamhdbyvyfkbecyer/Build/Products/Debug-iphoneos
-
Go to the "Products/Debug-iphoneos" directory and run: mkdir Payload
-
Copy the WebDriverAgentRunner-Runner.app to the Payload directory: cp -r WebDriverAgentRunner-Runner.app Payload
-
Finally, zip up the project as an *.ipa file: zip -r WebDriverAgent.ipa ./Payload
Make sure to specify relative
./Payloadto archive only Payload folder content -
Share built ipa via WDA_FILE variable in config/settings.yml file.
to override WDA_FILE artifacts per each device use
wda_fileandwda_bundleidiOS device properties and re-execute ansible playbook. -
The device connector installs WebDriverAgent from the ipa on Linux and on macOS (over usbmuxd of the host), a WebDriverAgent started on macOS by the host is used as is.
-
The bundle id is
com.facebook.WebDriverAgentRunner.xctrunner(WDA_BUNDLEIDby default): Xcode appends.xctrunnerto the bundle id of the runner, so a build with another one (--bundle-id <id>ofScripts/zebrunner-build.sh,PRODUCT_BUNDLE_IDENTIFIER) needsWDA_BUNDLEID: <id>.xctrunner(orwda_bundleidof the device). -
The iOS simulators of macOS take the unsigned simulator build (
WDA_SIMULATOR_FILEorwda_fileof the device, a.appfolder or its.zip):WebDriverAgentRunner-Build-Sim-arm64.zipof the appium/WebDriverAgent releases (upstream, without the changes of the fork) orbash Scripts/zebrunner-build.sh --sdk simof the fork.
The connector mounts the Developer Disk Image of the iOS version of the device before WebDriverAgent starts. The deploy of a host with iOS devices (./zebrunner.sh ansible, the download role) clones the images of Xcode of zebrunner/DeveloperDiskImage (shallow and sparse: the images the connector mounts only, without the Cryptex ones; about 0.5 GB per host) and updates them on every deploy; zebrunner-farm mounts the folder read-only into the connectors of the iOS devices but the simulators (/opt/zebrunner/ddi, DDI_DIR of mcloud-ios-connector). Without the folder, or without the image of the device in it, the connector downloads the image with go-ios as before (ios image auto, the internet).
| Setting | Default | What it is |
|---|---|---|
DDI_ENABLED |
true |
false: no clone and no mount, the connector downloads the images |
DDI_REPO |
https://github.com/zebrunner/DeveloperDiskImage.git |
the repository of the images, e.g. a mirror of the network of the host |
DDI_VERSION |
main |
its branch or tag |
DDI_DIR |
DeveloperDiskImages of the agent folder (out of git) |
the folder of the clone, an absolute path of its own: every deploy resets it to DDI_VERSION |
- A host with Android devices only clones nothing. A failed clone or update (no access to the repository) is reported (
The Developer Disk Images ... are not updated) and does not stop the deploy: the connectors mount the earlier clone or download the images. - A new folder is mounted into the connectors created after it:
zebrunner-farm restart iosrecreates them. - Do not change the clone: the next deploy drops a local change of it.
- Open in your browser
http://<PUBLIC_IP>/stf, authenticate yourself based on preconfigured auth system. - The connected device should be available in STF.
- Disconnect the device from the server. Device containers will be removed asap, then, in 15-30 sec, the device should change the state in STF to disconnected as well.
- Use different commands from
zebrunner-farm start/stop/restart/down/statusto manage all devices, a device by udid or name, or all devices of a platform (ios/android).Run
zebrunner-farmto see available options.
| Symptom | Cause and fix |
|---|---|
./zebrunner.sh ansible fails on Pull Zebrunner images and Check that images failed to pull exist locally with denied: Your authorization token has expired or 403 Forbidden |
an expired login to public.ecr.aws in ~/.docker/config.json of the user who runs the deploy (it pulls as this user, not as root): the images are public, docker logout public.ecr.aws (or log in again) |
Linux: ./zebrunner.sh ansible fails on Pull Zebrunner images with permission denied while trying to connect to the Docker daemon socket |
the user is not in the docker group: sudo usermod -aG docker $USER, log in again |
the pull fails with manifest unknown |
the tag of versions.yml (or of config/settings.yml) is not published, see the images of the current mcloud |
| Linux: the deploy passed, the devices plugged in before it have no containers | udev starts them on the next connection only: zebrunner-farm start |
./zebrunner.sh join stops with the TLS key of the server is not the pinned one |
the join string is of another MCloud (or the address of the join string leads elsewhere), or a man in the middle; a new TLS key of MCloud needs a new join string |
./zebrunner.sh join stops with MCloud refused the enrollment (HTTP 403) |
the invite is used, expired, replaced by a newer one of the label, or of another label: a new ./zebrunner.sh node-invite on the MCloud server |
| the containers are healthy, the devices are not in STF | ./zebrunner.sh doctor: MCloud does not know or revoked this host (a new invite and ./zebrunner.sh join), its address in MCloud is not STF_PROVIDER_HOST, the devices are not bound to it (the containers created before join: ./zebrunner.sh ansible, zebrunner-farm restart); the STF triproxies log Refused the ZeroMQ client ... with the unknown key for a key MCloud does not know, see ZeroMQ security |
the devices are in STF, not in the grid; Appium logs An attempt to register with the grid <hub URL> was unsuccessful |
the reason is in the log of Appium (device-<name>-<udid>-appium) or of mcloud-grid: is not reachable from the hub - STF_PROVIDER_HOST and appium_port are not reachable from the MCloud server (in its network the address of this host there, see Device hosts in a private network); an unknown or revoked node token, or a node url whose host is not the address of the invite - ./zebrunner.sh doctor, see Grid registration |
| the screenshots or the app installs of the devices fail | MCloud refuses the token of this host (./zebrunner.sh doctor), or PUBLIC_IP is not reachable from this host (e.g. the public address of a server of the same network behind a NAT without hairpin: resolve it to its address in the network) |
no video of the sessions, the devices have no -uploader container |
no S3_BUCKET in config/settings.yml: appium records no video and the devices get no uploader, see Recordings |
macOS: an iOS simulator is not in STF, the log of its -stf container stops after connecting to WdaMjpeg |
the amd64 mcloud-device under emulation: the arm64 one of a multi-platform mcloud-device image is needed on Apple Silicon (docker image inspect <image> --format '{{.Architecture}}') |
Run ./zebrunner.sh doctor first, see Doctor. Follow the below algorithm to identify any configuration issues with MCloud agent:
-
macOS: the devices listener logs into
logs/listener.logandlogs/listener.err.log, the usbmuxd sharing (socat) intologs/usbmuxd-socat.logandlogs/usbmuxd-socat.err.logof the MCloud agent directory.listener.logis rotated by newsyslog (10 MB, 7 bzip2 archives,/etc/newsyslog.d/zebrunner-mcloud.conf). Loaded agents are shown bylaunchctl list | grep zebrunner. -
Linux: enable the debug log level for udev rules:
sudo udevadm control --log-priority=debug. -
Linux: the connections and disconnections of a USB device and the output of
zebrunner-farmfor them are in the journal of its units:journalctl -f -u 'mcloud-device-*@<id>' -
If there are no updates during connection/disconnection, please, verify the correctness of:
- device udid values,
- presence of
/usr/local/bin/zebrunner-farmand of its settings/usr/local/etc/zebrunner-farm.env, - correctness of
/usr/local/bin/mcloud-devices.txtand/etc/udev/rules.d/90_mcloud.rulesfiles.
-
Read carefully the
zebrunner-farmoutput in syslog to identify the exact failure during containers creation. -
Analyze device container logs if the status is not
healthy:docker ps -a | grep device # device connector container (adb for Android, usbmuxd and WebDriverAgent for iOS): docker logs -f device-<Name>-<udid>-connector # appium container: docker logs -f device-<Name>-<udid>-appium # STF provider container: docker logs -f device-<Name>-<udid>-stf # artifacts uploader container: docker logs -f device-<Name>-<udid>-uploader
-
If you have any problems running ansible:
- Make sure you have sudo access and try to run ansible with sudo permissions.
- Try to run ansible commands manually:
agent.ymldeploys the role of the OS of the host (devicesof Linux,mac-devicesof macOS);devices.ymlandmac-devices.ymlof the earlier commands import it
To download the required components and register the devices:
ansible-playbook -vvv --ask-become-pass -i hosts agent.yml
To deploy without the download of the images (the role of the OS only):
ansible-playbook -vvv --ask-become-pass -i hosts agent.yml --tag registerDevices
Ubuntu 25.10+ has sudo-rs as
sudo: ansible does not recognize its password prompt and stops with "Timed out waiting for become success or become password prompt"../zebrunner.sh ansibleusessudo.ws(the original sudo,sudo apt install sudo) then, add--extra-vars ansible_become_exe=sudo.wsto the commands above.To run it non-interactively, keep the sudo password in an ansible-vault encrypted file instead of the command line (use
--vault-password-fileinstead of--ask-vault-passfor automation):ansible-vault create become.yml # with the line: ansible_become_password: PSWD ansible-playbook -vvv -i hosts --extra-vars @become.yml --ask-vault-pass agent.yml
The tests folder covers zebrunner.sh, the ansible roles and templates, zebrunner-farm, the macOS devices listener and the usbmuxd_watch binary. docker, sudo, launchctl, usbmuxd and other host tools are replaced by stubs, so the tests change nothing on the host.
-
Run all linters and tests on this machine, then in the Linux docker image of
tests/Dockerfile(macOS specific tests are skipped on Linux):make check
The linters of
tests/requirements-lint.txtare installed into.venvautomatically. On macOS install hadolint withbrew install hadolint: the hadolint-py wheel for macOS is broken. -
Run a part of the checks with
make lint,make testormake docker,makelists the targets. -
tests/golden/farm-linux.txtandtests/golden/farm-macos.txtare thedockerandlaunchctlcalls and the output ofzebrunner-farmfor every kind of device (tests/farm_golden_test.sh): a change of the containers fails the test. After an intended change regenerate them and review their diff in the pull request:UPDATE_GOLDEN=1 tests/farm_golden_test.sh
-
The roles:
devices(Linux) andmac-devices(macOS) deploy the agent of the OS;common, their dependency (meta/main.yml), has the values by default and the checks of the settings and the devices shared by both (roles/common/tasks/checks.yml, imported after the check of the OS; the checks of the devices of both OS areroles/common/files/devices.py, run by the deploy with the python of ansible and by./zebrunner.sh doctor), andzebrunner-farm(roles/common/tasks/farm.yml): one static script of both OS (roles/common/files/zebrunner-farm, the branches ofFARM_OS) with its settings rendered into an env file (roles/common/templates/zebrunner-farm.env); the tests run it withZEBRUNNER_FARM_ENV;downloadpulls the images and clones the Developer Disk Images of the iOS devices (also withcommonas its dependency). -
With
JUNIT_DIRset,tests/lint.shand every test write JUnit XML into it. CI publishes the results as annotations and a job summary, and keeps the XML files as artifacts. -
CI runs the tests on Linux and macOS on every push; a pull request runs them on macOS only when it changes a file the macOS tests use: all but the docs, the settings of the linters,
roles/devicesandtests/Dockerfile(the list of the jobchangesof.github/workflows/ci.yml). -
The
usbmuxd_watchbinary of themac-devicesrole is built fromroles/mac-devices/files/usbmuxd_watch.cand the pinned release tarballs (sha256) of libplist, libimobiledevice-glue and libusbmuxd. After a change of the source or of a library version, rebuild it on macOS with the Xcode command line tools and commit it. CI builds it the same way in its macOS tests job and runstests/usbmuxd_watch_test.shon the result too; it does not compare the result with the committed binary, which another Xcode builds differently:scripts/build_usbmuxd_watch.sh
The conventions of the style, the formatting and the tooling of this repository are in CONVENTIONS.md, the same in all the MCloud repositories.
