Device state machines

Understand how Moler devices use state machines to manage connections, and how CONNECTION_HOPS config drives automatic state transitions.

Every Moler device is a state machine. The state machine tracks which shell or environment the device is currently operating in and handles all transitions between those environments automatically. Your test code just says “go to this state” — Moler figures out the sequence of commands needed to get there.

Device states

States represent distinct environments on (or leading to) a device. The built-in states in the Unix device hierarchy are:

State	Meaning
NOT_CONNECTED	Initial state; no connection established yet
UNIX_LOCAL	Connected to the local shell on the test host
PROXY_PC	On a jump host / proxy machine (optional)
UNIX_REMOTE	SSH’d into the remote target machine
UNIX_REMOTE_ROOT	Elevated to root on the remote machine via su
UNIX_LOCAL_ROOT	Root on the local machine

When a device is created (via DeviceFactory.get_device()), Moler automatically opens the connection and drives the state machine from NOT_CONNECTED to the device’s initial state (UNIX_LOCAL for a local device, or UNIX_REMOTE if initial_state is configured that way).

22:30:19.747  MyMachine        |Connection to: 'MyMachine' has been opened.
22:30:19.748  MyMachine        |Changed state from 'NOT_CONNECTED' into 'UNIX_LOCAL'

CONNECTION_HOPS: defining transitions in config

The state machine’s transitions are entirely defined in the YAML configuration file under CONNECTION_HOPS. This is the key design decision in Moler: client code never knows how to reach a state — only the config file does.

Here is the RebexTestMachine config from the Moler README:

DEVICES:
  RebexTestMachine:
    DEVICE_CLASS: moler.device.unixremote.UnixRemote
    CONNECTION_HOPS:
      UNIX_LOCAL:          # from state
        UNIX_REMOTE:       # to state
          execute_command: ssh      # via this command
          command_params:           # with these parameters
            expected_prompt: demo@
            host: test.rebex.net
            login: demo
            password: password
            set_timeout: None

This says: “To get from UNIX_LOCAL to UNIX_REMOTE, run the ssh command with these parameters.”

You can change the host, credentials, or even the command used to reach a state by editing the YAML file alone — no Python changes required. This is useful for testing against different environments.

Calling goto_state()

To move the device to a target state, call goto_state():

from moler.config import load_config
from moler.device import DeviceFactory

load_config(config='my_devices.yml')

remote_unix = DeviceFactory.get_device(name='RebexTestMachine')
remote_unix.goto_state(state="UNIX_REMOTE")

Moler resolves the full path from the current state to the target state. If the device is in UNIX_LOCAL, it knows it must run the ssh command to reach UNIX_REMOTE. If the device is in NOT_CONNECTED, it knows it must first connect (reach UNIX_LOCAL) and then SSH.

Multi-hop state transitions

Moler can traverse multiple hops automatically. The internal _state_hops configuration defines intermediate states. For example, the UnixRemote device knows:

• NOT_CONNECTED → UNIX_REMOTE: go via UNIX_LOCAL first (connect), then SSH.
• UNIX_REMOTE → NOT_CONNECTED: go via PROXY_PC (or UNIX_LOCAL) first, then disconnect.

You never specify these intermediate steps — just the destination:

# From NOT_CONNECTED, Moler will:
# 1. open connection → UNIX_LOCAL
# 2. run ssh         → UNIX_REMOTE
remote_unix.goto_state(state="UNIX_REMOTE")

Full example: go remote, run ls, inspect result

from moler.config import load_config
from moler.device import DeviceFactory

load_config(config='my_devices.yml')

remote_unix = DeviceFactory.get_device(name='RebexTestMachine')
remote_unix.goto_state(state="UNIX_REMOTE")  # drives state machine

ls_cmd = remote_unix.get_cmd(cmd_name="ls", cmd_params={"options": "-l"})
remote_files = ls_cmd()

if 'readme.txt' in remote_files['files']:
    print("readme.txt file:")
    readme_file_info = remote_files['files']['readme.txt']
    for attr in readme_file_info:
        print("  {:<18}: {}".format(attr, readme_file_info[attr]))

readme.txt file:
  permissions       : -rw-------
  hard_links_count  : 1
  owner             : demo
  group             : users
  size_raw          : 403
  size_bytes        : 403
  date              : Apr 08  2014
  name              : readme.txt

The device log (moler.RebexTestMachine.log) records every step of the state transition:

22:30:19.901  |Changed state from 'NOT_CONNECTED' into 'UNIX_LOCAL'
22:30:19.921  |Command 'moler.cmd.unix.ssh.Ssh':'TERM=xterm-mono ssh -l demo test.rebex.net' started.
22:30:19.921 >|TERM=xterm-mono ssh -l demo test.rebex.net
22:30:20.762 <|Password:
22:30:20.763 >|*********
22:30:20.908 <|Welcome to Rebex Virtual Shell!
              |demo@ETNA:/$
22:30:20.909  |Changed state from 'UNIX_LOCAL' into 'UNIX_REMOTE'
22:30:20.917  |Command 'moler.cmd.unix.ssh.Ssh' finished.
22:30:20.920  |Command 'moler.cmd.unix.ls.Ls':'ls -l' started.

Auto-reconnect after dropped connection

One of the most valuable properties of the state machine is automatic recovery. If the connection to a device drops unexpectedly (network blip, session timeout, etc.), Moler detects the unexpected state change and can re-drive the state machine back to the desired state.

The device has a timeout_keep_state property (default: 10 seconds) that controls how long Moler waits before attempting to restore the previous state after an unexpected transition. This means that transient disconnections are handled automatically without any test code changes.

|Changed state from 'UNIX_REMOTE' into 'NOT_CONNECTED'
|Connection to: 'RebexTestMachine' dropped, re-establishing...
|Changed state from 'NOT_CONNECTED' into 'UNIX_LOCAL'
|Changed state from 'UNIX_LOCAL' into 'UNIX_REMOTE'

Configuring a device without a proxy

For direct SSH (no jump host), the CONNECTION_HOPS is simpler:

DEVICES:
  MyServer:
    DEVICE_CLASS: moler.device.unixremote.UnixRemote
    CONNECTION_HOPS:
      UNIX_LOCAL:
        UNIX_REMOTE:
          execute_command: ssh
          command_params:
            expected_prompt: "user@myserver"
            host: 192.168.1.100
            login: user
            password: secret

Configuring a device with a proxy / jump host

For environments with a jump host (proxy), add an additional hop via PROXY_PC:

DEVICES:
  TargetHost:
    DEVICE_CLASS: moler.device.unixremote.UnixRemote
    CONNECTION_HOPS:
      UNIX_LOCAL:
        PROXY_PC:
          execute_command: ssh
          command_params:
            expected_prompt: user@jumphost
            host: jumphost.example.com
            login: user
            password: jumppass
      PROXY_PC:
        UNIX_REMOTE:
          execute_command: ssh
          command_params:
            expected_prompt: user@target
            host: 10.0.0.50
            login: user
            password: targetpass

With this configuration, goto_state(state="UNIX_REMOTE") automatically SSH’s to the jump host first, then SSH’s from there to the target — all without any changes to test code.

The DEVICE_CLASS field determines which state machine implementation is used. moler.device.unixremote.UnixRemote provides the full UNIX_LOCAL → PROXY_PC → UNIX_REMOTE → UNIX_REMOTE_ROOT hierarchy. moler.device.unixlocal.UnixLocal provides only NOT_CONNECTED and UNIX_LOCAL.