Python pexpect library

Pexpect is a pure Python module for spawning child applications, controlling them, and responding to expected patterns in their output. Pexpect works like Don Libes’ Expect. Pexpect allows our script to spawn a child application and control it as if a human were typing commands; more information can be found on Pexpect’s documentation page: https://pexpect.readthedocs.io/en/stable/index.html.

Nowadays, we typically use libraries, such as Nornir, that abstract this line-by-line, low-level interaction. However, it is still useful to understand the interaction at least at a high level. If you are the impatient kind, just skim through the following Pexpect and Paramiko sections.

Similar to the original Tool Command Language (TCL) Expect module by Don Libes, Pexpect launches, or spawns, another process and watches over it in order to control the interaction. The Expect tool was originally developed to automate interactive processes such as FTP, Telnet, and rlogin, and was later expanded to include network automation. Unlike the original Expect, Pexpect is entirely written in Python, which does not require TCL or C extensions to be compiled. This allows us to use the familiar Python syntax and its rich standard library in our code.

Pexpect installation

The Pexpect installation process is straightforward:

(venv) $ pip install pexpect

Let’s do a quick test to make sure the package is usable; make sure we start the Python interactive shell from the virtual environment:

(venv) $ python
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pexpect
>>> dir(pexpect)
['EOF', 'ExceptionPexpect', 'Expecter', 'PY3', 'TIMEOUT', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__revision__', '__spec__', '__version__', 'exceptions', 'expect', 'is_executable_file', 'pty_spawn', 'run', 'runu', 'searcher_re', 'searcher_string', 'spawn', 'spawnbase',
 'spawnu', 'split_command_line', 'sys', 'utils', 'which']
    >>> exit()

Pexpect overview

For this chapter, we will use the 2_DC_Topology and work on the two IOSv devices, lax-edg-r1 and lax-edg-r2:

Figure 2.6: lax-edg-r1 and lax-edg-r2

The devices will each have a management address in the 192.16.2.x/24 range. In the example, lax-edg-r1 will have 192.168.2.51 and lax-edg-r2 will have 192.168.2.52 as the management IP. If this is the first time the device is powered up, it will need to generate an RSA key for SSH:

lax-edg-r2(config)#crypto key generate rsa

For older IOSv software images, we might also need to add the following lines to the ssh configuration (~/.ssh/config) depending on your platform:

Host 192.168.2.51
  HostKeyAlgorithms +ssh-rsa
  KexAlgorithms +diffie-hellman-group-exchange-sha1
Host 192.168.2.52
  HostKeyAlgorithms +ssh-rsa
  KexAlgorithms +diffie-hellman-group-exchange-sha1

With the devices ready, let’s take a look at how you would interact with the router if you were to telnet into the device:

(venv) $ $ telnet 192.168.2.51
Trying 192.168.2.51...
Connected to 192.168.2.51.
Escape character is '^]'.
<skip>
User Access Verification
Username: cisco
Password:

The device configuration uses the username of cisco, and the password is also cisco. Notice that the user is already in the privileged mode because of the privilege assigned in the configuration:

lax-edg-r1#sh run | i cisco
enable password cisco
username cisco privilege 15 secret 5 $1$SXY7$Hk6z8OmtloIzFpyw6as2G.
 password cisco
 password cisco

The auto-config also generated vty access for both telnet and SSH:

line con 0
 password cisco
line aux 0
line vty 0 4
 exec-timeout 720 0
 password cisco
 login local
 transport input telnet ssh

Let’s see a Pexpect example using the Python interactive shell:

>>> import pexpect
>>> child = pexpect.spawn('telnet 192.168.2.51')
>>> child.expect('Username')
0
>>> child.sendline('cisco')
6
>>> child.expect('Password')
0
>>> child.sendline('cisco')
6
>>> child.expect('lax-edg-r1#')
0
>>> child.sendline('show version | i V')
19
>>> child.before
b": \r\n************************************************************************\r\n* IOSv is strictly limited to use for evaluation, demonstration and IOS  *\r\n* education. IOSv is provided as-is and is not supported by Cisco's      *\r\n* Technical Advisory Center. Any use or disclosure, in whole or in part, *\r\n* of the IOSv Software or Documentation to any third party for any       *\r\n* purposes is expressly prohibited except as otherwise authorized by     *\r\n* Cisco in writing.                                                      *\r\n***********************************************************************\r\n"
>>> child.sendline('exit')
5
>>> exit()

Starting from Pexpect version 4.0, you can run Pexpect on the Windows platform. But, as noted in the Pexpect documentation, running Pexpect on Windows should be considered experimental for now.

In the previous interactive example, Pexpect spawns off a child process and watches over it in an interactive fashion. There are two important methods shown in the example, expect() and sendline(). The expect() line indicates the string in the Pexpect process looks for when the returned string is considered done. This is the expected pattern. In our example, we knew the router had sent us all the information when the hostname prompt (lax-edg-r1#) was returned. The sendline() method indicates which words should be sent to the remote device as the command. There is also a method called send(), but sendline() includes a linefeed, which is similar to pressing the Enter key at the end of the words you sent in the previous telnet session. From the router’s perspective, it is just as if someone typed in the text from a Terminal. In other words, we are tricking the routers into thinking they are interfacing with a human being when they are actually communicating with a computer.

The before and after properties will be set to the text printed by the child application. The before properties will be set to the text printed by the child application up to the expected pattern. The after string will contain the text that was matched by the expected pattern. In our case, the before text will be set to the output between the two expected matches (lax-edg-r1#), including the show version command. The after text is the router hostname prompt:

>>> child.sendline('show version | i V')
19
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.6(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 9Y0KJ2ZL98EQQVUED5T2Q\r\n'
>>> child.after
b'iosv-1#'

If you are wondering about the b' in front of the return, it is a Python byte string (https://docs.python.org/3.10/library/stdtypes.html).

What would happen if you expected the wrong term? For example, if we typed in username with the lowercase “u” instead of Username after spawning the child application, the Pexpect process would look for a string of username from the child process. In that case, the Pexpect process would just hang because the word username would never be returned by the router. The session would eventually time out, or we could manually exit out via Ctrl + C.

The expect() method waits for the child application to return a given string, so in the previous example, if you wanted to accommodate both lowercase and uppercase u, you could use the following term:

>>> child.expect('[Uu]sername')

The square bracket serves as an or operation that tells the child application to expect a lowercase or uppercase “u” followed by sername as the string. What we are telling the process is that we will accept either Username or username as the expected string. For more information on these different types of matching using a regular expression, go to: https://docs.python.org/3.10/library/re.html.

The expect() method can also contain a list of options instead of just a single string; these options can also be regular expressions themselves. Going back to the previous example, we can use the following list of options to accommodate the two different possible strings:

>>> child.expect(['Username', 'username'])

Generally speaking, use the regular expression for a single expect string when we can fit the different letters in a regular expression, whereas use the possible options if we need to catch completely different responses from the device, such as a password rejection. For example, if we use several different passwords for our login, we want to catch % Login invalid as well as the device prompt.

One important difference between Pexpect regular expressions and Python regular expressions is that Pexpect matching is non-greedy, which means they will match as little as possible when using special characters. Because Pexpect performs regular expressions on a stream, it cannot look ahead, as the child process generating the stream may not be finished. This means the special dollar sign character $ typically matching the end of the line is useless because .+ will always return no characters, and the .* pattern will match as little as possible. In general, just keep this in mind and be as specific as you can be on the expect match strings.

Let’s consider the following scenario:

>>> child.sendline('show run | i hostname')
22
>>> child.expect('lax-edg-r1')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.6(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 9Y0KJ2ZL98EQQVUED5T2Q\r\n'
>>>

Hmm... Something is not quite right here. Compare it to the Terminal output before; the output you expect would be hostname lax-edg-r1:

iosv-1#sh run | i hostname
hostname lax-edg-r1

Taking a closer look at the expected string will reveal the mistake. In this case, we were missing the hash (#) sign behind the lax-edg-r1 hostname. Therefore, the child application treated the second part of the return string as the expected string:

>>> child.sendline('show run | i hostname')
22
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'#show run | i hostname\r\nhostname lax-edg-r1\r\n'

You can see a pattern emerging from the usage of Pexpect after a few examples. The user maps out the sequence of interactions between the Pexpect process and the child application. With some Python variables and loops, we can start to construct a useful program that will help us gather information and make changes to network devices.

Our first Pexpect program

Our first program, chapter2_1.py, extends what we did in the last section with some additional code:

#!/usr/bin/env python
import pexpect
devices = {'iosv-1': {'prompt': 'lax-edg-r1#', 'ip': '192.168.2.51'},
           'iosv-2': {'prompt': 'lax-edg-r2#', 'ip': '192.168.2.52'}}
username = 'cisco'
password = 'cisco'
for device in devices.keys():
    device_prompt = devices[device]['prompt']
    child = pexpect.spawn('telnet ' + devices[device]['ip'])
    child.expect('Username:')
    child.sendline(username)
    child.expect('Password:')
    child.sendline(password)
    child.expect(device_prompt)
    child.sendline('show version | i V')
    child.expect(device_prompt)
    print(child.before)
    child.sendline('exit')

We used a nested dictionary in line 5:

devices = {'iosv-1': {'prompt': 'lax-edg-r1#', 'ip': '192.168.2.51'},
           'iosv-2': {'prompt': 'lax-edg-r2#', 'ip': '192.168.2.52'}}

The nested dictionary allows us to refer to the same device (such as lax-edg-r1) with the appropriate IP address and prompt symbol. We can then use those values for the expect() method later on in the loop.

The output prints out the show version | i V output on the screen for each of the devices:

$ python chapter2_1.py 
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\n'
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\n'

Now that we have seen a basic example of Pexpect, let us go deeper into more features of the library.

More Pexpect Features

In this section, we will look at more Pexpect features that might come in handy when certain situations arise.

If you have a slow or fast link to your remote device, the default expect() method timeout is 30 seconds, which can be increased or decreased via the timeout argument:

>>> child.expect('Username', timeout=5)

You can choose to pass the command back to the user using the interact() method. This is useful when you just want to automate certain parts of the initial task:

>>> child.sendline('show version | i V')
19
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-
-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\n'
>>> child.interact()
show version | i V
Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)
Processor board ID 98U40DKV403INHIULHYHB
lax-edg-r1#exit
Connection closed by foreign host.
>>>

You can get a lot of information about the child.spawn object by printing it out in string format:

>>> str(child)
"<pexpect.pty_spawn.spawn object at 0x7f068a9bf370>\ncommand: /usr/bin/telnet\nargs: ['/usr/bin/telnet', '192.168.2.51']\nbuffer (last 100 chars): b''\nbefore (last 100 chars): b'TERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\\r\\nProcessor board ID 98U40DKV403INHIULHYHB\\r\\n'\nafter: b'lax-edg-r1#'\nmatch: <re.Match object; span=(165, 176), match=b'lax-edg-r1#'>\nmatch_index: 0\nexitstatus: 1\nflag_eof: False\npid: 25510\nchild_fd: 5\nclosed: False\ntimeout: 30\ndelimiter: <class 'pexpect.exceptions.EOF'>\nlogfile: None\nlogfile_read: None\nlogfile_send: None\nmaxread: 2000\nignorecase: False\nsearchwindowsize: None\ndelaybeforesend: 0.05\ndelayafterclose: 0.1\ndelayafterterminate: 0.1"
>>>

The most useful debug tool for Pexpect is to log the output in a file:

>>> child = pexpect.spawn('telnet 192.168.2.51')
>>> child.logfile = open('debug', 'wb')

For more information on Pexpect features, check out: https://pexpect.readthedocs.io/en/stable/api/index.html

We have been working with Telnet so far in our examples, which leaves our communication in clear text during the session. In modern networks, we typically use secure shell (SSH) for management. In the next section, we will take a look at Pexpect with SSH.

Pexpect and SSH

Pexpect has a subclass called pxssh, which specializes in setting up SSH connections. The class adds methods for login, logout, and various tricky things to handle the different situations in the ssh login process. The procedures are mostly the same, with the exception of login() and logout():

>>> from pexpect import pxssh
>>> child = pxssh.pxssh()
>>> child.login('192.168.2.51', 'cisco', 'cisco', auto_prompt_reset=False) 
True
>>> child.sendline('show version | i V')
19
>>> child.expect('lax-edg-r1#')
0
>>> child.before
b'show version | i V\r\nCisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.8(3)M2, RELEASE SOFTWARE (fc2)\r\nProcessor board ID 98U40DKV403INHIULHYHB\r\n'
>>> child.logout()
>>>

Notice the auto_prompt_reset=False argument in the login() method. By default, pxssh uses the shell prompt to synchronize the output. But since it uses the PS1 option for most of bash-shell or c-shell, they will error out on Cisco or other network devices.

Pexpect complete example

As the final step, let’s put everything you have learned so far about Pexpect into a script. Putting code into a script makes it easier to use in a production environment, as well as easier to share with your colleagues. We will write our second script, chapter2_2.py:

#!/usr/bin/env python
import getpass
from pexpect import pxssh
devices = {'lax-edg-r1': {'prompt': 'lax-edg-r1#', 'ip': '192.168.2.51'},
           'lax-edg-r2': {'prompt': 'lax-edg-r2#', 'ip': '192.168.2.52'}}
commands = ['term length 0', 'show version', 'show run']
username = input('Username: ')
password = getpass.getpass('Password: ')
# Starts the loop for devices
for device in devices.keys():
    outputFileName = device + '_output.txt'
    device_prompt = devices[device]['prompt']
    child = pxssh.pxssh()
    child.login(devices[device]['ip'], username.strip(), password.strip(), auto_prompt_reset=False)
    # Starts the loop for commands and write to output
    with open(outputFileName, 'wb') as f:
        for command in commands:
            child.sendline(command)
            child.expect(device_prompt)
            f.write(child.before)
    child.logout()

The script further expands from our first Pexpect program with the following additional features:

• It uses SSH instead of Telnet.
• It supports multiple commands instead of just one by making the commands into a list (line 8) and loops through the commands (starting at line 20).
• It prompts the user for their username and password instead of hardcoding them in the script for better security posture.
• It writes the output in two files, lax-edg-r1_output.txt and lax-edg-r2_output.txt.

After the code is executed, we should see the two output files in the same directory. Besides Pexpect, Paramiko is another popular Python library used to handle interactive sessions.