expect Command in Linux

Introduction

In this comprehensive guide, you'll discover how to leverage the Linux expect utility to automate interactive command-line applications. The expect command is a versatile systemadmin automation tool that allows you to create scripts capable of interacting with programs that demand user input, such as SSH, FTP, and other interactive utilities.

By the conclusion of this tutorial, you will:

  • Grasp the purpose and fundamental syntax of the expect command
  • Develop scripts to automate SSH logins and streamline server access
  • Manage diverse prompts and responses within your expect scripts

The expect command can dramatically minimize manual intervention for recurring tasks, enhancing system administration and making routine tasks more efficient. You will begin by installing and exploring the basic syntax of expect, and then move towards developing scripts for automating SSH logins and managing various interactive prompts.

Understanding the expect Command and Its Basic Syntax

The expect command in Linux empowers you to automate interactive command-line programs that typically require user input. This is extremely valuable for tasks like automated logins, file transfers, or any situation where a program prompts for input.

Installing expect

Firstly, let us confirm that the expect package is installed on your system. Launch your terminal and execute:

which expect

If expect is already installed, its path (e.g., /usr/bin/expect) will be displayed. If not, installation is required:

sudo apt-get update
sudo apt-get install -y expect

Understanding Basic expect Syntax

The expect command utilizes a scripting language built upon Tcl (Tool Command Language). The core structure of an expect script comprises the following commands:

  1. spawn: Initiates a process for interaction
  2. expect: Awaits specific output from the spawned process
  3. send: Transmits input to the spawned process
  4. set timeout: Defines the waiting duration for expected output

Let's construct a basic expect script to illustrate these concepts. Open a text editor and create a file named hello.exp in your project directory:

cd ~/project
nano hello.exp

Insert the following content into the file:

#!/usr/bin/expect -f

## Set a timeout of 10 seconds
set timeout 10

## Spawn the bash process
spawn bash

## Wait for the bash prompt
expect "$ "

## Send a command to the bash process
send "echo Hello from expect\r"

## Wait for the bash prompt again
expect "$ "

## Exit the bash session
send "exit\r"

## Wait for the process to complete
expect eof

Save the file by pressing Ctrl+O, followed by Enter, and exit nano using Ctrl+X.

Make the script executable:

chmod +x ~/project/hello.exp

Execute the script now:

~/project/hello.exp

The output should resemble this:

spawn bash
$ echo Hello from expect
Hello from expect
$ exit
exit

Understanding Each Line of the Script

Let me detail the function of each line in the script:

  • #!/usr/bin/expect -f: This shebang line instructs the system to utilize the expect interpreter to execute the script.
  • set timeout 10: This establishes a 10-second timeout for subsequent expect commands.
  • spawn bash: This initiates a new bash shell process for expect to interact with.
  • expect "$ ": This awaits the appearance of the bash prompt.
  • send "echo Hello from expect\r": This transmits the command to the bash shell. The \r at the end simulates pressing Enter.
  • expect "$ ": This again awaits the bash prompt after command execution.
  • send "exit\r": This dispatches the exit command to close the bash shell.
  • expect eof: This waits for the spawned process to terminate.

This simple example showcases the fundamental functionality of expect. In the following steps, we'll employ these concepts to create more practical scripts.

Creating a Mock SSH Login Script with expect

In this section, we will create an expect script that simulates an SSH login process. As we cannot perform a real SSH login in this environment, we will build a mock script that demonstrates the principles.

Understanding SSH Authentication Flow

When connecting to a remote server via SSH, the typical interaction includes:

  1. Initiating the connection with ssh username@hostname
  2. Accepting the host key (if connecting for the first time)
  3. Entering your password when prompted
  4. Gaining access to the remote shell

Let's create a mock SSH environment to illustrate how expect can automate this process.

Creating a Mock SSH Server Script

First, let's create a script that simulates an SSH server prompting for a password:

cd ~/project
nano mock_ssh_server.sh

Enter the following content:

#!/bin/bash

echo "The authenticity of host 'mockserver' can't be established."
echo "RSA key fingerprint is SHA256:abcdefghijklmnopqrstuvwxyz123456."
echo "Are you sure you want to continue connecting (yes/no)? "
read answer

if [ "$answer" != "yes" ]; then
  echo "Host key verification failed."
  exit 1
fi

echo "Warning: Permanently added 'mockserver' (RSA) to the list of known hosts."
echo "Password: "
read -s password

if [ "$password" == "mockpassword" ]; then
  echo "Last login: Wed Nov 1 12:00:00 2023 from 192.168.1.100"
  echo "Welcome to Mock SSH Server"
  echo "mockuser@mockserver:~$ "

  while true; do
    read -p "" command
    if [ "$command" == "exit" ]; then
      echo "Connection to mockserver closed."
      exit 0
    else
      echo "Executing: $command"
      echo "mockuser@mockserver:~$ "
    fi
  done
else
  echo "Permission denied, please try again."
  exit 1
fi

Save the file and make it executable:

chmod +x ~/project/mock_ssh_server.sh

This script simulates:

  • The SSH host verification prompt
  • The password prompt
  • A simple shell that responds to commands

Creating an expect Script to Automate the Login

Now, let's create an expect script that automates interaction with our mock SSH server:

cd ~/project
nano ssh_login.exp

Enter the following content:

#!/usr/bin/expect -f

## Set variables
set timeout 10
set password "mockpassword"

## Start the mock SSH server
spawn ./mock_ssh_server.sh

## Handle the host verification prompt
expect "Are you sure you want to continue connecting (yes/no)? "
send "yes\r"

## Handle the password prompt
expect "Password: "
send "$password\r"

## Wait for the shell prompt
expect "mockuser@mockserver:~$ "

## Execute a command
send "ls -la\r"
expect "mockuser@mockserver:~$ "

## Exit the session
send "exit\r"

## Wait for the process to complete
expect eof

puts "\nSSH login automation completed successfully!"

Save the file and make it executable:

chmod +x ~/project/ssh_login.exp

Running the Automated Login Script

Now, let's run our expect script to automate interaction with the mock SSH server:

cd ~/project
./ssh_login.exp

You should see output similar to this:

spawn ./mock_ssh_server.sh
The authenticity of host 'mockserver' can't be established.
RSA key fingerprint is SHA256:abcdefghijklmnopqrstuvwxyz123456.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mockserver' (RSA) to the list of known hosts.
Password:
Last login: Wed Nov 1 12:00:00 2023 from 192.168.1.100
Welcome to Mock SSH Server
mockuser@mockserver:~$ ls -la
Executing: ls -la
mockuser@mockserver:~$ exit
Connection to mockserver closed.

SSH login automation completed successfully!

Explaining the Script

Let me explain the purpose of each part of our expect script:

  1. set timeout 10: Sets a global timeout of 10 seconds for all expect commands.
  2. set password "mockpassword": Stores the password in a variable.
  3. spawn ./mock_ssh_server.sh: Starts our mock SSH server script.
  4. expect "Are you sure you want to continue connecting (yes/no)? ": Waits for the host verification prompt.
  5. send "yes\r": Sends "yes" to accept the host key.
  6. expect "Password: ": Waits for the password prompt.
  7. send "$password\r": Sends the password.
  8. expect "mockuser@mockserver:~$ ": Waits for the shell prompt.
  9. send "ls -la\r": Sends a command to list files.
  10. expect "mockuser@mockserver:~$ ": Waits for the shell prompt again.
  11. send "exit\r": Sends the exit command to close the session.
  12. expect eof: Waits for the process to terminate.
  13. puts "\nSSH login automation completed successfully!": Prints a success message.

This example shows how expect can be used to automate the entire SSH login process, from accepting the host key to executing commands on the remote server and safely exiting.

Handling Multiple Prompts and Responses with expect

In real-world scenarios, interactive programs frequently present multiple prompts and may require different responses based on different conditions. In this section, we'll learn how to handle multiple prompts and conditional responses in expect scripts.

Understanding Conditional Expect Blocks

The expect command can be used with a pattern-action block structure to handle different possible prompts:

expect {
    "pattern1" { actions for pattern1 }
    "pattern2" { actions for pattern2 }
    timeout { actions for timeout }
    eof { actions for end of file }
}

This structure allows your script to respond differently depending on the output it receives.

Creating a Multi-Prompt Handling Script

Let's create a script that simulates a program with multiple prompts:

cd ~/project
nano multi_prompt.sh

Enter the following content:

#!/bin/bash

echo "Please select an option:"
echo "1. Show date and time"
echo "2. List files"
echo "3. Show system info"
echo "4. Exit"
echo -n "Enter your choice (1-4): "
read choice

case $choice in
  1)
    echo "Current date and time:"
    date
    ;;
  2)
    echo "File listing:"
    ls -la
    ;;
  3)
    echo "System information:"
    uname -a
    ;;
  4)
    echo "Exiting program..."
    exit 0
    ;;
  *)
    echo "Invalid option. Please enter a number between 1 and 4."
    exit 1
    ;;
esac

echo "Do you want to continue? (yes/no): "
read answer

if [ "$answer" == "yes" ]; then
  echo "Continuing..."
  echo "Operation completed successfully."
else
  echo "Exiting program..."
fi

Save the file and make it executable:

chmod +x ~/project/multi_prompt.sh

Now, let's create an expect script to interact with this program and handle all possible prompts:

cd ~/project
nano handle_prompts.exp

Enter the following content:

#!/usr/bin/expect -f

## Set a timeout
set timeout 10

## Start the multi-prompt program
spawn ./multi_prompt.sh

## Wait for the choice prompt
expect "Enter your choice (1-4): "

## Generate a random choice (1-3)
set choice [expr {int(rand() * 3) + 1}]
send "$choice\r"

## Process the result based on the choice
switch $choice {
    1 {
        expect "Current date and time:"
        expect "Do you want to continue? (yes/no): "
    }
    2 {
        expect "File listing:"
        expect "Do you want to continue? (yes/no): "
    }
    3 {
        expect "System information:"
        expect "Do you want to continue? (yes/no): "
    }
}

## Handle the continue prompt
expect {
    "Do you want to continue? (yes/no): " {
        ## 70% chance to say yes, 30% chance to say no
        if {rand() < 0.7} {
            send "yes\r"
            expect "Operation completed successfully."
        } else {
            send "no\r"
            expect "Exiting program..."
        }
    }
    timeout {
        puts "Timeout waiting for continue prompt"
        exit 1
    }
}

## Wait for the program to complete
expect eof

puts "\nMulti-prompt handling completed successfully!"

Save the file and make it executable:

chmod +x ~/project/handle_prompts.exp

Running the Multi-Prompt Handling Script

Now, let's run our expect script to interact with the multi-prompt program:

cd ~/project
./handle_prompts.exp

Each time you run this script, it will randomly select one of the options and randomly decide whether to continue or exit. Here is an example output:

spawn ./multi_prompt.sh
Please select an option:
1. Show date and time
2. List files
3. Show system info
4. Exit
Enter your choice (1-4): 2
File listing:
total 20
drwxr-xr-x 2 labex labex 4096 Nov  1 10:00 .
drwxr-xr-x 4 labex labex 4096 Nov  1 10:00 ..
-rwxr-xr-x 1 labex labex  345 Nov  1 10:00 handle_prompts.exp
-rwxr-xr-x 1 labex labex  578 Nov  1 10:00 multi_prompt.sh
-rwxr-xr-x 1 labex labex  221 Nov  1 10:00 ssh_login.exp
Do you want to continue? (yes/no): yes
Continuing...
Operation completed successfully.

Multi-prompt handling completed successfully!

Creating a More Advanced expect Script

Now let us create a more advanced script that can handle unexpected prompts and errors:

cd ~/project
nano advanced_expect.exp

Enter the following content:

#!/usr/bin/expect -f

## Set a timeout
set timeout 10

## Define variables
set program "./multi_prompt.sh"
set max_retries 3
set retry_count 0

## Define a procedure to handle errors
proc handle_error {} {
    global retry_count max_retries program
    incr retry_count

    if {$retry_count < $max_retries} {
        puts "\nRetrying... Attempt $retry_count of $max_retries"
        ## Start the program again
        spawn $program
        return 1
    } else {
        puts "\nMaximum retry attempts reached. Exiting."
        exit 1
    }
}

## Start the program
spawn $program

## Main interaction loop
while {$retry_count < $max_retries} {
    expect {
        "Enter your choice (1-4): " {
            send "1\r"  ## Always choose option 1 for deterministic behavior
        }
        "Invalid option" {
            puts "\nReceived invalid option message."
            if {[handle_error]} continue
        }
        "Current date and time:" {
            ## Successfully got date output
        }
        "Do you want to continue? (yes/no): " {
            send "yes\r"
        }
        "Operation completed successfully." {
            puts "\nAdvanced expect script completed successfully!"
            break
        }
        timeout {
            puts "\nTimeout occurred waiting for prompt."
            if {[handle_error]} continue
        }
        eof {
            puts "\nUnexpected end of file."
            if {[handle_error]} continue
        }
    }
}

## Wait for the program to complete
expect eof

Save the file and make it executable:

chmod +x ~/project/advanced_expect.exp

Run the advanced script:

cd ~/project
./advanced_expect.exp

Example output:

spawn ./multi_prompt.sh
Please select an option:
1. Show date and time
2. List files
3. Show system info
4. Exit
Enter your choice (1-4): 1
Current date and time:
Wed Nov  1 10:00:00 UTC 2023
Do you want to continue? (yes/no): yes
Continuing...
Operation completed successfully.

Advanced expect script completed successfully!

Understanding the Advanced Script

This advanced script demonstrates several important expect techniques:

  1. Error handling: It uses a retry mechanism to handle errors or unexpected responses.
  2. Procedures: It defines a custom procedure called handle_error for reusable error handling.
  3. Control flow: It uses a while loop to maintain the interaction until success or maximum retries.
  4. Multiple expect patterns: It handles multiple different patterns and takes appropriate actions for each.
  5. Pattern order: The order of patterns in the expect block matters - more specific patterns should come before more general ones.

These techniques can be applied to automate complex interactive programs where the flow might vary or errors might occur.

Creating Practical expect Scripts for Common Tasks

In this section, we will create practical expect scripts for common tasks that system administrators often need to automate. We will focus on file operations, user interactions, and system monitoring.

Automating File Transfer with expect

Let's create an expect script that automates the transfer of a file using the scp command. Since we cannot perform an actual file transfer in this environment, we will simulate it:

cd ~/project
nano file_transfer.sh

Enter the following content to simulate an SCP-like file transfer:

#!/bin/bash

echo "scp file transfer simulation"
echo "Source file: $1"
echo "Destination: $2"
echo "Password: "
read -s password

if [ "$password" == "transfer123" ]; then
  echo "Transferring file..."
  echo "0%"
  sleep 1
  echo "25%"
  sleep 1
  echo "50%"
  sleep 1
  echo "75%"
  sleep 1
  echo "100%"
  echo "File transfer completed successfully."
else
  echo "Authentication failed."
  exit 1
fi

Save the file and make it executable:

chmod +x ~/project/file_transfer.sh

Now, let's create an expect script to automate this file transfer:

cd ~/project
nano file_transfer.exp

Enter the following content:

#!/usr/bin/expect -f

## Set variables
set timeout 10
set source_file "local_file.txt"
set destination "user@remote:/path/to/destination/"
set password "transfer123"

## Create a dummy source file
spawn bash -c "echo 'This is a test file' > $source_file"
expect eof

## Start the file transfer simulation
spawn ./file_transfer.sh $source_file $destination

## Handle the password prompt
expect "Password: "
send "$password\r"

## Monitor the transfer progress
expect "0%"
puts "Transfer started..."

expect "25%"
puts "Transfer 1/4 complete..."

expect "50%"
puts "Transfer 1/2 complete..."

expect "75%"
puts "Transfer 3/4 complete..."

expect "100%"
puts "Transfer almost done..."

expect "File transfer completed successfully."
puts "File transfer automation completed!"

## Clean up the dummy file
spawn bash -c "rm $source_file"
expect eof

Save the file and make it executable:

chmod +x ~/project/file_transfer.exp

Run the file transfer automation script:

cd ~/project
./file_transfer.exp

Example output:

spawn bash -c echo 'This is a test file' > local_file.txt
spawn ./file_transfer.sh local_file.txt user@remote:/path/to/destination/
scp file transfer simulation
Source file: local_file.txt
Destination: user@remote:/path/to/destination/
Password:
Transferring file...
0%
Transfer started...
25%
Transfer 1/4 complete...
50%
Transfer 1/2 complete...
75%
Transfer 3/4 complete...
100%
Transfer almost done...
File transfer completed successfully.
File transfer automation completed!
spawn bash -c rm local_file.txt

Automating User Creation with expect

Now, let's create an expect script that automates user creation. Again, we will simulate this process:

cd ~/project
nano create_user.sh

Enter the following content:

#!/bin/bash

echo "User creation utility"
echo "Please enter new username: "
read username

echo "Please enter password for $username: "
read -s password

echo "Please confirm password: "
read -s password_confirm

if [ "$password" != "$password_confirm" ]; then
  echo "Error: Passwords do not match."
  exit 1
fi

echo "Creating user $username..."
echo "User $username created successfully."
echo "Do you want to add this user to the sudo group? (yes/no): "
read sudo_choice

if [ "$sudo_choice" == "yes" ]; then
  echo "Adding $username to sudo group..."
  echo "User $username added to sudo group."
fi

echo "User setup completed."

Save the file and make it executable:

chmod +x ~/project/create_user.sh

Now, let's create an expect script to automate user creation:

cd ~/project
nano create_user.exp

Enter the following content:

#!/usr/bin/expect -f

## Set variables
set timeout 10
set username "testuser"
set password "P@ssw0rd123"
set add_sudo "yes"

## Start the user creation utility
spawn ./create_user.sh

## Handle the username prompt
expect "Please enter new username: "
send "$username\r"

## Handle the password prompt
expect "Please enter password for $username: "
send "$password\r"

## Handle the password confirmation prompt
expect "Please confirm password: "
send "$password\r"

## Wait for the user creation confirmation
expect "User $username created successfully."

## Handle the sudo prompt
expect "Do you want to add this user to the sudo group? (yes/no): "
send "$add_sudo\r"

## If we chose to add to sudo, wait for confirmation
if {$add_sudo == "yes"} {
    expect "User $username added to sudo group."
}

## Wait for completion
expect "User setup completed."

puts "\nUser creation automation completed successfully!"

Save the file and make it executable:

chmod +x ~/project/create_user.exp

Run the user creation automation script:

cd ~/project
./create_user.exp

Example output:

spawn ./create_user.sh
User creation utility
Please enter new username:
testuser
Please enter password for testuser:
Please confirm password:
Creating user testuser...
User testuser created successfully.
Do you want to add this user to the sudo group? (yes/no):
yes
Adding testuser to sudo group...
User testuser added to sudo group.
User setup completed.

User creation automation completed successfully!

Understanding Practical expect Scripts

The practical scripts we created demonstrate several important concepts for real-world automation:

  1. Sequential Interaction: Both scripts follow a defined sequence of prompts and responses.
  2. Progress Monitoring: The file transfer script monitors progress and provides user-friendly updates.
  3. Conditional Logic: The user creation script uses conditional logic to handle the sudo option.
  4. Environment Setup and Cleanup: The file transfer script creates and cleans up test files.

These techniques can be applied to automate many common system administration tasks, such as:

  • Remote backups
  • Software installations
  • System configuration
  • Batch operations

By mastering expect, you can automate complex interactive processes that would otherwise require manual intervention, saving time and reducing the potential for human error.

Summary

In this lab, you have learned how to use the expect command in Linux to automate interactive command-line applications. You have gained practical experience with:

  • Installing and understanding the basic syntax of the expect command
  • Creating scripts to automate SSH logins and handle various authentication prompts
  • Handling multiple prompts and responses in expect scripts
  • Creating practical automation scripts for common system administration tasks

The expect command is a powerful tool for system administrators and developers who need to automate interactive processes. By using expect, you can eliminate the need for manual intervention in repetitive tasks, saving time and reducing the risk of human error.

Some key takeaways from this lab:

  • The expect command uses a pattern-action model to interact with programs
  • Scripts can be made more robust by handling various possible prompts and error conditions
  • Complex interactions can be automated using conditional logic and custom procedures
  • Practical automation can significantly improve efficiency for common system tasks

With the skills you have learned in this lab, you can now create your own automation scripts for various interactive command-line applications.

400+ Linux Commands