Concurrency in Network Automation - How to Use Concurrent Futures with Netmiko

Recently I published a blog post on how to standardize the VLAN names across 100+ switches. One of the comments I received on the post was that if the script is connecting to 100+ switches, why not use concurrency so, the script will run much faster? I tweaked the code a bit and added concurrent.futures module support.

I highly recommend checking the original post here beforehand - https://www.packetswitch.co.uk/using-python-to-standardize/

It took just under four seconds to run the script without concurrency. Let's configure concurrency and see how long will it take.

suresh@mac:~/Documents/vlan_standard|⇒  python no_concurrent.py

Connecting to 10.10.20.18
vlan 10
switch-01(config-vlan)#name MGMT
switch-01(config-vlan)#end
switch-01#
Disconnecting from 10.10.20.18

Connecting to 10.10.20.19
vlan 32
switch-02(config-vlan)#name admin
switch-02(config-vlan)#end
switch-02#
Disconnecting from 10.10.20.19

Connecting to 10.10.20.20
vlan 33
switch-03(config-vlan)#name network
switch-03(config-vlan)#end
switch-03#
Disconnecting from 10.10.20.20

Start time: 2023-03-23 00:14:38
End time: 2023-03-23 00:14:42

Time taken: 3.67 seconds

Adding Concurrent Option

I only have three switches in my lab so, going to tell the script to connect to a maximum of five switches at a time (Since we only have three switches, the script will just connect to all of them at once)

I've also included the time module to calculate how long it takes to run the code.

import concurrent.futures
import time
from netmiko import ConnectHandler

start_time = time.time()

switch_list = ['10.10.20.18', '10.10.20.19', '10.10.20.20']
device_inventroy = []
vlans = {
    '10': 'MGMT',
    '20': 'DATA',
    '30': 'active_directory',
    '31': 'web_servers',
    '32': 'admin',
    '33': 'network'
}

for ip in switch_list:
    device = {
        "device_type": "cisco_ios",
        "host": ip,
        "username": "cisco",
        "password": 'cisco123',
        "secret": 'cisco123'  # Enable password
    }
    device_inventroy.append(device)

def configure_switch(switch):
    print(f"Connecting to {switch['host']}")
    connection = ConnectHandler(**switch)
    output = connection.send_command('show vlan', use_textfsm=True)
    connection.enable()  # Enable method
    connection.config_mode()  # Global config mode

    current_vlans_dict = {}
    for vlan in output:
        current_vlans_dict[vlan['vlan_id']] = vlan['name']

    for k, v in vlans.items():
        if k in current_vlans_dict and v != current_vlans_dict[k]:
            commands = [f"vlan {k}", f"name {v}"]
            config_output = connection.send_config_set(commands)
            print(config_output)

    print(f"Disconnecting from {switch['host']}")
    connection.disconnect()

# Use ThreadPoolExecutor with a max_workers of 5
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    executor.map(configure_switch, device_inventroy)

end_time = time.time()
elapsed_time = end_time - start_time

print(f"Start time: {time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(start_time))}")
print(f"End time: {time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(end_time))}")
print(f"Time taken: {elapsed_time:.2f} seconds")

Code Breakdown

Import the necessary libraries

• concurrent.futures: to handle concurrent execution of tasks.
• time: to measure the time taken by the script.
• netmiko.ConnectHandler: to establish a connection with the devices.

Record the start time using time.time()

Create a list of switch IP addresses switch_list, an empty list for device inventory device_inventroy, and a dictionary of VLAN IDs and their names vlans

Iterate through the switch_list and create a dictionary containing device information (device_type, IP address, username, password, and enable password). Append the device dictionary to device_inventroy.

Define a function configure_switch(switch) to manage the VLAN configuration on the switches:

• Print the IP address of the switch being connected to.
• Establish a connection to the switch using ConnectHandler class
• Retrieve the current VLAN configuration using the show vlan command and parse the output using TextFSM.
• Enable privileged mode and enter the global configuration mode.
• Create a dictionary of the current VLANs on the switch that includes VLAN-ID and Name.
• Iterate through the VLANs in the vlans dictionary:
• If the VLAN ID exists and the name does not match the current configuration, update the VLAN name by sending the appropriate commands.
• Print the switch's IP address upon disconnecting.

Use a ThreadPoolExecutor with a maximum of 5 concurrent workers to configure VLANs on the devices in the device_inventroy list concurrently. The configure_switch function is applied to each device using the executor.map() method.

Record the end time using time.time() and calculate the elapsed time. Finally, Print the start time, end time, and the time taken by the script to complete.

Concurrent Futures Explained

In the provided code, concurrent.futures is used to handle the parallel execution of tasks. It provides a high-level interface for asynchronously executing functions or methods. The main components of concurrent.futures used in the code are ThreadPoolExecutor and executor.map().

ThreadPoolExecutor is a class within concurrent.futures that creates a pool of worker threads for parallel execution. When you create a ThreadPoolExecutor instance, you can specify the maximum number of worker threads that can run concurrently using the max_workers parameter. In the given code, max_workers is set to 5, meaning up to 5 tasks can be executed in parallel.

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:

The with statement is used here to create a context manager, which ensures that resources are properly acquired and released when the block of code is executed.

executor.map() is a method provided by ThreadPoolExecutor that takes a function and an iterable (e.g., list, tuple, etc.) as arguments. It applies the function to each item in the iterable and returns an iterator over the results. The execution of the function is distributed across the worker threads in the ThreadPoolExecutor, allowing for parallel execution of tasks.

In the code, executor.map() is used to apply the configure_switch function to each switch in the device_inventroy list. This allows the configuration of switches to be executed in parallel, with up to 5 switches being configured at once. This can save a lot of time if you are working with a larger number of devices.

executor.map(configure_switch, device_inventroy)

In summary, the concurrent.futures part of the code creates a thread pool with a maximum of 5 worker threads and then applies the configure_switch function to each switch in the device_inventroy list in parallel, allowing for faster and more efficient execution of the script.

The execution took just over 1 second, unbelievable.

suresh@mac:~/Documents/vlan_standard|⇒  python concurrent_test.py

Connecting to 10.10.20.18
Connecting to 10.10.20.19
Connecting to 10.10.20.20

vlan 10
switch-01(config-vlan)#name MGMT
switch-01(config-vlan)#end
switch-01#
Disconnecting from 10.10.20.18

vlan 32
switch-02(config-vlan)#name admin
switch-02(config-vlan)#end
switch-02#
Disconnecting from 10.10.20.19

vlan 33
switch-03(config-vlan)#name network
switch-03(config-vlan)#end
switch-03#
Disconnecting from 10.10.20.20

Start time: 2023-03-22 23:55:28
End time: 2023-03-22 23:55:29

Time taken: 1.46 seconds

Conclusion

In summary, the concurrent.futures part of the code creates a thread pool with a maximum of 5 worker threads and then applies the configure_switch function to each switch in the device_inventroy list in parallel, allowing for faster and more efficient execution of the script.