How to use TextFSM with Netmiko

In this blog post, we will go through the steps required to get structured output from network devices using TextFSM, Netmiko, and ntc_templates and how they work together to streamline your Network Automation tasks.

If you are not familiar with Netmiko, please check out my previous blog posts here.

Part 1 - Introduction to Netmiko

Part - 2 Netmiko with Multiple Devices

Why Structured data is important for Network Automation?

When you run CLI commands on network devices, typically the output presented to you is just a string of data (semi-structured data ). For example, if you look at the output of show interfaces status command, the output contains rows and rows of strings. Suppose, you want to identify all the interfaces that are down and change the description on all of those specific ports, what would you do? How can you perform an action on just a subset of items?

The answer is simply converting the semi-structured data to a structured format using tools such as TextFSM.

💡

Please note that you don't necessarily need to use TextFSM to convert semi-structured data to structured data. You can also use Python's native regex but it can lead into some complications.

What is TextFSM?

TextFSM is a Python module which implements a template-based state machine for parsing semi-formatted text. It was originally developed to allow programmatic access to the information returned from the CLI of networking devices. I will try to cover TextFSM in more detail in an upcoming post.

TextFSM takes two inputs - a template file, and a text file (such as CLI outputs) and returns a list of records that contains the data parsed from the text. Now imagine that there are hundreds of show commands available for Cisco and there are many network vendors out there. It would be impossible for an individual to create a template for each and every command.

Well, the awesome people from Network to Code have already done the heavy lifting and created a repository to hold the templates. There are hundreds of templates available for multiple vendors such as Arista, Cisco, Palo Alto and many more. Anyone can contribute to the repo by adding the templates here https://github.com/networktocode/ntc-templates

Example

To demonstrate the power of TextFSM, let's go through an example. Our ultimate goal here is to identify the interfaces that have a status notconnect and configure a description on them.

Cisco IOS 'show command' output

As I mentioned before, when you run a show command on a Cisco device, you just get a string of data as shown below.

sw-01#show interfaces status 

Port      Name               Status       Vlan       Duplex  Speed Type 
Gi0/0                        connected    10         a-full   auto RJ45
Gi0/1                        connected    1          a-full   auto RJ45
Gi0/2                        notconnect   1          a-full   auto RJ45
Gi0/3                        connected    1          a-full   auto RJ45
Gi1/0                        notconnect   1          a-full   auto RJ45
Gi1/1                        notconnect   1          a-full   auto RJ45
Gi1/2                        connected    1          a-full   auto RJ45
Gi1/3                        connected    1          a-full   auto RJ45

You also get the exact same output when using netmiko without any parsing.

from netmiko import ConnectHandler

sw_01 = {
    "device_type": "cisco_ios",
    "host": "10.10.20.12",
    "username": "cisco",
    "password": "Cisco123",
    "secret": 'Cisco123'
}

connection = ConnectHandler(**sw_01)

output = connection.send_command('show interfaces status')
print(output)

connection.disconnect()

sureshv@mac:~/Documents/netmiko-textfsm|⇒  python netmiko-ios.py

Port      Name               Status       Vlan       Duplex  Speed Type 
Gi0/0                        connected    10         a-full   auto RJ45
Gi0/1                        connected    1          a-full   auto RJ45
Gi0/2                        notconnect   1          a-full   auto RJ45
Gi0/3                        connected    1          a-full   auto RJ45
Gi1/0                        notconnect   1          a-full   auto RJ45
Gi1/1                        notconnect   1          a-full   auto RJ45
Gi1/2                        connected    1          a-full   auto RJ45
Gi1/3                        connected    1          a-full   auto RJ45

How to get structured data?

By using a simple argument use_textfsm=True (line #14), netmiko will return structured data using the great TextFSM collection ntc_templates

If you want to learn more about what's going on behind the scenes, I highly recommend reading their documentation here https://pyneng.readthedocs.io/en/latest/book/21_textfsm/textfsm_clitable.html

from netmiko import ConnectHandler
import json

sw_01 = {
    "device_type": "cisco_ios",
    "host": "10.10.20.12",
    "username": "cisco",
    "password": "Cisco123",
    "secret": 'Cisco123'
}

connection = ConnectHandler(**sw_01)

output = connection.send_command('show interfaces status', use_textfsm=True)

connection.disconnect()

print(json.dumps(output, indent = 4))

Just like that, we get this awesome list/dictionary combination that we can easily manipulate.

sureshv@mac:~/Documents/netmiko-textfsm|⇒  python netmiko-ios.py
[
    {
        "port": "Gi0/0",
        "name": "",
        "status": "connected",
        "vlan": "10",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi0/1",
        "name": "",
        "status": "connected",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi0/2",
        "name": "",
        "status": "notconnect",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi0/3",
        "name": "",
        "status": "connected",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi1/0",
        "name": "",
        "status": "notconnect",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi1/1",
        "name": "",
        "status": "notconnect",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi1/2",
        "name": "",
        "status": "connected",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    },
    {
        "port": "Gi1/3",
        "name": "",
        "status": "connected",
        "vlan": "1",
        "duplex": "a-full",
        "speed": "auto",
        "type": "RJ45",
        "fc_mode": ""
    }
]

Find interfaces that are not connected

Let's say we want to find all the interfaces that have the status notconnect and change the description on them, how do we find them? Well, it is much easier than you may think. All you need to do is, write a simple if/else statement.

In a nutshell, we are saying, if the interface status is notconnect give me the port numbersI'm using Python list comprehension to find the ports and add them to a new list called not_connect_interfaces

from netmiko import ConnectHandler
import json

sw_01 = {
    "device_type": "cisco_ios",
    "host": "10.10.20.12",
    "username": "cisco",
    "password": "Cisco123",
    "secret": 'Cisco123'
}

connection = ConnectHandler(**sw_01)

output = connection.send_command('show interfaces status', use_textfsm=True)

connection.disconnect()

not_connect_interfaces = [item['port'] for item in output if item['status'] == 'notconnect' ]

print(not_connect_interfaces)

sureshv@mac:~/Documents/netmiko-textfsm|⇒  python netmiko-ios.py
['Gi0/2', 'Gi1/0', 'Gi1/1']

As you can see above, we have three interfaces that have the status of notconnect

Configuring the Description

Now that we have the port numbers, we need to tell netmiko to go into those interfaces and change the description.

from netmiko import ConnectHandler
import json

sw_01 = {
    "device_type": "cisco_ios",
    "host": "10.10.20.12",
    "username": "cisco",
    "password": "Cisco123",
    "secret": 'Cisco123'
}

connection = ConnectHandler(**sw_01)
output = connection.send_command('show interfaces status', use_textfsm=True)

not_connect_interfaces = [item['port'] for item in output if item['status'] == 'notconnect' ]

connection.enable() # Enable method
connection.config_mode() # Global config mode

for interface in not_connect_interfaces:
    commands = [f"interface {interface}", 'description NOT-CONNECTED']
    config_output = connection.send_config_set(commands)
    print(config_output)

connection.disconnect()

Let's run the script and look at the output. As expected, netmiko logins to the switch, switch to global-config mode, go into each interface and change the description as shown below.

sureshv@mac:~/Documents/netmiko-textfsm|⇒  python netmiko-ios.py
interface Gi0/2
sw-01(config-if)#description NOT-CONNECTED
sw-01(config-if)#end
sw-01#
configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
sw-01(config)#interface Gi1/0
sw-01(config-if)#description NOT-CONNECTED
sw-01(config-if)#end
sw-01#
configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
sw-01(config)#interface Gi1/1
sw-01(config-if)#description NOT-CONNECTED
sw-01(config-if)#end
sw-01#

We can also login to the switch and verify it.

sw-01#show interfaces description 
Interface                      Status         Protocol Description
Gi0/0                          up             up       
Gi0/1                          up             up       
Gi0/2                          down           down     NOT-CONNECTED
Gi0/3                          up             up       
Gi1/0                          down           down     NOT-CONNECTED
Gi1/1                          down           down     NOT-CONNECTED
Gi1/2                          up             up       
Gi1/3                          up             up       
Vl10                           up             up

Closing thoughts

I have to say that I'm using TextFSM a lot nowadays with my automation tasks. Please let me know in the comments if you use it regularly too.

Reference

https://pynet.twb-tech.com/blog/netmiko-and-textfsm.html

https://github.com/networktocode/ntc-templates

https://github.com/google/textfsm