In this blog post, we will go through the steps required to get structured output from network devices using TextFSM, Netmiko, and ntc_templates and how they work together to streamline your Network Automation tasks.
If you are not familiar with Netmiko, please check out my previous blog posts here.
Why Structured data is important for Network Automation?
When you run CLI commands on network devices, typically the output presented to you is just a string of data (semi-structured data ). For example, if you look at the output of show interfaces status command, the output contains rows and rows of strings. Suppose, you want to identify all the interfaces that are down and change the description on all of those specific ports, what would you do? How can you perform an action on just a subset of items?
The answer is simply converting the semi-structured data to a structured format using tools such as TextFSM.
regex but it can lead into some complications. What is TextFSM?
TextFSM is a Python module which implements a template-based state machine for parsing semi-formatted text. It was originally developed to allow programmatic access to the information returned from the CLI of networking devices. I will try to cover TextFSM in more detail in an upcoming post.
TextFSM takes two inputs - a template file, and a text file (such as CLI outputs) and returns a list of records that contains the data parsed from the text. Now imagine that there are hundreds of show commands available for Cisco and there are many network vendors out there. It would be impossible for an individual to create a template for each and every command.
Well, the awesome people from Network to Code have already done the heavy lifting and created a repository to hold the templates. There are hundreds of templates available for multiple vendors such as Arista, Cisco, Palo Alto and many more. Anyone can contribute to the repo by adding the templates here https://github.com/networktocode/ntc-templates
Example
To demonstrate the power of TextFSM, let's go through an example. Our ultimate goal here is to identify the interfaces that have a status notconnect and configure a description on them.
Cisco IOS 'show command' output
As I mentioned before, when you run a show command on a Cisco device, you just get a string of data as shown below.
sw-01#show interfaces status
Port Name Status Vlan Duplex Speed Type
Gi0/0 connected 10 a-full auto RJ45
Gi0/1 connected 1 a-full auto RJ45
Gi0/2 notconnect 1 a-full auto RJ45
Gi0/3 connected 1 a-full auto RJ45
Gi1/0 notconnect 1 a-full auto RJ45
Gi1/1 notconnect 1 a-full auto RJ45
Gi1/2 connected 1 a-full auto RJ45
Gi1/3 connected 1 a-full auto RJ45You also get the exact same output when using netmiko without any parsing.
from netmiko import ConnectHandler
sw_01 = {
"device_type": "cisco_ios",
"host": "10.10.20.12",
"username": "cisco",
"password": "Cisco123",
"secret": 'Cisco123'
}
connection = ConnectHandler(**sw_01)
output = connection.send_command('show interfaces status')
print(output)
connection.disconnect()sureshv@mac:~/Documents/netmiko-textfsm|⇒ python netmiko-ios.py
Port Name Status Vlan Duplex Speed Type
Gi0/0 connected 10 a-full auto RJ45
Gi0/1 connected 1 a-full auto RJ45
Gi0/2 notconnect 1 a-full auto RJ45
Gi0/3 connected 1 a-full auto RJ45
Gi1/0 notconnect 1 a-full auto RJ45
Gi1/1 notconnect 1 a-full auto RJ45
Gi1/2 connected 1 a-full auto RJ45
Gi1/3 connected 1 a-full auto RJ45How to get structured data?
By using a simple argument use_textfsm=True (line #14), netmiko will return structured data using the great TextFSM collection ntc_templates
If you want to learn more about what's going on behind the scenes, I highly recommend reading their documentation here https://pyneng.readthedocs.io/en/latest/book/21_textfsm/textfsm_clitable.html
from netmiko import ConnectHandler
import json
sw_01 = {
"device_type": "cisco_ios",
"host": "10.10.20.12",
"username": "cisco",
"password": "Cisco123",
"secret": 'Cisco123'
}
connection = ConnectHandler(**sw_01)
output = connection.send_command('show interfaces status', use_textfsm=True)
connection.disconnect()
print(json.dumps(output, indent = 4))Just like that, we get this awesome list/dictionary combination that we can easily manipulate.
sureshv@mac:~/Documents/netmiko-textfsm|⇒ python netmiko-ios.py
[
{
"port": "Gi0/0",
"name": "",
"status": "connected",
"vlan": "10",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi0/1",
"name": "",
"status": "connected",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi0/2",
"name": "",
"status": "notconnect",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi0/3",
"name": "",
"status": "connected",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi1/0",
"name": "",
"status": "notconnect",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi1/1",
"name": "",
"status": "notconnect",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi1/2",
"name": "",
"status": "connected",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
},
{
"port": "Gi1/3",
"name": "",
"status": "connected",
"vlan": "1",
"duplex": "a-full",
"speed": "auto",
"type": "RJ45",
"fc_mode": ""
}
]Find interfaces that are not connected
Let's say we want to find all the interfaces that have the status notconnect and change the description on them, how do we find them? Well, it is much easier than you may think. All you need to do is, write a simple if/else statement.
In a nutshell, we are saying, if the interface status is notconnect give me the port numbersI'm using Python list comprehension to find the ports and add them to a new list called not_connect_interfaces
from netmiko import ConnectHandler
import json
sw_01 = {
"device_type": "cisco_ios",
"host": "10.10.20.12",
"username": "cisco",
"password": "Cisco123",
"secret": 'Cisco123'
}
connection = ConnectHandler(**sw_01)
output = connection.send_command('show interfaces status', use_textfsm=True)
connection.disconnect()
not_connect_interfaces = [item['port'] for item in output if item['status'] == 'notconnect' ]
print(not_connect_interfaces)sureshv@mac:~/Documents/netmiko-textfsm|⇒ python netmiko-ios.py
['Gi0/2', 'Gi1/0', 'Gi1/1']As you can see above, we have three interfaces that have the status of notconnect
Configuring the Description
Now that we have the port numbers, we need to tell netmiko to go into those interfaces and change the description.
from netmiko import ConnectHandler
import json
sw_01 = {
"device_type": "cisco_ios",
"host": "10.10.20.12",
"username": "cisco",
"password": "Cisco123",
"secret": 'Cisco123'
}
connection = ConnectHandler(**sw_01)
output = connection.send_command('show interfaces status', use_textfsm=True)
not_connect_interfaces = [item['port'] for item in output if item['status'] == 'notconnect' ]
connection.enable() # Enable method
connection.config_mode() # Global config mode
for interface in not_connect_interfaces:
commands = [f"interface {interface}", 'description NOT-CONNECTED']
config_output = connection.send_config_set(commands)
print(config_output)
connection.disconnect()Let's run the script and look at the output. As expected, netmiko logins to the switch, switch to global-config mode, go into each interface and change the description as shown below.
sureshv@mac:~/Documents/netmiko-textfsm|⇒ python netmiko-ios.py
interface Gi0/2
sw-01(config-if)#description NOT-CONNECTED
sw-01(config-if)#end
sw-01#
configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
sw-01(config)#interface Gi1/0
sw-01(config-if)#description NOT-CONNECTED
sw-01(config-if)#end
sw-01#
configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
sw-01(config)#interface Gi1/1
sw-01(config-if)#description NOT-CONNECTED
sw-01(config-if)#end
sw-01#We can also login to the switch and verify it.
sw-01#show interfaces description
Interface Status Protocol Description
Gi0/0 up up
Gi0/1 up up
Gi0/2 down down NOT-CONNECTED
Gi0/3 up up
Gi1/0 down down NOT-CONNECTED
Gi1/1 down down NOT-CONNECTED
Gi1/2 up up
Gi1/3 up up
Vl10 up up Closing thoughts
I have to say that I'm using TextFSM a lot nowadays with my automation tasks. Please let me know in the comments if you use it regularly too.
Reference
https://pynet.twb-tech.com/blog/netmiko-and-textfsm.html
https://github.com/networktocode/ntc-templates
https://github.com/google/textfsm
