In this blog post, we will walk you through how to clean up Palo Alto Firewall Objects and Rules using a simple Python script. The script is designed to search for a specific IP address or an entire subnet and remove any associated references.
The Problem
Have you ever found yourself in a situation where you've decommissioned a server or maybe even an entire subnet, and now you're faced with the task of cleaning up your firewall? If you're using Palo Alto, you probably know that you can't just remove an address object; you first need to eliminate all its references from address groups and rules.
This can become especially cumbersome if a single object is referenced in multiple places—you'll have to remove them one by one. Now, imagine having to do this for an entire subnet where multiple objects are involved. If this sounds familiar, read on to find out how to make this process easier using a simple Python Script.
If you are looking for a more sophisticated solution, feel free to check my other blog post on how to achieve this via the 'pan-os-php' library.
Suresh Vina
Prerequisites and Cautions
This script is designed to interact with Panorama, if you intend to run this script directly against individual firewalls, modifications to the code will be required.
The script performs changes but does not automatically commit or push these to the managed devices. You will need to manually commit these changes through the Panorama GUI.
Please exercise extreme caution when using this script. Ensure that you verify that only the intended configurations are being removed. Use this script at your own risk.
What does the Script do?
Upon running the script, it connects to the specified Panorama and carries out the following steps for each subnet or host in the provided list.
- Address Objects Processing: The script searches for and removes any address objects that refer to the decommissioned subnet or host. If an address object is a FQDN, it does a DNS lookup to resolve the IP address.
- Address Groups Processing: The script looks for any address groups that contain members referencing the decommissioned subnet or host, and removes these members from the group. If a group becomes empty, it's flagged for removal.
- Security Rules Processing: The script scans all security rules for any source or destination references to the decommissioned subnet or host, or to any of the flagged address groups. These references are then removed. If a rule has no members left in either the source or destination, the rule is deleted.
- Cleanup: After all rules have been processed, any empty address groups and address objects are deleted from the Panorama.
Challenges and Order of Operations
The order of operations is crucial when deleting resources due to the interconnected nature of objects, address-groups, and rules in Palo Alto firewalls. This script follows a systematic approach to handle this complexity:
- Address-Groups Processing: The first step is to remove objects from address-groups. This operation poses a challenge. For instance, if an address-group contains two address objects, and we attempt to remove both, Palo Alto will objects because an address-group can't be left empty. To fix this issue, we add such address-groups to a Python list and track them. However, we can't remove these address-groups yet because they may still be referenced in the policies.
- Rules Processing: The second step involves removing objects and object groups from the rules. Similar to the previous step, if a rule contains a single source/destination object and we try to remove that, Palo Alto will object because the source/destination field can't be left empty. In such cases, we add the entire rule to a removal list and then remove them entirely later.
- Processing All Device Groups: Before we start removing the objects/groups, it is necessary to process all the device groups. Because the object can be referenced in different device-groups.
- Handling Nested Address Groups: Another challenge to note is nested address groups. Before removing an address group, we need to check if the group is part of another address-group. If so, we must carefully handle this nested structure to prevent disruptions.
All these challenges underline the importance of adopting a careful and systematic approach when deleting or modifying resources in Palo Alto firewalls. Our script respects these complexities and constraints, ensuring the efficient and safe cleanup of the specified subnets or hosts.
The Script
In this section, we will delve into the nuts and bolts of the script that automates the cleanup process for Palo Alto Firewalls. The project is split into two separate Python files. The first script, paloaltoapi.py, contains the RestAPI classes and methods that handle communication with the Palo Alto device. The second script, main.py, imports these classes and methods to execute the main logic that performs the cleanup of address objects, address groups, and firewall rules.
If you want to learn more about OOP Python, please check out my other blog post below.
Suresh Vina
To clone from my GitHub, you can use this link.
paloaltoapi.py
import requests
import json
import xml.etree.ElementTree as ET
class PaloAltoAPI:
def __init__(self, base_url, username, password):
self.base_url = base_url
self.username = username
self.password = password
requests.packages.urllib3.disable_warnings()
def get_key(self):
query = {'type':'keygen', 'user':self.username, 'password':self.password}
response = requests.get(f"{self.base_url}/api", params=query, verify=False)
root = ET.fromstring(response.text)
api_key = root.find(".//key").text
return {'X-PAN-KEY': api_key}
def build_url(self, endpoint, device_group, name=None):
location = {'location': 'device-group', 'device-group': device_group}
if device_group == 'shared':
location = {'location': 'shared'}
if name is not None:
location['name'] = name
return f"{self.base_url}/restapi/v10.2/{endpoint}", location
def get_resource(self, endpoint, device_group):
url, location = self.build_url(endpoint, device_group)
headers = self.get_key()
response = requests.get(url, params=location, verify=False, headers=headers)
return response.json()['result']['entry']
def create_resource(self, endpoint, device_group, payload, name):
url, location = self.build_url(endpoint, device_group, name)
headers = self.get_key()
response = requests.post(url, params=location, verify=False, headers=headers, data=payload)
return response.text
def edit_resource(self, endpoint, device_group, payload, name):
url, location = self.build_url(endpoint, device_group, name)
headers = self.get_key()
response = requests.put(url, params=location, verify=False, headers=headers, data=payload)
return response.text
def delete_resource(self, endpoint, device_group, name):
url, location = self.build_url(endpoint, device_group, name)
headers = self.get_key()
response = requests.delete(url, params=location, verify=False, headers=headers)
return response.textpaloaltoapi.py
The first script, paloaltoapi.py, serves as the backbone for API interactions with the Palo Alto device. It contains a class named PaloAltoAPI that encapsulates all the necessary functionalities for communicating with the firewall. The class is responsible for a variety of actions, including authentication, building API endpoints, and performing CRUD (Create, Read, Update, Delete) operations on firewall resources.
To handle these tasks, methods like get_key, build_url, get_resource, create_resource, edit_resource, and delete_resource are defined within the class. Overall, this script abstracts the complex API interactions, making it easier to execute higher-level logic in the main script.
main.py
import os
from paloaltoapi import PaloAltoAPI
import ipaddress
import json
import socket
import copy
device_groups = {
"branch_offices": "PreRules",
"head_office": "PreRules"
}
def is_valid_ip(ip_str):
try:
ipaddress.ip_network(ip_str, strict=False)
return True
except:
return False
def process_addr_objects(address_obj, subnet):
try:
if 'ip-netmask' in address_obj and ipaddress.ip_network(address_obj['ip-netmask'], strict=False).subnet_of(subnet):
return True
elif 'fqdn' in address_obj and ipaddress.ip_network(socket.gethostbyname(address_obj["fqdn"]), strict=False).subnet_of(subnet):
return True
except (socket.gaierror, TypeError):
pass
return False
hosts = ['192.168.10.15/32', '192.168.20.0/24'] # Please add the required subnet/host (use '/32' for host)
pan_object = PaloAltoAPI('https://my-panorama.company.local', os.environ.get('username'), os.environ.get('password'))
addresses = pan_object.get_resource("Objects/Addresses", 'shared')
address_groups = pan_object.get_resource("Objects/AddressGroups", 'shared')
objects_to_romove = []
groups_to_remove = []
print('Connected to Panorama')
print('===========================================')
for host in hosts:
print(f"Searching for {host}\n")
subnet = ipaddress.ip_network(host)
# Process Address Objects
for object in addresses:
if process_addr_objects(object, subnet):
objects_to_romove.append(object['@name'])
# Process Address Groups
for group in address_groups:
new_members = copy.deepcopy(group['static']['member'])
for member in group['static']['member']:
address_object = next((object for object in addresses if object["@name"] == member), None)
if address_object:
if process_addr_objects(address_object, subnet):
new_members.remove(member)
print(f"Removing {address_object['@name']} from {group['@name']}")
if group['static']['member'] != new_members:
if not new_members:
groups_to_remove.append(group['@name'])
print()
else:
group['static']['member'] = new_members
data = json.dumps(
{
"entry": group
}
)
edit_group = pan_object.edit_resource("Objects/AddressGroups", 'shared', data, group['@name'])
print(f"{edit_group}\n")
groups_to_remove = list(set(groups_to_remove))
# Process Rules
for dg, rule_base in device_groups.items():
rules_to_remove = []
rules = pan_object.get_resource(f"Policies/Security{rule_base}", dg)
for rule in rules:
original_rule = copy.deepcopy(rule)
for list_key in ['source', 'destination']:
new_members = copy.deepcopy(rule[list_key]['member'])
for address in rule[list_key]['member']:
if address in groups_to_remove:
new_members.remove(address)
print(f"Removing {address} from {rule['@name']}")
elif address == 'any':
continue
else:
valid_ip = is_valid_ip(address)
if valid_ip:
try:
if ipaddress.ip_network(address, strict=False).subnet_of(subnet):
new_members.remove(address)
print(f"Removing {address} from {rule['@name']}")
except TypeError:
pass
else:
address_obj = next((object for object in addresses if object["@name"] == address), None)
if address_obj:
if process_addr_objects(address_obj, subnet):
new_members.remove(address)
objects_to_romove.append(address)
print(f"Removing {address} from {rule['@name']}")
if new_members:
rule[list_key]['member'] = new_members
else:
rules_to_remove.append(rule['@name'])
print()
if original_rule != rule:
data = json.dumps(
{
"entry": rule
}
)
edit_rules = pan_object.edit_resource(f"Policies/Security{rule_base}", dg, data, rule['@name'])
print(f"{edit_rules}\n")
# Remove Empty Rules
for rule in rules_to_remove:
print(f"Deleting empty rule - {rule}")
delete_rules = pan_object.delete_resource(f"Policies/Security{rule_base}", dg, rule)
print(f"{delete_rules}\n")
print(f"Objects to Remove - {list(set(objects_to_romove))}")
print(f"Groups to remove - {groups_to_remove}")
print()
# Check for Nested Groups
if groups_to_remove:
for group in address_groups:
new_members = copy.deepcopy(group['static']['member'])
for member in group['static']['member']:
if member in groups_to_remove:
print(f"Removing {member} from {group['@name']} ")
new_members.remove(member)
if group['static']['member'] != new_members:
group['static']['member'] = new_members
data = json.dumps(
{
"entry": group
}
)
edit_group = pan_object.edit_resource("Objects/AddressGroups", 'shared', data, group['@name'])
print(f"{edit_group}\n")
# Remove Empty Groups
for group in groups_to_remove:
print(f"Deleting Group - {group}")
delete_groups = pan_object.delete_resource("Objects/AddressGroups", 'shared', group)
print(delete_groups)
print()
# Remove Objects
for object in list(set(objects_to_romove)):
print(f"Deleting Object - {object}")
delete_objects = pan_object.delete_resource("Objects/Addresses", 'shared', object)
print(delete_objects)
print()main.py
Initialization
First, all the necessary Python libraries and the custom class PaloAltoAPI are imported. Once that's done, a PaloAltoAPI object is initialized by passing in the Panorama URL and the credentials, which are fetched from environment variables. This object is then used to make initial API calls to fetch existing address objects and address groups that are part of the Shared device group.
Device Groups and Rule Base
A dictionary called device_groups is set up to map the names of device groups to their associated rule bases. You can configure this to use either PreRules or PostRules, allowing the script to work flexibly with your particular setup.
IP Validation
A utility function named is_valid_ip is defined to check if a string is a valid IP address or subnet. This function is later used when processing security rules to validate the IPs.
Address Object Processing
Another function named process_addr_objects is also defined. Its job is to take an address object and a subnet as parameters and then determine if the object belongs to the subnet.
Main Loop
The script then enters its main loop, iterating over each host or subnet specified in a list called hosts. Inside this loop, multiple actions are performed in a specific sequence:
- Address Objects: For each subnet or host in the
hostslist, the script goes through all the existing address objects. Using theprocess_addr_objectsfunction, it checks if each object falls under the subnet/host being processed. Any matching objects are marked for removal. - Address Groups: Next, the script checks all address groups. If a group contains members that have been marked for removal, those members are removed from the group. Entire groups that end up being empty are flagged for removal.
- Security Rules: After that, the script processes security rules. For each device group, both the source and destination fields in each rule are checked. Any references to the flagged address groups or address objects are removed. If a rule ends up having no references in either the source or destination, that rule is flagged for removal.
Clean-Up Actions
Finally, once all the rules have been processed, the script proceeds to the clean-up phase.
- Removing empty address groups
- Deleting flagged address objects
- Deleting any security rules that are now empty
Closing Up
So there you have it, folks! We've walked through how you can make your life a bit easier by automating some tedious Palo Alto firewall cleanup tasks. As with any automation, please double-check your work before committing any changes. I'd love to hear your experiences or any suggestions for improvement. Feedback is always welcome.
