Cisco ISE URT

Cisco ISE Upgrade Readiness Tool (URT) helps detect and fix any data upgrade issues before you start the upgrade process.

Most of the upgrade failures occur because of data upgrade/corruption issues. The URT is designed to validate the data before the upgrade to identify, and report or fix the issue. If the URT fails it will create a file which can be shared with TAC for further troubleshooting.

The URT can be downloaded from the ISE download page on Cisco. The URT tool can be run on a Secondary Administration Node (PAN), for high availability and other deployments with multiple nodes, or on the Standalone Node for a single-node deployment. There is no downtime for running this tool.

In multiple-node deployments, do not run the URT on the Primary PAN.

The following prerequisites are checked by the URT tool:

Version compatibility
Persona checks
Disk space
NTP server
Memory
System and trusted certificate validation

In this example, I will show you how to run the URT on a multi-node deployment.

diagram

Step - 1 Download and copy URT to the Secondary PAN

I am upgrading from 2.3 to 2.6 so, I will use ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz file. Once downloaded, copy the file to the secondary PAN.

I am using SFTP to transfer files into Cisco ISE.

There are many ways you can copy files into ISE but I find using SFTP server is very easy and straightforward.

1. First, upload the files to the SFTP server from your laptop.

sftp> put ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz

Uploading ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz to /home/suresh/ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz

ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.   7%   38MB   1.0MB/s   07:16 ETA

2. Create a local repository on ISE

ISE Repositories can be configured from both the GUI and the CLI of the ISE.

Repositories configured from CLI of the ISE node are local to each node and get removed upon reload of the node. Repositories configured from the GUI of the ISE are replicated to all nodes in deployment and don't get removed upon reload of the node.

local repo

3. Add the SFTP server host key to ISE if it does not exist

ise02/suresh# crypto host_key add host 10.10.1.10
host key fingerprint added
# Host 10.10.1.10 found: line 1 
10.10.1.10 RSA SHA256:3RGth6suxkcugOZ0kNsdf45Hgyngnty44

3. Copy the file to Cisco ISE from the SFTP server.

ISE02/suresh# copy sftp://10.10.1.10//home/suresh/ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz disk:/

Username: suresh
Password: 

ise02/suresh# dir

Directory of disk:/

        949 Dec 18 2019 15:47:01  &1
       4096 Oct 06 2019 03:04:46  corefiles/
  508937949 May 11 2020 11:12:48  ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz

           Usage for disk: filesystem 
                 2069942272 bytes total used
                27169460224 bytes free
                30829043712 bytes available
                

ISE02/suresh# show repository local

ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz

The file can be downloaded from here: https://software.cisco.com/download/home/283801620/type/283802505/release/2.6.0

urt-tool

Step -2 Install the Upgrade Readiness Tool

The URT identifies issues with data that might cause an upgrade failure, and reports or fixes the issues.

This tool will perform following tasks:

Prerequisite checks
Clone config database
Copy upgrade files
Data upgrade on cloned database
Time estimate for upgrade

Enter the application install command to install the URT. I pretty much said "yes" to everything.

application install <application-bundle> <repository-name>

ISE02/suresh# application install ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz local

Save the current ADE-OS running configuration? (yes/no) [yes] ? yes
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...

###########################################
# Installing Upgrade Readiness Tool (URT) #
###########################################

Checking ISE version compatibility
- Successful

Checking ISE persona
- Successful

Along with Administration, other services (MNT,PROFILER,SESSION) are enabled on this node. Installing and running URT might consume additional resources.
Do you want to proceed with installing and running URT now (y/n):y

Checking if URT is recent(<45 days old)
- Note: URT is 457 days old and its version is 1.0.0. There might be a recent URT bundle on CCO, please verify on CCO
Do you want to proceed with this version which is 457 days old (y/n):y
Proceeding with this version of URT itself

Installing URT bundle
- Successful

########################################
# Running Upgrade Readiness Tool (URT) #
########################################
This tool will perform following tasks:
1. Pre-requisite checks
2. Clone config database
3. Copy upgrade files
4. Data upgrade on cloned database
5. Time estimate for upgrade

Pre-requisite checks
====================
Disk Space sanity check
- Successful
NTP sanity
- Successful
Appliance/VM compatibility
- Successful
Trust Cert Validation
- Successful
System Cert Validation
- Successful
Invalid MDMServerNames in Authorization Policies check
- Successful
6 out of 6 pre-requisite checks passed

Clone config database
=====================
 [########################################] 100%  Successful                                         

Copy upgrade files
==================
- N/A

Data upgrade on cloned database
===============================
Modifying upgrade scripts to run on cloned database
- Successful

Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
- Successful

Running sanity after schema upgrade on cloned database
- Successful

Running data upgrade on cloned database
- Data upgrade step 1/45, UPSUpgradeHandler(2.4.0.101)... Done in 0 seconds.
- Data upgrade step 2/45, UPSUpgradeHandler(2.4.0.116)... Done in 0 seconds.
- Data upgrade step 3/45, MachineAuthenticationSettingsRegistration(2.4.0.120)... Done in 0 seconds.
- Data upgrade step 4/45, GuestAccessUpgradeService(2.4.0.126)... Done in 13 seconds.
- Data upgrade step 5/45, RegisterPostureTypes(2.4.0.127)... Done in 1 seconds.
- Data upgrade step 6/45, UPSUpgradeHandler(2.4.0.127)... Done in 0 seconds.
- Data upgrade step 7/45, UPSUpgradeHandler(2.4.0.134)... Done in 0 seconds.
- Data upgrade step 8/45, NSFUpgradeService(2.4.0.140)... Done in 0 seconds.
- Data upgrade step 9/45, NSFUpgradeService(2.4.0.155)... Done in 1 seconds.
- Data upgrade step 10/45, NSFUpgradeService(2.4.0.160)... Done in 0 seconds.
- Data upgrade step 11/45, NSFUpgradeService(2.4.0.161)... Done in 2 seconds.
- Data upgrade step 12/45, NSFUpgradeService(2.4.0.179)... Done in 0 seconds.
- Data upgrade step 13/45, NetworkAccessUpgrade(2.4.0.182)... Done in 2 seconds.
- Data upgrade step 14/45, StorageUpgradeService(2.4.0.183)... Done in 0 seconds.
- Data upgrade step 15/45, DnsHostnameResolutionRegistration(2.4.0.190)... Done in 0 seconds.
- Data upgrade step 16/45, CertMgmtUpgradeService(2.4.0.200)... .Done in 68 seconds.
- Data upgrade step 17/45, NSFUpgradeService(2.4.0.214)... Done in 0 seconds.
- Data upgrade step 18/45, ERSDictionaryRegistration(2.4.0.215)... Done in 0 seconds.
- Data upgrade step 19/45, NetworkAccessUpgrade(2.4.0.216)... Done in 0 seconds.
- Data upgrade step 20/45, ProfilerUpgradeService(2.4.0.227)... Done in 0 seconds.
- Data upgrade step 21/45, ProfilerUpgradeService(2.4.0.228)... Done in 5 seconds.
- Data upgrade step 22/45, ProfilerUpgradeService(2.4.0.229)... Done in 0 seconds.
- Data upgrade step 23/45, NetworkAccessUpgrade(2.4.0.240)... Done in 0 seconds.
- Data upgrade step 24/45, CertMgmtUpgradeService(2.4.0.293)... Done in 2 seconds.
- Data upgrade step 25/45, ProvisioningUpgradeService(2.4.0.299)... Done in 2 seconds.
- Data upgrade step 26/45, NSFUpgradeService(2.5.0.129)... Done in 0 seconds.
- Data upgrade step 27/45, NSFUpgradeService(2.5.0.130)... Done in 1 seconds.
- Data upgrade step 28/45, NSFUpgradeService(2.5.0.168)... Done in 0 seconds.
- Data upgrade step 29/45, NSFUpgradeService(2.5.0.183)... Done in 0 seconds.
- Data upgrade step 30/45, NSFUpgradeService(2.5.0.196)... Done in 0 seconds.
- Data upgrade step 31/45, GuestAccessUpgradeService(2.5.0.199)... Done in 7 seconds.
- Data upgrade step 32/45, UPSUpgradeHandler(2.5.0.200)... Done in 5 seconds.
- Data upgrade step 33/45, LSDSettingsRegistration(2.5.0.225)... Done in 0 seconds.
- Data upgrade step 34/45, NSFUpgradeService(2.5.0.236)... Done in 0 seconds.
- Data upgrade step 35/45, CertMgmtUpgradeService(2.5.0.276)... Done in 19 seconds.
- Data upgrade step 36/45, ProfilerUpgradeService(2.5.0.288)... Done in 0 seconds.
- Data upgrade step 37/45, UPSUpgradeHandler(2.5.0.316)... Done in 1 seconds.
- Data upgrade step 38/45, UPSUpgradeHandler(2.5.0.320)... Done in 0 seconds.
- Data upgrade step 39/45, RegisterPostureTypes(2.6.0.103)... Done in 0 seconds.
- Data upgrade step 40/45, ProvisioningUpgradeService(2.6.0.103)... Done in 0 seconds.
- Data upgrade step 41/45, UPSUpgradeHandler(2.6.0.108)... Done in 0 seconds.
- Data upgrade step 42/45, UPSUpgradeHandler(2.6.0.154)... Done in 0 seconds.
- Data upgrade step 43/45, NSFUpgradeService(2.6.0.156)... Done in 0 seconds.
- Data upgrade step 44/45, ProfilerUpgradeService(2.6.0.156)... Done in 0 seconds.
- Data upgrade step 45/45, GuestAccessUpgradeService(2.6.0.156)... Done in 10 seconds.
- Successful

Running data upgrade for node specific data on cloned database
- Successful

Time estimate for upgrade
=========================
(Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes is not considered in calculating estimates)
Estimated time for each node (in mins):
ISE01(SECONDARY PAP,MNT,PDP):124
ISE02(PRIMARY PAP,MNT,PDP):118


Final cleanup before exiting...

Application successfully installed
ISE02/suresh#

Once the URT has finished all the checks, we should be able to see the time estimate for the upgrade. If you get all "successful" message, the actual ISE upgrade should go smooth.

We can now remove URT from ISE.

ISE02/suresh# application remove urt
Continue with application removal? (y/n) [n] ? y

Application successfully uninstalled

removal

If URT fails

If URT fails for some reason, it will create a log file which can be shared with TAC for troubleshooting.

For example, if you have an expired certificate the URT will fail and generate a log file as shown below.

***Output omitted***

Pre-requisite checks
====================
Disk Space sanity check
- Successful
NTP sanity
- Successful
Appliance/VM compatibility
- Successful
Trust Cert Validation
Trust certificate with friendly name 'Default self-signed server certificate' is invalid: The certificate has expired.
The certificate has expired.
System certificate with friendly name 'Default self-signed saml server certificate - CN=SAML-ISE02.example.local' is invalid: The certificate has expired.
% Error:  One or more system certificates are invalid (see above), please update with valid system certificate(s) before continuing. Upgrade cannot continue.
/opt/CSCOcpm/upgrade/bin/isedbupgrade-functions.sh: line 101: [: -le: unary operator expected
- Failed
Invalid MDMServerNames in Authorization Policies check
- Successful
4 out of 6 pre-requisite checks passed
Some pre-requisite checks have failed. Hence exiting...

Final cleanup before exiting...

Collecting log files ...
- Encrypting logs bundle...
Please enter encryption password:  
Please enter encryption password again to verify: 
Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug
% Post-install step failed. Please check the logs for more details.

On the next post, I will share the ISE upgrade procedure (2.3 >> 2.6)

Reference

Cisco Identity Services Engine Upgrade Guide, Release 2.6 - Prepare for Upgrade [Support] - Cisco

Prepare for Upgrade

Cisco

Thanks for reading

As always, your feedback and comments are more than welcome.