Cisco ASA VPN Troubleshooting and Debugging

Setting up a site-to-site VPN is easy, but the real hard work starts when the tunnel doesn't come up. It gets even harder when we don't control both ends of the VPN. Often, the other side is managed by a different company, and we can't just log in to check what's going on.

Even though it's tempting to just blame the other side when there's a problem, it's important for us to do our own checks first. We need to ensure that our side of the connection is working as it should.

In this blog post, we're going to talk about how to do that. We'll go through some basic steps for troubleshooting a Cisco ASA Site-to-Site VPN. By the end, you'll have a better idea of how to figure out what's going wrong and how to fix it.

If you are not familiar with ASA Site-Site-VPN, please check out my other blog post below.

A Few Things to Consider

In this post, we're focusing on troubleshooting with IKEv1. But don't worry if you're using IKEv2 — the process is pretty much the same.

Please also note that in our examples, we have Cisco ASA firewalls on both sides of the VPN. But even if you're dealing with a different firewall on the other side, the information you'll get from the ASA when you're debugging will still be the same. So, the steps we'll talk about here will be helpful regardless of the firewall you're working with on the other end.

A Couple of Useful Commands

Before we start, let's talk about two commands that are really helpful for troubleshooting. They are show crypto isakmp sa and show crypto ipsec sa.

These commands show us important info about our VPN. We'll be using them a lot in this blog post. So, remember these commands, as they'll help us understand what's going on with our VPN connection.

The show crypto isakmp sa command is used to check the status of the Phase-1 tunnel. On the other hand, the show crypto ipsec sa command is used for checking the Phase-2 tunnel. Like the first command, this one will show if the Phase-2 tunnel is up and running properly.

Be Careful When Debugging

When you're troubleshooting with debug commands, it's important to be cautious. Debugging can create a large amount of output on your terminal, and it can get overwhelming pretty quickly.

If you ever feel like the output is getting too much to handle, or if it's just not giving you the information you need, you can use the undebug all command. This command will immediately stop all debugging.

Diagram

Phase-1 Issues

Mismatch Policies

Phase-1 of setting up a VPN involves creating a secure channel between two VPN devices. It's where your devices agree on how to safely exchange information. But if something goes wrong here, your VPN connection won't even get off the ground.

In my personal experience, a common issue during this stage is misconfiguration of IKEv1 policies. It's so important to double-check your configurations and ensure there's at least one policy that matches on both ends.

To demonstrate the Phase-1 mismatch, I purposely misconfigured the phase-1 policy and initiated the tunnel from my side. Let's look at the debug and see what it shows.

debug crypto ikev1 1

💡

The '1' at the end of the command specifies the level of detail we want in our debug output. The higher the number, the more detailed the information we get. In this case, '1' means we're asking for a basic level of detail. If we want more information, we can replace '1' with a higher number (1-255)

Jun 05 09:05:25 [IKEv1]IP = 195.95.10.1, Information Exchange processing failed
Jun 05 09:05:37 [IKEv1]IP = 195.95.10.1, Information Exchange processing failed
Jun 05 09:05:45 [IKEv1]IP = 195.95.10.1, Information Exchange processing failed

If there is a Phase-1 issue, you should see a similar message as shown above. If you want to see a bit more detail, you can level up the debug. I'm going to use 32 for this example.

our-asa# debug crypto ikev1 32

ciscoasa# Jun 05 08:44:25 [IKEv1 DEBUG]Pitcher: received a key acquire message, spi 0x0
Jun 05 08:44:25 [IKEv1]IP = 195.95.10.1, IKE Initiator: New Phase 1, Intf INSIDE, IKE Peer 195.95.10.1  local Proxy Address 192.168.20.0, remote Proxy Address 172.16.10.0,  Crypto map (VPN-MAP)
Jun 05 08:44:25 [IKEv1 DEBUG]IP = 195.95.10.1, constructing ISAKMP SA payload
Jun 05 08:44:25 [IKEv1 DEBUG]IP = 195.95.10.1, constructing NAT-Traversal VID ver 02 payload
Jun 05 08:44:25 [IKEv1 DEBUG]IP = 195.95.10.1, constructing NAT-Traversal VID ver 03 payload
Jun 05 08:44:25 [IKEv1 DEBUG]IP = 195.95.10.1, constructing NAT-Traversal VID ver RFC payload
Jun 05 08:44:25 [IKEv1 DEBUG]IP = 195.95.10.1, constructing Fragmentation VID + extended capabilities payload
Jun 05 08:44:25 [IKEv1]IP = 195.95.10.1, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 172
Jun 05 08:44:25 [IKEv1]IKE Receiver: Packet received on 185.15.10.1:500 from 195.95.10.1:500
Jun 05 08:44:25 [IKEv1]IP = 195.95.10.1, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 100
Jun 05 08:44:25 [IKEv1]IP = 195.95.10.1, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 100
Jun 05 08:44:25 [IKEv1]IP = 195.95.10.1, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping
Jun 05 08:44:25 [IKEv1]IP = 195.95.10.1, Information Exchange processing failed
Jun 05 08:44:33 [IKEv1]IP = 195.95.10.1, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 172
Jun 05 08:44:33 [IKEv1]IKE Receiver: Packet received on 185.15.10.1:500 from 195.95.10.1:500
Jun 05 08:44:33 [IKEv1]IP = 195.95.10.1, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 100
Jun 05 08:44:33 [IKEv1]IP = 195.95.10.1, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length : 100
Jun 05 08:44:33 [IKEv1]IP = 195.95.10.1, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping
Jun 05 08:44:33 [IKEv1]IP = 195.95.10.1, Information Exchange processing failed

You can also use the show command to see if Phase-1 is up as shown below.

our-asa# show isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 195.95.10.1
    Type    : user            Role    : initiator 
    Rekey   : no              State   : MM_WAIT_MSG2

The tunnel that gets stuck on MM_WAIT_MSG2 usually indicates there are no matching phase-1 policies on the remote end or our firewall is unable to reach the peer device.

If you run a packet-tracer command with a Phase-1 policy mismatch, the VPN phase also results in DROP as shown below.

our-asa# packet-tracer input insIDE tcp 192.168.20.10 25000 172.16.10.10 80

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 185.15.10.14 using egress ifc  OUTSIDE

Phase: 2
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 3
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: QOS     
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:

Result:
input-interface: INSIDE
input-status: up
input-line-status: up
output-interface: OUTSIDE
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Mismatched Pre-Shared Keys

To demonstrate, I'm going to change the PSK on one side.

Depending on which side of the ASA you are on, you will see either MM_WAIT_MSG5 or MM_WAIT_MSG6message.

remote-asa# show crypto isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 185.15.10.1
    Type    : L2L             Role    : responder 
    Rekey   : no              State   : MM_WAIT_MSG5

our-asa# show crypto isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 195.95.10.1
    Type    : L2L             Role    : initiator 
    Rekey   : no              State   : MM_WAIT_MSG6

There are no IKEv2 SAs

If you look at the debug message, you would see something like this.

{ ..... }
Jun 05 16:20:48 [IKEv1]Group = 195.95.10.1, IP = 195.95.10.1, Error, peer has indicated that something is wrong with our message.  This could indicate a pre-shared key mismatch.
{ ..... }

Tunnel-Group Misconfiguration

Another thing to watch out for is the tunnel-group configuration. In a VPN setup, the tunnel-group configuration is a key part of establishing a secure connection. It contains the details about the remote peer that your ASA needs to set up the VPN.

If the other side of your VPN is missing the tunnel-group configuration, or if it's set up incorrectly, you could run into problems. Specifically, you might find that the VPN gets stuck at MM_WAIT_MSG4 status when you check with the show crypto isakmp sa command. This is a clue that there's a problem with the tunnel-group configuration on the other side.

our-asa# show crypto isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 195.95.10.1
    Type    : user            Role    : initiator 
    Rekey   : no              State   : MM_WAIT_MSG4

There are no IKEv2 SAs

To conclude, If Phase-1 is correctly configured you should see the output State: MM_ACTIVE If you see a State other than MM_ACTIVE, then the issue is definitely with Phase-1.

our-asa# show isa sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 195.95.10.1
    Type    : L2L             Role    : initiator 
    Rekey   : no              State   : MM_ACTIVE 

There are no IKEv2 SAs

Phase-2 Issues

When we move to Phase-2 of the VPN setup, there are two main things we need to look at. First is the Phase-2 policies, and the second is the ACLs that define the interesting traffic.

The interesting traffic should match on both sides of the VPN. This means that the traffic we want to protect should be the same at both ends. However, the way we write the ACLs for this traffic will be inverted.

For example, if you're setting up a VPN between Site-1 and Site-B, your ACL at Site-1 might say that the source is Subnet-A and the destination is Subnet-B. But when you go to write the ACL at Site-B, you'll flip that around. The source will be Subnet-B, and the destination will be Subnet-A.

If you have a mismatch Phase-2 policy or ACL, the debug should output something like this.

debug crypto ikev1 1

our-asa# Jun 05 09:00:42 [IKEv1]Group = 195.95.10.1, IP = 195.95.10.1, Removing peer from correlator table failed, no match!
Jun 05 09:00:42 [IKEv1]Group = 195.95.10.1, IP = 195.95.10.1, Session is being torn down. Reason: User Requested
Jun 05 09:00:48 [IKEv1]Group = 195.95.10.1, IP = 195.95.10.1, Removing peer from correlator table failed, no match!
Jun 05 09:00:48 [IKEv1]Group = 195.95.10.1, IP = 195.95.10.1, Session is being torn down. Reason: User Requested

You can verify if Phase-2 is working correctly by using the following command. If there is no phase-2 tunnel the output will be blank for that particular peer.

our-asa# show crypto ipsec sa
interface: OUTSIDE
    Crypto map tag: VPN-MAP, seq num: 1, local addr: 185.15.10.1

      access-list local_remote extended permit ip 192.168.20.0 255.255.255.0 172.16.10.0 255.255.255.0 
      local ident (addr/mask/prot/port): (192.168.20.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (172.16.10.0/255.255.255.0/0/0)
      current_peer: 195.95.10.1


      #pkts encaps: 19, #pkts encrypt: 19, #pkts digest: 19
      #pkts decaps: 19, #pkts decrypt: 19, #pkts verify: 19
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 19, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

{ TRUNCATED }

NAT Exemption

NAT exemption tells the firewall not to perform NAT on VPN traffic. When we create a VPN, we want the original IP addresses to remain intact as they traverse the VPN tunnel. This is especially important for the 'interesting traffic' that we're protecting with our VPN.

If the third party hasn't configured NAT exemption correctly, the problem might not be immediately obvious. Even when you run debug commands or use packet-tracer, everything might seem to be working as expected.

If you end up in this situation where everything looks good but you can't reach the other side, use the following command and look specifically for the number of packets being encrypted and decrypted.

First, check if Phase-1 is up - Yes

our-asa# show crypto isakmp sa

IKEv1 SAs:

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 195.95.10.1
    Type    : L2L             Role    : initiator 
    Rekey   : no              State   : MM_ACTIVE 

There are no IKEv2 SAs

Next check, if Phase-2 is up - Yes, but!

our-asa# show crypto ipsec sa 
interface: OUTSIDE
    Crypto map tag: VPN-MAP, seq num: 1, local addr: 185.15.10.1

      access-list local_remote extended permit ip 192.168.20.0 255.255.255.0 172.16.10.0 255.255.255.0 
      local ident (addr/mask/prot/port): (192.168.20.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (172.16.10.0/255.255.255.0/0/0)
      current_peer: 195.95.10.1


      #pkts encaps: 40, #pkts encrypt: 40, #pkts digest: 40
      #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 40, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 185.15.10.1/0, remote crypto endpt.: 195.95.10.1/0
      path mtu 1500, ipsec overhead 74(44), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: D7B26952

But! - If you look closely, you can see the number of packets encrypted is 40 but the number of packets decrypted is zero. This indicates that we are sending the traffic through the tunnel but we are not receiving any traffic. This usually means the other party haven't configured the NAT exemption properly.

Conclusion

To wrap up, troubleshooting a Cisco ASA VPN can be a tough task, but it's doable. You just need to double-check your settings, be careful with debugging, and understand how the different parts like Phase-1 and Phase-2.