Hello, I have used VPN debug commands on ASAs before and they have been very helpful, but this has always been on IOS 7 & 8. I'm working on 2 ASA's now that are on 9.1.3 and I can't establish a site to site VPN.
Two-factor authentication, or multifactor authentication, for VPN-access has been around for a long time to ensure only trusted devices, personal and consultants are able to access the corporate network from the outside. With the rise of smartphones in the past 10 years, there are now several authentication engines that leverage Push Notifications with simple Permit or Deny buttons that can be used to perform two-factor authentication without having the user to deal with One-Time Passwords (OTP) via SMS or code-generating apps.
Writer’s Update September 23rd 2020 - starting with AnyConnect 4.8.02045 - the default authentication timeout has been increased from 12 seconds to 30 seconds as per TAC information and documented in TAC-case CSCve01989
- Cisco Vpn Debug Log, Cisco Clientless Vpn Rdp Plugin, Vpn Umontrealal, Incapable Connecter Nordvpn.
- As promised I mentioned we were going to go over some debug output from 2 Cisco ISRs establishing an IPSec VPN. Now I’m not going to go over every line in the debug but I’ll touch on some of the things to look out for. Now, from the beginning.
- From the Applications folder, click the AnyConnect VPN icon to open the user interface. A new pane labeled Cisco AnyConnect VPN Client will pop up. Click on the gear shaped icon lower left panel Select the Statistics tab.
- Find the service ‘Cisco AnyConnect Network Access Manager‘. Right click and stop the service. Now Cisco AnyConnect will say ‘NAM service unavailable‘ Now Windows will use the native WiFi client to connect to the WiFi; When you reboot your system, the NAM service will start and may cause problem again.
The Setup
This is the set of devices and services relevant to the point of this article:
The Problem
The first time I was tasked with implementing two-factor authentication using a Push Notification smartphone app I pretty quickly discovered that there was something wrong in my setup. I had an Cisco ASA-firewall (which users connect to over the internet) talking to a RADIUS-server which did lookups in an Active Directory using LDAP to validate users’ credentials. The RADIUS-server also issued Push Notifications to users’ smartphones. The idea was that users first must login using Active Directory credentials (username and password) and if those are correct then the RADIUS-server would send a Push Notification to the user’s smartphone, asking them to authenticate the session by pressing “Permit” on their smartphone app.
Sure thing I did get a push notification… and then another one… and another one. Before I even had the chance to “permit” the first Push Notification, I had already received another two of them.
It didn’t take to long to figure out that something was triggering multiple Push Notifications but nailing down where they came wrong wasn’t too obvious. Naturally I started working my way through the ASA and trying different Timeouts and Retry timers for my RADIUS-server and my RADIUS Group configuration. No matter how much I tinkered with this I was still getting sent too many Push Notifications sent to my smartphone every time I tried to logging into the VPN.
Following the stream of packets
Next up, I fired up the Packet Capture tool in ASDM and caught the packets which would tell the truth of what was going on. You can see below what the packet capture looked like when opened in Wireshark.
Frame #1 is the result of me pressing Enter after I have entered my credentials in the AnyConnect VPN-client software. The ASA sends a RADIUS-request packet to the RADIUS-server with my credentials in it. Frame #2 shows the ASA doing a Retry of this request after 10 seconds has past. This is because the Retry timeout of the RADIUS-server is configured to 10 seconds in ASA, which is the maximum number of seconds for Retry. It can only be configured for less, not more.
But then something else enters the flow of packets: another new RADIUS session. After approximately 12 seconds, another RADIUS-Request packet is sent from the ASA towards the RADIUS-server and it is using a different session-ID compared to the first RADIUS-sessions started in frame #1.
Another 8 seconds pass (for a total of 20 seconds since frame#1) and the ASA can be seen doing another Retry inside the first RADIUS-session.
At 22 seconds we see the ASA doing a Retry of the second RADIUS-session (12 + 10 seconds).
We see another 2 seconds pass and when the clock hits 24 seconds a third RADIUS session has appeared before us.
What is happening
Writer’s Update September 23rd 2020 - starting with AnyConnect 4.8.02045 - the default authentication timeout has been increased from 12 seconds to 30 seconds as per TAC information and documented in TAC-case CSCve01989
Doing similar captures over and over again while changing timers on the ASA between every attempt, I could quickly narrow down that there was an interval of 12 seconds that kept showing up and creating new RADIUS sessions while others were still in progress. Throwing together “ASA VPN” and “12 seconds” on your favorite search engine will probably earn you the answer and that is that Cisco’s AnyConnect VPN-client software has an authentication timeout of 12 seconds (by default). That means that every 12 seconds, AnyConnect will re-send your credentials to the ASA, which in turn will send them to the RADIUS-server, creating a whole new RADIUS session. If you fire up the VPN Profile Editor you can see this default value at the bottom of the Preferences (Part 2) window:
Effectively, the user will only have 12 seconds on them to unlock their smartphone, launch the app and press “Permit” inside the app. Some two-factor authentication apps ever have passwords of their own, making it ever harder for users to keep up.
The Solution
So, how do we increase the 12 seconds authentication timeout of AnyConnect? For corporate computer managed by your organization, you can simply create a AnyConnect VPN Profile using the VPN Profile Editor. In my case I increased the Authentication Timeout to 120 seconds (which is the maximum value you can set according to the documentation) and after that I had no trouble releasing this solution into the production network.
An important note is that changing the authentication timeout doesn’t take effect unless you configure it together with a pre-defined VPN-server (the FQDN/DNS-name of the ASA VPN service). Please note that is has to be a FQDN that is defined in the VPN Profile, simply pre-filling the IP-address of the outside interface of your ASA will not work!! This is a very odd and specific configuration that I assume is a bug that has been along for the ride for a long time based on what I’ve been reading online.
This solutions isn’t all great though because VPN-solutions are often used to grant access to consultants outside of your organization and usually you don’t have control of their computer. What you can do is create a VPN Profile directly in your ASA and assign it to the Group Policy that is to be used by consultants but the issue here is that the profile is not downloaded and activated until the consultant has performed a successful authentication first. This means the first login is gonna be a bit tough and short on time but once they get through it they will be able to download the VPN profile which increases the AnyConnect authentication timeout to whatever value your set it to in the profile (preferably 60 seconds or more).
How are One-Time Passwords solutions different?
Many of these two-factor authentication apps are also capable of generating One-Time Passwords inside the app instead of relying on Push Notifications. There are also solutions that use SMS to send out the OTP-code to users. The OTP-code is then entered into AnyConnect by the user after they have completed their first authentication (which validates their credentials towards an LDAP or AD-server).
To demonstrate this I have setup another AAA-server that is only using OTP-SMS to perform 2FA-authentication. The setup is as follows:
And when 2FA takes place using OTP-SMS instead of push notifications, the packet capture looks like this:
The big difference here is that the ASA receives a RADIUS Challenge packet from the RADIUS-server after the user’s credentials have been verified. This packets makes the ASA tells AnyConnect that an additional piece of authentication needs to take place. Because of this, AnyConnect will know it has to wait for the user to input the OTP-code and as a result will not re-send user credentials every 12 seconds.
Final words
Browsing the Cisco Support forums you can find quite a few discussions about how to deal with these kinds of 2FA/MFA solutions where timeouts are a pain in the butt and I thought it was worth taking a deep dive into the issue at hand and going through the steps of what is actually happening. Cisco has sort of acknowledged the weird issue with the Authentication Timeout in AnyConnect requiring an FQDN being present in the configuration, through the Knowledge Base article of their own 2FA/MFA product called Duo. You can see Duo’s statement on this issue on this link https://help.duo.com/s/article/1146?language=en_US
Introduction
This document describes how to enable AnyConnect Network Access Manager (NAM) logging as well as to collect and interpret the logs. The examples included in the document describe different authentication scenarios and the logs that reflect the steps taken by Network Access Manager to authenticate the client.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Enable NAM Logging
If an issue is identified that may be related to NAM module, the first step is to enable Extended Logging feature. This must be done on the client endpoint while NAM module is running.
Step 1. Open AnyConnect window and make sure it's in focus.
Step 2. Press this key combination, Left Shift + Left Alt + L. There is no response.
Step 3. Right click on AnyConnect icon in Windows System Tray. A menu pops up.
Step 4. Select Extended Logging so it has a check mark displayed. NAM now logs detailed debug messages.
Configure NAM Packet Capture
When Extended Logging is enabled, NAM also keeps a packet capture buffer going. The buffer is by default limited to about 1MB. If packet capture is needed, it may be beneficial to increase buffer size so it captures more activities. To extend the buffer, a XML setting file must be manually modified.
Cisco Anyconnect Debug Dap
Step 1. On the Windows PC, browse to:
C:ProgramDataCiscoCisco AnyConnect Secure Mobility ClientNetwork Access Managersystem
Step 2. Open file internalConfiguration.xml.
Step 3. Locate XML tag <packetCaptureFileSize>1</packetCaptureFileSize> and adjust the value to 10 for a 10MB buffer size, and so on.
Step 4. Reboot the client PC for the change to take effect.
Log Collection
NAM log collection is done via Diagnostic And Reporting Tool (DART), which is a module of AnyConnect suite. In the installer, select a module and use AnnyConnect full installation ISO to install. The Cisco Media Services Interface (MSI) installer can also be found inside the ISO.
After you enable Extended Logging and perform a test, simply run DART and go through the dialogue, the log bundle is located by default on the Windows Desktop.
In addition to DART bundle, the NAM message log is also helpful to locate the relevant data in the NAM log. In order to find the NAM message log, navigate to AnyConnect settings window > Network Access Manager > Message History. The message log contains timestamp of each network connection event, which can be used to find the logs relevant to the event.
Reading NAM Logs
NAM logs, especially after you enable Extended Logging, contains a large amount of data, most of which are irrelevant and can be ignored. This section lists out the debug lines to demonstrate each step NAM takes to establish a network connection. When you work through a log, these key phrases may be helpful to locate part of the log relevant to the issue.
Log Summary of a Network Connection without 802.1x Enabled Authentication
Explanation: This indicates that the user has selected a network from NAM module, and NAM has received a userEvent of START.
Explanation: Both Access State Machine and Network State Machine have been started.
Explanation: The IPv4 instance got cancelled in order to reset the states.
Explanation: The adapter with ID 484E4FEF-392C-436F-97F0-CD7206CD7D48 was selected to connect to network test123, which is the name of the network connection configured in NAM.
Explanation: NAM has successfully engaged the adapter for this network. Now NAM tries to associate (connect) to this network (which happens to be wireless):
Explanation: openNoEncryption indicates that the network is configured as open. On the Wireless Lan Controller it uses MAC Authentication Bypass (MAB) to authenticate.
Explanation: cs can be seen a lot in NAM logs. These are irrelevant logs and should be ignored.
Explanation: These are Simple Object Access Protocol (SOAP) messages used to tell AnyConnect GUI to display the connection status message such as Associating in this case. Any error messages displayed on NAM window can be found in one of the SOAP messages in the log which can be used to locate the issue easily.
Explanation: NAM receives an AUTH_SUCCESS event, which misleads because there is no authentication which currently happened. You are get this event simply because you connect to an open network, so by default authentication is successful.
Explanation: Association to Service Set Identifier (SSID) is successful, time to handle authentication.
Explanation: Since this is an open network, it is by default authenticated. At this point, NAM is connected to the network and now starts DHCP process:
Explanation: NAM successfully acquires an IP address.
Explanation: Once an IP address is received NAM will send ARP (Address Resolution Protocol) request to the gateway (Get-Connectivity). Once the ARP response is received the client is connected.
Log Summary of a Network Connection using 802.1x and PEAP over Wired Network
Explanation: NAM started to connect to network WiredPEAP.
Explanation: NAM matched an adapter to this network.
Explanation: NAM started connecting to this wired network.
Explanation: Client sends EAPOL_START.
Explanation: Client receives Identity Request from the switch, it now looks for a credential to send back.
Explanation: By default, Anyconnect sends anonymous as unprotected identity (outter identity), so here it tries anonymous and see if the server is OK with it. The fact that the identity is anonymous as opposed to host/anonymous indicates that it's a user authentication, rather than machine authentication.
Explanation: RADIUS server sends an Extensible Authentication Protocol-Transport Layer Security (EAP-TLS) frame without any content. Its purpose is to negotiate EAP-TLS protocol with the client.
Explanation: NAM recognizes server's request to use EAP-TLS but the client is configured to use Protected Extensible Authentication Protocol (PEAP). This is the reason that NAM sends back a counter-offer for PEAP.
Explanation: RADIUS server accepts the outter/unprotected identity.
Explanation: The Protected portion of PEAP (to establish a secure tunnel to exchange inner credentials) starts, after client receives a confirmation from RADIUS server to continue the use of PEAP.
Explanation: NAM sends a client hello encapsulated in EAP message and waits for server hello to come. The server's hello contains ISE certificate, so it takes some time to finish transferring.
Explanation: NAM extracted the subject name of the ISE server from server certificate. Since it doesn't have server certificate installed in the trust store, you do not find it there.
Explanation: NAM looks for the inner/protected identity to be sent to RADIUS server after tunnel is established. In this case, 'Automatically use my Windows logon name and password' option has been enabled on the wired adapter, so NAM uses windows logon credentials instead of asking the user for it.
Explanation: NAM sent client key and cipher spec to server and received confirmation. SSL negotation is successful and a tunnel is established.
Explanation: Protected identity is sent to the server, who accepts the identity. Now server requests password.
Explanation: NAM receives password request and sends password to server.
Cisco Anyconnect Log File
Explanation: Server receives the password, verifies it and sends EAP-Success. Authentication is successful at this point, and client proceeds as it gets the IP address from DHCP.