Cisco Nexus 3548 - Configuring Active Latency Monitoring Hidden Cisco commands

In my previous blogtorial 'Cisco Nexus 3548 - Configuring Active Buffer Monitoring' I demonstrated how we can monitor buffer spaces, however what if I wanted to know the latency profile of the entire switch or per port. Enter the world of Cisco hidden commands and that's where you will find 'Active Latency Monitoring' quietly lurking around. These set of commands will enable switch latency profile measurements and give you the per port latency statistics such as total frame count within a time period, the min/max/avg latency for those frames.

For some reason, Cisco decided to keep it hidden although it may be added to the next release. As of now use this command with caution, since it is hidden it is not supported in the configuration (yet). 

Here is how you configure 'Active Latency Monitoring' on a Cisco Nexus 3548. 

Notes:

This is FILO latency so first in last byte out latency so you will not see ~ 230ns as promised by Cisco. If this was measuring FIFO latency then you might see the ~ 230ns per port latency. So the main thing is that you do a baseline trending and whenever there are spikes you can then calculate the difference and figure out the delay/latency.

Here is how to configure Active Latency Monitoring.

 TestNex(config)#hardware profile latency monitor threshold-avg 1000000 threshold-max 2000000  

You can set different threshold-avg and max to get syslog alerts.

These are the show commands to view the Active Latency Monitoring statistics.

Summary view:

 TestNex# show hardware profile latency monitor summary  
 Latency monitoring is an unsupported and untested feature, use at your own risk.  
 All latency information provided is measured as FILO (First In Last Out).  
 11/24/2013 12:07:33  
 Device instance 0  
 Total Switch  
 ============  
                   3s        30s       1hr       All Time  
 Min Latency (ns)  297       293       292       292  
 Max Latency (ns)  608       1957      2688      2688  
 Avg Latency (ns)  332       342       343       342  
 Std Deviation    23.95      8.31      0.75      0.73  

 Ethernet1/1  
 ===========  
                   3s        30s       1hr       All Time  
 Min Latency (ns)  400       398       398       398  
 Max Latency (ns)  416       1935      2041      2041  
 Avg Latency (ns)  408       446       448       448  
 Std Deviation    n/a       14.17      20.20      20.26  

Interpreting this output for Ethernet1/1 - In the past 3 seconds, the minimum latency for all packets egressing the interface was ~ 400ns, the maximum was ~ 416ns and average latency for all packets ~ 408ns. It goes on to show you the 30s, the 1hr and All time. Although I am not sure what the 'All Time' means whether it keeps it for a day or just for 2 hours. These counters can be valuable to trend overtime, and analyze for spikes and for capacity planning.

There is also a way to get raw counters which will output much more granular statistics collected every 3 seconds.

 TestNex# show hardware profile latency monitor raw interface Ethernet1/1  
 Latency monitoring is an unsupported and untested feature, use at your own risk.  
 All latency information provided is measured as FILO (First In Last Out).  
 11/24/2013 12:15:11  
 Device instance 0  
 Format:  
     timestamp  
     ifindex  
     fcnt  
     min_latency  
     max_latency  
     avg_latency  
     std_deviation  
 11/24/2013 12:15:08  
 0x1a000000  
 4  
 290  
 308  
 295  
 0.00  
 11/24/2013 12:15:05  
 0x1a000000  
 8  
 291  
 311  
 301  
 0.00  
 11/24/2013 12:15:02  
 0x1a000000  
 5  
 305  
 425  
 330  
 0.00  

Although I am not sure how realistic it is to monitor these counters every 3 seconds for every interface, perhaps just for uplinks.

Many more articles to come so stay tuned.

Please reshare/subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.