Cisco Nexus 3548 - Configuring Active Buffer Monitoring

It never ceases to amaze me that people value time as money. To me time is more than money. I can make money and I can make more if I worked at it, but time -- that's a different story. I cannot make time, I cannot buy more time, I cannot take it from someone else -- Once it's gone it's gone forever. As I sit here on the train (my daily hour commute) the time it's taking me to write this blogtorial is gone. I can never get it back so hopefully I am doing something worthwhile, something that will help someone, and something that I enjoy. With all that aside let's go into how to configure 'Active Buffer Monitoring' on a Cisco Nexus 3548.

When you are down to nanoseconds on port to port latency, monitoring your buffer space becomes more crucial than ever. Because whenever you buffer your switch is now store and forward for those buffered bytes instead of cut through. If you're switch is store and forward then you are not getting the ~ 230 nanoseconds port-to-port latency that you just spent $20K for.

Before we can dive into configuring buffer monitoring it would be beneficial to understand buffer blocks and how the ports are mapped. First there are 3 buffer blocks each containing 6MB worth of buffer space which gives us a total of 18MB shared buffer. Ports to these buffer blocks (3 x 6MB) are mapped as follows.


Port 01-04, 13-16, 25-28, 37-40 are assigned to buffer block 1 (6MB total)
Port 05-08, 17-20, 29-32, 41-44 are assigned to buffer block 2 (6MB total)
Port 09-12, 21-24, 33-36, 45-48 are assigned to buffer block 3 (6MB total)

Courtesy of www.cisco.com

Each buffer blocks are then split up into multiple buckets for efficient memory management and they are as follows.

384KB - 768KB - 1152KB - 1536KB - 1920KB - 2304KB - 2688KB - 3072KB - 3456KB - 3840KB - 4224KB - 4608KB - 4992KB - 5376KB - 5760KB - 6144KB

Aside from these each port has a static 12KB buffer which is not mentioned in the white papers. Also worthwhile to mention is that there are separate unicast and multicast queues. Therefore, you can only watch either multicast buffer blocks or unicast buffer blocks.

First, here is how to configure and read unicast active buffer monitoring output. In this example I am configuring unicast buffer monitoring with a sample hardware polling rate of 1ms. Default is 4ms. So every one millisecond the hardware counters are polled. Although the hardware counters are polled every 1ms the system output only shows 1s at a time. More on this below when we review the output of the show commands.

 TestNex#config t  
 TestNex(config)#hardware profile buffer monitor unicast sampling 1000000  
 TestNex(config#exit  
 TestNex# show hardware profile buffer monitor ?  
  brief     Show brief buffer monitor data  
  buffer-block Buffer block  
  detail    Show detail buffer monitor data  
  interface   Show buffer monitoring data of an interface  
  multicast   Multicast buffer block  
  summary    Show summary buffer monitor data  

There are options to add thresholds and alert via syslog messages.

I usually do a summary, brief and then do a detail to investigate further. For example ...

 TestNex# show hardware profile buffer monitor detail  
 Detail CLI issued at: 11/14/2013 21:47:05  
 Legend -  
 384KB - between  1 and 384KB of shared buffer consumed by port  
 768KB - between 385 and 768KB of shared buffer consumed by port  
 307us - estimated max time to drain the buffer at 10Gbps  
 Active Buffer Monitoring for port Ethernet1/1 is: Active  
 KBytes              384  768  1152 1536 1920 2304 2688 3072 3456 3840 4224 4608 4992 5376 5760 6144  
 us @ 10Gbps         307  614  921  1228 1535 1842 2149 2456 2763 3070 3377 3684 3991 4298 4605 4912  
                     ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----  
 11/14/2013 21:47:04   0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0  
 11/14/2013 21:47:03   0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0  
 11/14/2013 21:47:02   0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0  
 11/14/2013 21:47:01   125  0    0    0    0    0    0    0    0    0    0    0    0    0    0    0  

Notice the system output is every 1s even though the hardware polling is set to 1ms.

How do you read this output? It is sort of confusing at first but once you get used to this it is straight forward. In this output you notice 125 under 384KB bucket for Port Ethernet1/1 at 21:47:01. This means that for 125 x hardware sampling rate the buffer was utilized. So 125 x 1ms (which is my hardware sampling rate) is equal to 125ms. So for 125ms between 21:47:01 and 21:47:02 the 384KB bucket was being utilized by packets egressing Ethernet1/1.

Notes: 

• History is kept for an hr and then copied to the bootflash. So at any given point in time the switch keeps 2 hours worth of buffer data. 
• There are python APIs to ship this data somewhere else for historical and trending purpose. 
• Switch polls every "hardware sample rate" but only reports to the system every second which is why you see the command output listed every second. 
• The counters are 8 bit counters so the value can only go up to 255. 

How to configure multicast active buffer monitoring?

 TestNex#config t  
 TestNex(config)#hardware profile buffer monitor multicast sampling 1000000  
 TestNex(config#exit  
 TestNex# show hardware profile buffer monitor ?  
  brief     Show brief buffer monitor data  
  buffer-block Buffer block  
  detail    Show detail buffer monitor data  
  interface   Show buffer monitoring data of an interface  
  multicast   Multicast buffer block  
  summary    Show summary buffer monitor data  

Caveats with multicast 'Active Buffer Monitoring':

You cannot watch it per interface. Although you have separate multicast tx queues -- you cannot watch them individually. The output will only show you which buffer block is being utilized.

Many more articles to come so ....

Please subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.