When you are down to nanoseconds on port to port latency, monitoring your buffer space becomes more crucial than ever. Because whenever you buffer your switch is now store and forward for those buffered bytes instead of cut through. If you're switch is store and forward then you are not getting the ~ 230 nanoseconds port-to-port latency that you just spent $20K for.
Before we can dive into configuring buffer monitoring it would be beneficial to understand buffer blocks and how the ports are mapped. First there are 3 buffer blocks each containing 6MB worth of buffer space which gives us a total of 18MB shared buffer. Ports to these buffer blocks (3 x 6MB) are mapped as follows.
Port 01-04, 13-16, 25-28, 37-40 are assigned to buffer block 1 (6MB total)
Port 05-08, 17-20, 29-32, 41-44 are assigned to buffer block 2 (6MB total)
Port 09-12, 21-24, 33-36, 45-48 are assigned to buffer block 3 (6MB total)
|Courtesy of www.cisco.com|
Each buffer blocks are then split up into multiple buckets for efficient memory management and they are as follows.
384KB - 768KB - 1152KB - 1536KB - 1920KB - 2304KB - 2688KB - 3072KB - 3456KB - 3840KB - 4224KB - 4608KB - 4992KB - 5376KB - 5760KB - 6144KB
Aside from these each port has a static 12KB buffer which is not mentioned in the white papers. Also worthwhile to mention is that there are separate unicast and multicast queues. Therefore, you can only watch either multicast buffer blocks or unicast buffer blocks.
First, here is how to configure and read unicast active buffer monitoring output. In this example I am configuring unicast buffer monitoring with a sample hardware polling rate of 1ms. Default is 4ms. So every one millisecond the hardware counters are polled. Although the hardware counters are polled every 1ms the system output only shows 1s at a time. More on this below when we review the output of the show commands.
TestNex#config t TestNex(config)#hardware profile buffer monitor unicast sampling 1000000 TestNex(config#exit TestNex# show hardware profile buffer monitor ? brief Show brief buffer monitor data buffer-block Buffer block detail Show detail buffer monitor data interface Show buffer monitoring data of an interface multicast Multicast buffer block summary Show summary buffer monitor data
There are options to add thresholds and alert via syslog messages.
I usually do a summary, brief and then do a detail to investigate further. For example ...
TestNex# show hardware profile buffer monitor detail Detail CLI issued at: 11/14/2013 21:47:05 Legend - 384KB - between 1 and 384KB of shared buffer consumed by port 768KB - between 385 and 768KB of shared buffer consumed by port 307us - estimated max time to drain the buffer at 10Gbps Active Buffer Monitoring for port Ethernet1/1 is: Active KBytes 384 768 1152 1536 1920 2304 2688 3072 3456 3840 4224 4608 4992 5376 5760 6144 us @ 10Gbps 307 614 921 1228 1535 1842 2149 2456 2763 3070 3377 3684 3991 4298 4605 4912 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 11/14/2013 21:47:04 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11/14/2013 21:47:03 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11/14/2013 21:47:02 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11/14/2013 21:47:01 125 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Notice the system output is every 1s even though the hardware polling is set to 1ms.
How do you read this output? It is sort of confusing at first but once you get used to this it is straight forward. In this output you notice 125 under 384KB bucket for Port Ethernet1/1 at 21:47:01. This means that for 125 x hardware sampling rate the buffer was utilized. So 125 x 1ms (which is my hardware sampling rate) is equal to 125ms. So for 125ms between 21:47:01 and 21:47:02 the 384KB bucket was being utilized by packets egressing Ethernet1/1.
- History is kept for an hr and then copied to the bootflash. So at any given point in time the switch keeps 2 hours worth of buffer data.
- There are python APIs to ship this data somewhere else for historical and trending purpose.
- Switch polls every "hardware sample rate" but only reports to the system every second which is why you see the command output listed every second.
- The counters are 8 bit counters so the value can only go up to 255.
TestNex#config t TestNex(config)#hardware profile buffer monitor multicast sampling 1000000 TestNex(config#exit TestNex# show hardware profile buffer monitor ? brief Show brief buffer monitor data buffer-block Buffer block detail Show detail buffer monitor data interface Show buffer monitoring data of an interface multicast Multicast buffer block summary Show summary buffer monitor data
Caveats with multicast 'Active Buffer Monitoring':
You cannot watch it per interface. Although you have separate multicast tx queues -- you cannot watch them individually. The output will only show you which buffer block is being utilized.
Many more articles to come so ....
Please subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.