Cisco Nexus 3548 - Configuring Active Latency Monitoring Hidden Cisco commands

In my previous blogtorial 'Cisco Nexus 3548 - Configuring Active Buffer Monitoring' I demonstrated how we can monitor buffer spaces, however what if I wanted to know the latency profile of the entire switch or per port. Enter the world of Cisco hidden commands and that's where you will find 'Active Latency Monitoring' quietly lurking around. These set of commands will enable switch latency profile measurements and give you the per port latency statistics such as total frame count within a time period, the min/max/avg latency for those frames.

For some reason, Cisco decided to keep it hidden although it may be added to the next release. As of now use this command with caution, since it is hidden it is not supported in the configuration (yet). 

Here is how you configure 'Active Latency Monitoring' on a Cisco Nexus 3548. 

Cisco Nexus 3548 - Configuring Active Buffer Monitoring

It never ceases to amaze me that people value time as money. To me time is more than money. I can make money and I can make more if I worked at it, but time -- that's a different story. I cannot make time, I cannot buy more time, I cannot take it from someone else -- Once it's gone it's gone forever. As I sit here on the train (my daily hour commute) the time it's taking me to write this blogtorial is gone. I can never get it back so hopefully I am doing something worthwhile, something that will help someone, and something that I enjoy. With all that aside let's go into how to configure 'Active Buffer Monitoring' on a Cisco Nexus 3548.

When you are down to nanoseconds on port to port latency, monitoring your buffer space becomes more crucial than ever. Because whenever you buffer your switch is now store and forward for those buffered bytes instead of cut through. If you're switch is store and forward then you are not getting the ~ 230 nanoseconds port-to-port latency that you just spent $20K for.

Before we can dive into configuring buffer monitoring it would be beneficial to understand buffer blocks and how the ports are mapped. First there are 3 buffer blocks each containing 6MB worth of buffer space which gives us a total of 18MB shared buffer. Ports to these buffer blocks (3 x 6MB) are mapped as follows.

VRF Route Leaking from Global Routing Table

This blogtorial is in response to one of my reader's question "How do I use internet with VRF?". In this blogtorial we will see how we can leak routes between a VRF and the global routing table. In my opinion, route leaking between two VRFs is so much easier and straightforward than leaking between VRFs and the global routing table. However, in any case, let's get started and see one of 2 ways (perhaps there are more ways?) to leak routes between a VRF and the global routing table.

Here is the topology -- let's get started.

Entire config can be downloaded here.

BGP - Manipulating BGP communities

I like my desk to be organized..Do you? I suppose we can "desk" that conversation for another time :) Anyways if you are into being organized then BGP communities (32-bit value) are a great way to tag,organize your routes and make intelligent routing decisions. In this blogtorial I will show you briefly on how to set communities, delete communities (specific one or all of them), and appending to the existing communities.

Here is the topology we will be using.

Entire config can be downloaded here.

Configuring ASA NAT - Static 1-1 NAT

This blog will be a continuation of my previous blogtorial about NATs on ASA. One key difference to keep in mind between dynamic and static NAT is that Static NATs allow for the possibility for outside hosts to initiate connection.

Here is our base topology.

Let's get started!!!

Full configuration of the devices can be downloaded here.

Centralized log management - Syslog-ng, Phpsyslog-ng and MySQL back-end

Should I split this blogtorial in different parts and spoon feed it or are my readers man (or woman) enough to just take this blog as a whole and tear it apart? I decided my readers are hungry enough to handle the beast that centralized Syslog management really is. If you follow this blogtorial step by step you will have a fully functioning centralized Syslog management with a database back-end to store the logs, PHP web GUI front-end to view the logs and a solid Syslog server collecting all of the data.

Here is what we will cover in this blogtorial: 

• Installing and configuring MySQL - This will house the database and the necessary tables to organize the logs.  
• Installing httpd/apache and PHP libraries - This will be the webserver to house the Syslog-ng web GUI front-end (phpsyslog-ng)
• Installing phpsyslog-ng - This will be the web based GUI to view the logs.
• Installing syslog-ng - This will be the app collecting the logs and sending it to MySQL. 

Grab some chips, salsa and your favorite beer or a glass of wine because it is going to be a long one but in the end you should have a robust centralized log management system.

Configuring ASA NAT - Dynamic NAT - Object Network NAT

In this blogtorial we will see how we can take an inside LAN subnet and dynamically NAT using a POOL of addresses. Make sure the pool of addresses being used is reachable from the destination.

Here is the topology and the entire config can be downloaded here.

Objective:

Inside LAN (10.1.1.0/24) subnet should be dynamically translated to one of the IP addresses from the POOL (204.11.72.0/24). 

Configuring ASA Active/Standby failvover - ASA High Availability

When will this fail? How can we increase resiliency? How can we achieve close to 100% up-time? Questions one should be asking when designing any system. One way to answer these critical design questions -- redundancy!!. So in this blogtorial we will see how we can deploy two ASAs in a redundant design. When it comes to ASA high availability there are two modes: Active/Active where both ASAs are forwarding traffic and the other is Active/Standby where only the primary ASA is responsible for forwarding traffic and the other is in a hot-standby state waiting to forward traffic soon as the primary one fails. Each mode has its own set of pros and cons. Please review the Cisco website for a full list of Guidelines and Limitations.

Full configurations are located here so you can lab this up in GNS3.

With this in mind, let's get started on this topology.

TCP - TCP small window size causing latency

If you need a primer on window size and scaling you can check out my previous blogtorial that I posted a while ago. Today a client called and complained about latency. The basic premise was that they sent a New Order Single (FIX) and they didn't see the execution report for about 11 seconds. Application logs however showed that it was executed within sub microseconds and so why this 11 second delay? Obviously network equipment is not going to buffer the packets for multiple seconds. In order to troubleshoot this I turn to man's best friend (not dogs) -- but rather sniffers / packet captures ... perhaps I should have said nerd's best friend. :-) Once I started looking at the packet captures it all came together. I won't post the packet capture however a screen shot cant hurt.

Configuring ASDM for ASA on GNS3

ASDM is a GUI tool that you can use to configure ASAs. Although I prefer to use CLI, ASDM does come in handy from time to time. In this blogtorial we will see how to configure ASDM so that we can use it to configure ASAs running on GNS3. If you need a primer on how to get ASAs working on GNS3 then see my previous blogtorial.

Overview: 

• Download ASDM image
• Create a Loopback interface
• Connect it to the ASA
• Assign IP addresses
• Download TFTP server
• Copy ASDM image to the ASA
• Install ASA on the computer
• Connect to the ASA using ASDM 

GNS3 - Configuring ASA on GNS3

I generally use GNS3 to emulate routers but recently I have been using it to do some ASA labs. So here is how I emulate ASAs on GNS3.

First thing to take care of is acquiring the ASA images needed to emulate an ASA. Files needed can be downloaded here.

Once you've downloaded the files it is time to configure GNS3 to run the ASA. Follow the screenshots.

Dot1Q Tunnel / q-in-q Tunneling

Dot1Q tunnel or q-in-q tunneling is a technique generally used by service providers to extend customer's VLANs to different locations. It is done by encapsulating the customer VLAN inside another 802.1q encapsulation and because of this we must increase the system mtu on the service provider switches to something greater than 1500. In this blogtorial, we will take a look at a very simple topology with easy to follow configurations.

Here is the topology. 

Our objective: 

VLAN 10 in a Customer Site in California (SW1) needs to be extended to a Customer Site in Illinois (SW4).

BGP - Unicast NLRI to Multicast NLRI - translate-update cisco

I have been extremely busy with work, school and CCIE studies so I have not been able to post as much as I want to. There are numerous drafts ready to be posted so hopefully I'll have sometime to share it on here. Alright, enough with the excuses and let's get started. In this blogtorial we will see how we can take a unicast route received from a unicast BGP neighbor and install it in the multicast routing table and advertise it to MBGP peers using translate-update feature. Usually this is done for in-congruent BGP topology where a MBGP capable router peers with router that is incapable of MBGP.

Here is the topology.

The entire configs can be downloaded here so you can lab it up in GNS3.

BGP - Troubleshooting Lab 2

Objective

• R3 and R4 must have 5.5.5.5/32 and 6.6.6.6/32 in their BGP/routing table. 

Sounds simple..? Well how about you figure it out then :) .. Here is the topology. 

Restrictions

• Static routes should not be added anywhere!! In other words, get BGP to work properly. 

You should be able to figure out the issues with the following show commands below.  

BGP - Troubleshooting Lab

In this blogtorial, I will be posting configs of the routers and a few commands. Please leave me comments on what you think the issue maybe.

Consider this topology below. 

Objective: 

• 11.11.11.11/32 should be pingable from R4. 
• Static routes or artificial routes cannot be installed anywhere!! 

Hint -- There are three issues.

The entire config can be downloaded here so you can lab it up in GNS3.

BGP - inject-map - Conditional Route Injection

In my recently published posts we discussed how to aggregate addresses in BGP and the various optional parameters associated with the "aggregate-address" command. However, there could be instances where we may want to do just the opposite -- un-aggregate the addresses. In this blogtorial we will see how we can achieve this. Consider a very simple topology below and we'll dive right in.

BGP - aggregate-address suppress-map

In my previous blogtorial we utilized the aggregate-address command to summarize routes in BGP. In this blogtorial we will take a look at one of the optional parameters we can use with the aggregate-address command to manipulate routes. As you noticed the "aggregate-address <network> <netmask>" command actually advertised the aggregate route and the more specific routes.

 R2(config-router)#aggregate-address 11.11.11.0 255.255.255.0 ?  
  advertise-map Set condition to advertise attribute  
  as-set     Generate AS set path information  

  attribute-map Set attributes of aggregate  
  nlri      Nlri aggregate applies to  
  route-map   Set parameters of aggregate  
  summary-only  Filter more specific routes from updates  
  suppress-map  Conditionally filter more specific routes from updates  
  <cr>  

You could presumably use summary-only to filter all the more specific routes, however what if you wanted to suppress only a selected few of the more specific routes and advertise the rest of the specific routes along with the aggregate? This is where "suppress-map" or "advertise-map" comes to the rescue  If you only had a few specific routes to advertise but a lot of specific routes to suppress out of an aggregate then you would use the "advertise-map" to advertise just what you want. However, if you only had a few specific routes to suppress but more specific prefixes to advertise then you would use a "suppress-map" to suppress what you want and advertise the rest. See below for "suppress-map" in action.

BGP - aggregate-address as-set

In my previous blogtorial we utilized the aggregate-address command to summarize routes in BGP. In this blogtorial we will take a look at one of the optional parameters we can use with the aggregate-address command to manipulate routes.

 R2(config-router)#aggregate-address 11.11.11.0 255.255.255.0 ?  
  advertise-map Set condition to advertise attribute  
  as-set     Generate AS set path information  
  attribute-map Set attributes of aggregate  
  nlri      Nlri aggregate applies to  
  route-map   Set parameters of aggregate  
  summary-only  Filter more specific routes from updates  
  suppress-map  Conditionally filter more specific routes from updates  
  <cr>  

The optional parameter we will look at in this blogtorial is the as-set option. In my previous post we talked about "atomic-aggregate" and how the aggregate routes clears the AS_PATH information along with other attributes of the route such as no-export community. For example, let's take a look at the R3.

BGP - aggregate-address summary-only

In my previous blogtorial we utilized the aggregate-address command to summarize routes in BGP. In this blogtorial we will take a look at one of the optional parameters we can use with the aggregate-address command to manipulate routes.

 R2(config-router)#aggregate-address 11.11.11.0 255.255.255.0 ?  
  advertise-map Set condition to advertise attribute  
  as-set     Generate AS set path information  
  attribute-map Set attributes of aggregate  
  nlri      Nlri aggregate applies to  
  route-map   Set parameters of aggregate  
  summary-only  Filter more specific routes from updates  
  suppress-map  Conditionally filter more specific routes from updates  
  <cr>  




The optional parameter we will look at in this blogtorial is the summary-only option. In my previous post we saw that the aggregate-address command advertised the aggregate route along with the more specific prefixes. What if we wanted to suppress those more specific prefixes and advertise only the summary? summary-only command suppresses all of the more specific prefixes and advertises only the aggregate route to the neighbors.

BGP - Aggregation - aggregate-address

BGP Routing table of the Internet is enormous and its over 400K routes which puts a considerable amount of stress on the edge devices anytime there is a flap or a change in routing topology. Although there are mechanism in place to minimize the impact it is still in our best interest to keep the BGP routing table as small as possible. "aggregate-address" command is one of the ways in which we can combat the issues created by large BGP tables.

In this blogtorial we will look at the "aggregate-address" command and its options (in subsequent posts) to see how we can aggregate routes in BGP. Consider this simple 3 router topology. 

Complete configs can be downloaded here.

BGP Conditional Advertisements - IF Statements in BGP

I was chatting with a colleague of mine and I stated "I wish we could remember everything we read..." and he responded "If we could remember everything, then we wouldn't need to 'write it down' - thus depriving others of the opportunity to learn from our experience." - Mike O. Very well said Mike!! So back to my blogtorial -- How to do "if statements" with BGP? Or better known as "BGP Conditional Route Advertisement". The scenario is described in the topology below.

There are two different scenarios we will take a look at in this blogtorial.

• If 5.5.5.0/25 exists in the BGP table then advertise the aggregate address 5.5.5.0/24 to both ISPs. 
• or the opposite: only If 5.5.5.0/25 does not exist in the BGP table then advertise the aggregate address 5.5.5.0/24 to both ISPs.

Complete configs can be downloaded here.

Understanding IP Fragmentation

Can you imagine eating a whole steak in one bite? Probably not. You have to cut it in pieces and take it little at a time. How about a big project? You can't successfully complete it unless you split it up into multiple little projects and have deadlines for each one. Same goes for TCP segments. When Layer 4 TCP sends a big segment the Network Layer (Layer 3) Fra - gme - nts (fragments) the segments into smaller packets dictated by the MTU (Maximum Transmission Unit) of the outgoing interface. This brings up an interesting issue, how does the receiver know that the packets being received are a smaller chunk of a bigger datagram? How does the receiver know where to place these fragments? What happens if the fragments arrive out of order? Well in this blogtorial we will try and answer all of these questions and take a look at the fragmentation process and how the receiver puts it all together.

Consider this simple topology and let's get started.

Troubleshooting OSPF - Why routes are not in the routing table?

I was checking something out today in one of our core routers and came up on an issue that got me going around in circles for a while. So I decided it would be useful to post about it. I have studied the CCNP material and I can assure you that this is NOT covered in the CCNP material. However, if you are going for your CCIE then you should know this. 

Basic setup of the routers is as follows. 

Now the issue was R2 wouldn't learn the route 4.4.4.4/32 from R1 as I would have expected. Why?? Well it took me a while and I started to tear apart the OSPF database, cost, filtering, etc but nothing seemed out of the ordinary. So let's walk through the troubleshooting process and see how I got my answer.

BGP Community - No-export

I am continuing from my previous post on BGP communities. Since the topology is the same we will skip the interface and BGP configurations and dive right into what is no-export community, how to configure it and see how it affects routing updates.

Same topology as my previous blogtorial BGP Communities No-Advertise 

Complete configs can be downloaded here.

BGP Community No-Advertise

Much like a community in real life -- a group of "people" (usually) with common beliefs, A BGP community "is a group of destinations which share some common attribute" - RFC 1997.

There are numerous BGP communities, but basically it is translated and interpreted as a 32bit value. BGP communities are optional transitive attributes meaning they are not required and they can be passed on to other AS only if the router(s) decides to do so. 

These are the well-known communities. More being suggested and drafted in other RFCs.

INTERNET - By default all destinations are assigned this community. 

NO_EXPORT - Do not send the route to any eBGP neighbors. 

NO_EXPORT_SUBCONFED - Do not send the route to any sub-confederations. 

NO_ADVERTISE - Do not send the route to any neighbors (iBGP or eBGP).

LOCAL-AS - Do not send to anyone other than your LOCAL-AS (So this will prevent the router from sending to sub-confederations). 

ip multicast helper-map - Converting Broadcast and Multicast

If you work in the financial industry you are bound to encounter a situation where you will have an application that publishes udp data as broadcast and you will need to convert it to multicast -- maybe even back to broadcast if needed. This is very similar to the "ip helper address" because "ip helper-address" converts DHCP (udp) to unicast packets and sends it to the DHCP server.

Couple of very important roles of the routers in this setup are FHR and LHR. First-hop router that is connected directly to the source which is broadcasting the data and the Last-hop router that is connected to the destination subnet which can convert the multicast back to broadcast. 

Consider this simple topology and let's get started. 

Complete configs can be found here.

Understanding TCP SLOW START

In my opinion, TCP Slow Start is a rudimentary concept that should be mastered by any Network Engineer. It maybe a good idea to read through my previous post about Window Size and Scaling before continuing. To understand TCP Slow Start there are a few terms that we must be familiar with. 

TCP Slow Start - Congestion control mechanism which controls the growth of the sending rate.  

IW - Initial Window 

CWND - Congestion Window (The send window of the sender)

RWND - Receive Window (The window size of the receiver) 

SMSS - Sender Maximum Segment Size (Maximum amount of bytes that can be stuffed into a packet on the sender side) 

ISST - Initial Slow Start Threshold

SST - Slow Start Threshold

FlightSize - Amount of unacknowledged data that can be on the wire. This is usually set to the CWND however the actual formula is min(cwnd, rwnd). In most cases the CWND will be smaller than the RWND. 

Converting Multicast to Unicast

In my previous blogtorial "Converting unicast to multicast" we discussed how to convert unicast to multicast and what the benefits are. Suppose that for one reason or another we had to convert from multicast to unicast -- how would we go about doing that? Well in this blogtorial we'll do just that.

Let's get started.

In this scenario, R2 will convert multicast coming in on Fast1/0 and convert it to unicast on Fast1/1.

Complete configs can be found here.

Enabling TCP TimeStamp Linux and Windows

I've had a couple of comments from my previous TCP TimeStamp - Demystified post about how to enable TCP TimeStamp option on different operation systems. Therefore, this is a follow up post on how to enable TCP TimeStamp option on Linux and Windows.

Since I am pro-Linux, we will start with Linux. 

Linux:

In Linux, TCP TimeStamp is enabled by default. You can check this by running this command. 

Converting Unicast to Multicast

Imagine that you had to do a presentation in front of an audience. Would it make sense to stand on a stage and give the presentation or sit with each individual and repeat the presentation multiple times? Well obviously standing on the stage and giving the presentation once to many audiences is much more efficient – this concept in networking is referred to as one-to-many multicast. One stream distributed to multiple receivers is just one of the advantages of running multicast on your network. If there are many receivers listening for one stream, there is no need for the server to send the same stream to multiple destinations – just use multicast.

But what if the server is running an outdated software and can not do multicast? No worries, today we are going to see how we can convert unicast into multicast. This blogtorial assumes that the reader is somewhat familiar with multicast concepts and configurations such RP, IGMP and PIM.

Here is our objective:

• Software BuiBui residing on Comp1 is sending unicast data.
• Instead of Comp1 wasting bandwidth and sending the same data to Comp4 (C4) and Comp5 (C5), we need to convert the unicast to multicast data and forward it to Comp4 (C4) and Comp5 (C5).

Consider the topology below and let’s get started.

Complete configs can be found here.

TCP Timestamp - Demystified

What controls you? Some may argue that it’s their jobs, their bosses or their parents but only a selected few will say time. In reality time controls all of us -- what/when/how we do anything is all dictated by time. We all understand how important time is and TCP is no exception which is why RFC 1323 implements a TCP option known as “TIMESTAMP”. This optional 10 byte (1B Kind + 1B Length + 4B TSVal + 4B TSecr) field allows us to time stamp at the TCP level which can be used for TCP RTT calculations and PAWS (Protect Against Wrapped Sequence numbers). I was curious about the TCP TIMESTAMP option so I took a packet capture to investigate it further. However, I was unable to determine how to interpret the fields or how to utilize it to calculate TCP RTT. So I did what most of you would have done – googled it. Much to my dismay, I couldn't really find anything other than RFCs and sites stating the same definition that is already published in the RFC. So I decided to write a blogtorial about it so my fellow searchers/googlers can easily understand on how to utilize TSVal and TSecr to calculate the TCP RTT.

Timestamp packet capture can be downloaded here.

Let’s get started.

Show run/copy run start on Cisco takes forever?

Are you frustrated that "Show run", "Show run interface <something>, "write mem" or "copy run start" is taking forever on Cisco routers or switches? Rest assure that you are not alone, there are many of us who feel your pain. It even takes longer to return these results on a system containing numerous lines in the running configuration. Why? Because the NVGEN process needs to query everything about your router and then output these results. It's painstakingly long especially when you are in the middle of troubleshooting something. 30 seconds may as well be 30 years. Not to mention that it almost results in a CPU spike sending false alerts everywhere and to everyone. To return the results of the aforementioned commands much faster enable the configuration performance enhancement by typing "parser config cache interface" in the global config mode.

Understanding TCP Window Size / Window Scaling

The easiest way to understand TCP Window size is to observe two people having a conversation. Every so often, the talker will wait for the listener to acknowledge that they have heard everything up to that point. Once the listener acknowledges then the talker begins to talk again. The amount of words spoken before waiting for an acknowledgement from the listener is much like the TCP window size. The official definition of the window size is "the amount of octets that can be transmitted without receiving an acknowledgement from the other side".

BGP - Maximum-path - hidden command

Two is better than one -- More is better or is it? Well I guess we can ruminate on semantics but I think we would rather see how we can implement equal cost multi-path (ECMP) routing using eBGP. By default BGP only installs THE BEST PATH after going through the BGP Bestpath selection algorithm (Cisco).

Consider the simple topology and let's dive in.

Complete config can be found here.

BGP - Configuring mBGP (Multicast BGP)

Do you influence or do you manipulate people to get what you want? What exactly is the difference? I mean in the end, you end up getting what you want right? Wrong, The difference lies in loyalty, if you influence someone then you get them to want to do what you want them to do. If you coerce or manipulate you may achieve short term success, however in the long term you will loose their trust and they will perceive you as someone who violated their trust rather than a leader. Therefore a great leader will influence others to achieve long term success. Although, sometimes you have no choice but to manipulate, for example my wife wants to influence me to do the dishes. Well that's just not going to happen -- she's never going to convince me that I want to do dishes, so she settles for manipulation :). Now that we've gotten my daily tidbit out of the way let's get into the topic at hand which is -- mBGP (Multicast BGP) not to be confused with Multi-protocol BGP which is used by MPLS, IPv6 etc.

Imagine you have Unicast Servers and Multicast Sources on the same network. Furthermore, imagine that you have 2 paths to reach that network, however you want to separate Unicast and Multicast traffic between different paths. How can we achieve this? Well there are a couple of options.

One, we can add static mroutes downstream or you could run mBGP and since the title of this blogtorial is configuring mBGP we'll choose the latter option. 

Consider this simple topology and let's get started. 

Complete configs can be found here.

BGP - Neighbor Allowas-in

My apologies -- It's been a while since I have posted, although I been pondering upon a few topics. I have been extremely busy with work, family, studies... -- Yes that's right I am going for my M.S in C.S :). In any case, enough about my boring personal life and excuses lets get into this action packed blogtorial. Here is a simple problem that I ran into the other night. I get a call from my colleague asking me if I would take a look and see why the routes from an eBGP neighbor are not being put into the routing table. After a couple of minutes of troubleshooting I pin-pointed the issue to a BGP loop prevention mechanism "called AS_Path: If you see your own AS in an update, drop it". Fortunately, there is a way to get around this in case we need to.

RFC 1771 

Section 9.3 states "If the local AS appears in the AS path of the new route being considered, then that new route cannot be viewed as better than any other route.  If such a route were ever used, a routing loop    would result."


To illustrate this in action, let's follow this simple topology.

Complete configurations can be found here.