High CPU on Nexus 3K - Solved

In this blogtorial, I will demonstrate how I used 'ethanalyzer' on a Cisco Nexus 3K to solve an intermittent issue -- random adjacency drops of various routing protocols. Before we get into the details. let me first share with you on how I got involved in this troubleshooting to begin with. My good friend and colleague "BGP" bill aka self proclaimed "Multicast Guru" 😊 turns around says "Hey Weezy, you are a CCIE right? I have an open case with Cisco for a month why don't you just solve this issue?". Hmmmm ... above all else, one thing I've learned from the journey of becoming a CCIE is that I know very little. There is so much more to learn than you can imagine. In any case, I thought it was a noble challenge and as a bonus it peaked my curiosity. So Bill and I started chatting about the issue and as we were going through the motions, he gave me a great piece of information that would eventually steer me down the path to solving the issue, "Pete reported that he saw HIGH CPU usage compared to other similar routers".

So my 2 questions were:

TCP SACK Demystified

In this blogtorial, we will briefly explore TCP SACK (Selective Acknowledgement) option and the benefits of using TCP Selective Acknowledgement. TCP SACK is negotiated in the initial 3-way TCP handshake and both parties must agree to use this option or TCP SACK will not be permitted.

Let's walk through an example and observe what happens when SACK is not enabled.

BGP Wedgies - Demystified

Usually when we configure BGP we expect the network to converge correctly after all the peerings come up, however under rare circumstances this is not the case. In this blogtorial, we will explore one such corner case scenario in which depending on the order of operation, the BGP topology can end up in a unpredictable topology or a BGP Wedgie.

Here is the topology.

As usual let's go ahead and get the routers configured and give this topology a wedgie.

Troubleshooting eBGP peering

Simple .... BGP topology ... can you get eBGP up and running between R1 and R2?

Objective:

Establish successful and stable eBGP relationship between R1 and R2. With the current start-up config, BGP is IDLE. 

Topology diagram and startup config are posted below.

Configuring OSPF - Summarization Lab

As the CCIE Lab exam requires you to the think outside the box a bit, I will do the same on this lab. Here is a simple topology and the basic start up configuration.

Objectives:

• Summarize 192.168.1.2/32 from Area 2 as 192.168.1.0/24 into area 0.

Restrictions:

• Do not use area-range command or summary-address to summarize the routes into Area 0.
• R3 and other routers in Area 0 or any new additional areas should only have the summary address to reach 192.168.1.2. 
• If prefix 192.168.1.2/32 is withdrawn (i.e shutdown loopback0 on R2) from R1 OSPF database then 192.168.1.0/24 should not be summarized/advertised into area 0.

Cisco BUG - PIM join does not make it to CPU

Cisco BUG

CSCue52328 Bug Details

PIM joins for some groups not processed at CPU level

Symptom: Under very rare conditions PIM joins from a downstream switch are not forwarded my mcastfwd to PIM to be processed. This has been seen under the following conditions a) Downstream device sending PIM joins is a Arista switch b) There is a local receiver sending IGMP joins for the same group for which PIM joins are received c) Exact trigger is unknown at this point, could be the remote receivers (which result in PIM joins) flapping

Troubleshooting OSPF - LAB 1

Here is a quick lab I put together to really test your knowledge about OSPF Path selection and some of the hidden rules behind the path selection.

Simple topology below.

Objective: 

• R3 should choose R2 over Gi1.23 to reach R1 loopback 150.1.1.1/32. Imagine that this is a 40Gbit link where as Gi1.13 is just a 100Mb link. Currently R3 is choosing Gi1.13 to reach R1 loopback 150.1.1.1/32. 

Restrictions:

• Do not configure any new routing protocols.
• Do not change Area numbers.
• Do not create any NEW areas. 
• Do not change OSPF cost values
• R3 should choose R1 with OSPF as the routing protocol so no STATIC routes etc. 

Configuration excerpts below so you can configure it in your lab.

EIGRP - Troubleshooting LAB 1

Objective:

• When Gig2 is shutdown on R1, metric for the route to reach Loopback1 should not change on R2.  

Here is a simple topology and let's get started.

BGP - Troubleshooting Lab 3

Objective: 

• Establish iBGP between R1 and R2. 

Restrictions: 

• Cannot change AS numbers on any of the routers.
• No other static routes can be added 
• No interface changes
• No tunnels

Topology below:

I came across this issue the other day and it took me a few minutes to figure it out. So let's see how y'all do. 

Post your comments below on how you would solve this issue. 

Linux RP_Filter RPF_Check

"Network Engineer T-Shirt ... Fixing your network one misconfigured server at a time" -- I saw that shirt today online and I should have purchased it :). I came across an issue today where multicast was being received on the Server on eth1 (confirmed by tcpdump) but it was not showing up in the application. After checking out PIM/IGMP/Mroutes/Routers/Switches I started checking out the Server because the network looked like it was configured correctly.

Convergence between SVI vs Routed Interface - Cisco 3548 NXOS

Convergence of SVI vs. Routed link on a Cisco 3548 Nexus running A1.1c

Link failures were simulated by “shut” on the remote interface.

SVI – L2 VLAN

 switch#  
 2014 Aug 28 10:04:06.348524 switch %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/9 is down (Link failure)  
 2014 Aug 28 10:04:06.478105 urib: "direct": 1.1.1.0/24 no more next hops  
 2014 Aug 28 10:04:06.478419 urib: 1.1.1.0/24 Deleting & Freeing  
 2014 Aug 28 10:04:06.479344 urib: "local": 1.1.1.1/32 no more next hops  
 2014 Aug 28 10:04:06.479618 urib: 1.1.1.1/32 Deleting & Freeing  
 2014 Aug 28 10:04:06.479954 urib: "broadcast": 1.1.1.255/32 no more next hops  
 2014 Aug 28 10:04:06.480215 urib: 1.1.1.255/32 Deleting & Freeing  
 2014 Aug 28 10:04:06.480542 urib: "broadcast": 1.1.1.0/32 no more next hops  
 2014 Aug 28 10:04:06.480951 urib: 1.1.1.0/32 Deleting & Freeing  
 2014 Aug 28 10:04:06.696703 urib: "am": 1.1.1.2/32 no more next hops  
 2014 Aug 28 10:04:06.697117 urib: 1.1.1.2/32 Deleting & Freeing  

Route deletion = 478419 – 348524 =  129895us

Checkpoint Firewall FTP issues - 'quote password' or Account command ('ACCT')

I usually post stuff about Cisco, but recently I got exposure to Checkpoint so I am adding Checkpoint to my library. I am currently working on a 3 part series on "how-to" install virtual Checkpoint Firewalls on a Linux KVM hypervisor. A quick search on Google reveals 0 posts related to this, so mine might be the first "how-to" on this topic. Stay tuned for more Checkpoint related posts in the near future.

Here is my first post on Checkpoint regarding the FTP issue "Wrong username or password". I will try to keep it short!!  

Trying to FTP from command prompt on a windows machine.

 Command line ftp:  
 ftp ###.###.###.###  
 Connected to ###.###.###.### (###.###.###.###).  
 220 Check Point FireWall-1 Secure FTP server running on XXXXXX  
 Name (###.###.###.###:XXXXX): usernamejoe  
 331 password: you can use password@password  
 Password:  
 200 password: you can use 'quote password' or Account command ('ACCT')  
 ftp> pass  
 Passive mode off.  
 ftp> ls  
 421-Access denied - wrong user name or password   
 421 aborted  

Cisco Nexus ERROR MSG - SFP Validation Failed

I came across the other day when I was bringing up a new connection on a Cisco 3548 Nexus.

"SFP validation failed"

This happens when the speed is set on an interface and the SFP does not support it. To fix it get into interface config mode and type.

 switchport host
 shut
 no speed
 no duplex
 no shut

Many more articles to come so ....

Please subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.

Cannot SSH into ASA - ssh_exchange_identification: Connection closed by remote host

I usually write quiet a bit but I figured I would keep track of these small error messages also. So here it goes .. last week I ran across an issue on one of the ASAs. Here is the error message.

"ssh_exchange_identification: Connection closed by remote host"

How to fix it?

Telnet or console into the ASA and type this in the global config mode.

ASA(config)#crypto key generate rsa modules 1024

Save the config (wr).

That should resolve your issue.

Many more articles to come so ....

Please subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.

Troubleshooting OSPF - Why routes are not in the routing table?

I was checking something out today in one of our core routers and came up on an issue that got me going around in circles for a while. So I decided it would be useful to post about it. I have studied the CCNP material and I can assure you that this is NOT covered in the CCNP material. However, if you are going for your CCIE then you should know this. 

Basic setup of the routers is as follows. 

Now the issue was R2 wouldn't learn the route 4.4.4.4/32 from R1 as I would have expected. Why?? Well it took me a while and I started to tear apart the OSPF database, cost, filtering, etc but nothing seemed out of the ordinary. So let's walk through the troubleshooting process and see how I got my answer.

BGP - Neighbor Allowas-in

My apologies -- It's been a while since I have posted, although I been pondering upon a few topics. I have been extremely busy with work, family, studies... -- Yes that's right I am going for my M.S in C.S :). In any case, enough about my boring personal life and excuses lets get into this action packed blogtorial. Here is a simple problem that I ran into the other night. I get a call from my colleague asking me if I would take a look and see why the routes from an eBGP neighbor are not being put into the routing table. After a couple of minutes of troubleshooting I pin-pointed the issue to a BGP loop prevention mechanism "called AS_Path: If you see your own AS in an update, drop it". Fortunately, there is a way to get around this in case we need to.

RFC 1771 

Section 9.3 states "If the local AS appears in the AS path of the new route being considered, then that new route cannot be viewed as better than any other route.  If such a route were ever used, a routing loop    would result."


To illustrate this in action, let's follow this simple topology.

Complete configurations can be found here.

Troubleshooting - Loop-back detected - %PM-4-ERR_DISABLE

Recently, I was tasked with troubleshooting a weird problem where one of the interfaces was err-disabled. Fun Monday mornings :) and sifting through the logs I came up on this error message. 

%PM-4-ERR_DISABLE: loopback error detected on Fa1/0/20, putting Fa1/0/20 in err-disable state

%ETHCNTR-3-LOOP_BACK_DETECTED: Loop-back detected on FastEthernet1/0/20.

Turns out that this was because of an interface somewhere in our network was configured incorrectly which then created a loop in our network. The STP topology in our network corrected/recovered itself within a few seconds and fa1/0/20 should have "err-disabled recovered", however "errdisable recovery cause loopback" was not configured. 

Basically this error message states that the keepalives which are on by default are looping back to the source interface.  

For example, if a keepalive was sent on fa1/0/20 and if that keepalive packet loops around the network and shows up at the source interface then that must mean that there is a loop in our network and the interface which sent out the keepalive will be err-disable.  

Notice that the keepalive are set to 10sec by default on all interfaces. 

Workarounds: 

• Fix the underlying misconfigured interface downstream and/or upstream.
• Disable keepalives on interfaces. 

Many more articles to come so stay tuned. "Join this site" on the right / click +1 below ------->