Wednesday, May 16, 2018

Configuring OTV - OTV Configuration and Verification

In this blogtorial, I will go through OTV configuration on virtualized routers on Eve-NG and go through a few verification commands. I will also touch on key OTV terminologies and design considerations.

If you need any assistance on how to get Eve-NG up and running on google compute, please see my previous blogtorial - Configuring Eve-NG on Google Compute Engine

Here is the topology. Our goal is to establish layer 2 connectivity between 5.5.5.5 (ESXi-West-1) and 5.5.5.10 (ESXi-East-1). One of the key advantages of using OTV versus other technologies such as VPLS is that spanning-tree domain will not be extended between the sites. Other benefits include ARP suppression, Unknown flooding suppression, Hardware acceleration, and multi path tunneling.



First thing we will configure is the transport network or the underlay. We will start with the devices in the East Site.

IP Transport / Underlay

CORE-East-1

I had to bring up two interfaces between the virtual core and edge because I couldn't configure just one port as trunk. In real devices this should not be necessary.

 hostname CORE-East-1  
 !  
 ip cef  
 !  
 spanning-tree mode rapid-pvst  
 spanning-tree extend system-id  
 spanning-tree vlan 5 priority 4096  
 !  
 interface Ethernet0/0  
  description ESXI-VMS  
  switchport access vlan 5  
  switchport mode access  
 !  
 interface Ethernet0/2  
  description interface_to_extend_the_vlan  
  switchport mode access  
  switchport access vlan 5  
 !  
 interface Ethernet0/3  
  description interface_to_bring_up_OTV_site_vlan  
  switchport access vlan 56  
  switchport mode access  
 !  
 interface Vlan5  
  description ESXI-VMS  
  ip address 5.5.5.1 255.255.255.0  
 !  

OTV-East-Edge-1 

For now we will configure just the IP interfaces and enable OSPF so we have Layer 3 IP reachability between the sites. Notice that pim is not enabled on the interface connecting to the ISN only IGMP. PIM is only needed on the ISN and DCI.

 license grace-period  
 !  
 hostname OTV-East-Edge-1  
 !  
 ip pim rp-address 2.2.2.1 group-list 224.0.0.0/4
 ip pim ssm range 232.0.0.0/8
 !
 !!--Enable the required features
 feature ospf  
 feature otv 
 feature pim
 !  
 !!--Create the necessary VLANs
 vlan 5  
  name ESXI-VMS  
 vlan 56  
  name OTV_SITE_VLAN  
 !  
 interface Ethernet2/1  
  ip address 3.3.3.1/30  
 !!--IGMP version 3 (SSM) is required for OTV 
  ip igmp version 3 
  no shutdown  
 !  
 interface Ethernet2/2  
  switchport  
  switchport access vlan 5  
  no shutdown  
 !  
 interface Ethernet2/3  
  switchport  
  switchport access vlan 56  
  no shutdown  
 !  
 !!--Enabling OSPF so we have unicast reachability
 router ospf 1  
  network 0.0.0.0/0 area 0.0.0.0  

ISN-East-1 

Now let's get the Data Center Edge configured. It's considered best practices to bring the DCI (Data Center Interconnects) into its own set of routers, therefore in this topology I am bringing them into a pair of ISN routers (InterSite Network Routers), but technically they could have gone directly into the Core Routers (perhaps in its own VDC).

 hostname ISN-East-1  
 !  
 ip multicast-routing  
 !  
 ip cef  
 !  
 interface Ethernet0/0  
  description DCI - Data Center Interconnect  
  no switchport  
  ip address 2.2.2.2 255.255.255.252  
  ip pim sparse-mode  
 !  
 interface Ethernet0/1  
  description Connected_TO_OTV_EDGE  
  no switchport  
  ip pim sparse-mode
  ip address 3.3.3.2 255.255.255.252  
  ip igmp version 3  
 !  
 router ospf 1  
  network 0.0.0.0 255.255.255.255 area 0  
 !  
 ip pim rp-address 2.2.2.1  

Now let's get the West Site configured the same as the East Site and then we will move on to the configurations pertaining to the overlay network and configure OTV.

Core-West-1 

  hostname CORE-West-1   
  !   
  ip cef   
  !   
  spanning-tree mode rapid-pvst   
  spanning-tree extend system-id   
  spanning-tree vlan 5 priority 4096   
  !   
  interface Ethernet0/0   
  description ESXI-VMS   
  switchport access vlan 5   
  switchport mode access   
  !   
  interface Ethernet0/2   
  description interface_to_extend_the_vlan   
  switchport mode access   
  switchport access vlan 5   
  !   
  interface Ethernet0/3   
  description interface_to_bring_up_OTV_site_vlan   
  switchport access vlan 55  
  switchport mode access   
  !   
  interface Vlan5   
  description ESXI-VMS   
  ip address 5.5.5.1 255.255.255.0   
  !   

OTV-West-Edge-1 
  license grace-period   
  !   
  hostname OTV-West-Edge-1   
  !   
  !!--Enable the required features  
  feature ospf   
  feature otv 
  feature pim  
  !   
  ip pim rp-address 2.2.2.1 group-list 224.0.0.0/4
  ip pim ssm range 232.0.0.0/8
  !
  !!--Create the necessary VLANs  
  vlan 5   
  name ESXI-VMS   
  vlan 55   
  name OTV_SITE_VLAN   
  !   
  interface Ethernet2/1   
  ip address 1.1.1.1/30   
  !!--IGMP version 3 (SSM) is required for OTV   
  ip igmp version 3   
  no shutdown   
  !   
  interface Ethernet2/2   
  switchport   
  switchport access vlan 5   
  no shutdown   
  !   
  interface Ethernet2/3   
  switchport   
  switchport access vlan 56   
  no shutdown   
  !   
  !!--Enabling OSPF so we have unicast reachability  
  router ospf 1   
  network 0.0.0.0/0 area 0.0.0.0   

ISN-West-1
 hostname ISN-West-1   
  !   
  ip multicast-routing   
  !   
  ip cef   
  !   
  interface Ethernet0/0   
  description DCI - Data Center Interconnect   
  no switchport   
  ip address 2.2.2.1 255.255.255.252   
  ip pim sparse-mode   
  !   
  interface Ethernet0/1   
  description Connected_TO_OTV_EDGE   
  no switchport   
  ip address 1.1.1.2 255.255.255.252   
  ip pim sparse-mode   
  ip igmp version 3   
  !   
  router ospf 1   
  network 0.0.0.0 255.255.255.255 area 0   
  !   
  ip pim rp-address 2.2.2.1   

At this point, we should have full IP unicast reachability between the devices. Multicast routing and PIM enabled on the underlay network.
 OTV-West-Edge-1# sh ip int brief  
 IP Interface Status for VRF "default"(1)  
 Interface      IP Address   Interface Status  
 Ethernet2/1     1.1.1.1     protocol-up/link-up/admin-up  
 OTV-West-Edge-1# ping 1.1.1.2  
 PING 1.1.1.2 (1.1.1.2): 56 data bytes  
 64 bytes from 1.1.1.2: icmp_seq=0 ttl=254 time=1.54 ms  
 64 bytes from 1.1.1.2: icmp_seq=1 ttl=254 time=1.406 ms  
 64 bytes from 1.1.1.2: icmp_seq=2 ttl=254 time=1.068 ms  
 64 bytes from 1.1.1.2: icmp_seq=3 ttl=254 time=1.016 ms  
 64 bytes from 1.1.1.2: icmp_seq=4 ttl=254 time=1.032 ms  
 --- 1.1.1.2 ping statistics ---  
 5 packets transmitted, 5 packets received, 0.00% packet loss  
 round-trip min/avg/max = 1.016/1.212/1.54 ms  
 OTV-West-Edge-1# ping 3.3.3.2  
 PING 3.3.3.2 (3.3.3.2): 56 data bytes  
 64 bytes from 3.3.3.2: icmp_seq=0 ttl=253 time=1.781 ms  
 64 bytes from 3.3.3.2: icmp_seq=1 ttl=253 time=1.67 ms  
 64 bytes from 3.3.3.2: icmp_seq=2 ttl=253 time=1.534 ms  
 64 bytes from 3.3.3.2: icmp_seq=3 ttl=253 time=1.574 ms  
 64 bytes from 3.3.3.2: icmp_seq=4 ttl=253 time=1.62 ms  
 --- 3.3.3.2 ping statistics ---  
 5 packets transmitted, 5 packets received, 0.00% packet loss  
 round-trip min/avg/max = 1.534/1.635/1.781 ms  
 OTV-West-Edge-1# show ip route  
 IP Route Table for VRF "default"  
 '*' denotes best ucast next-hop  
 '**' denotes best mcast next-hop  
 '[x/y]' denotes [preference/metric]  
 1.1.1.0/30, ubest/mbest: 1/0, attached  
   *via 1.1.1.1, Eth2/1, [0/0], 03:45:40, direct  
 1.1.1.1/32, ubest/mbest: 1/0, attached  
   *via 1.1.1.1, Eth2/1, [0/0], 03:45:40, local  
 2.2.2.0/30, ubest/mbest: 1/0  
   *via 1.1.1.2, Eth2/1, [110/50], 03:45:28, ospf-1, intra  
 3.3.3.0/30, ubest/mbest: 1/0  
   *via 1.1.1.2, Eth2/1, [110/60], 03:45:28, ospf-1, intra  

Overlay Transport Virtualization Configurations

High-level

  • Enable the feature
  • Create and specify site VLAN 
  • Configure OTV Site ID *I couldn't configure this due to the NXOS version. This is needed when you have multiple edge devices within the same site.*
  • Specify join interface *Make sure to enable IGMP v3 on the join interface*
  • Specify control group *Used to dynamically discover remote neighbors*
  • Specify data group *Used for instance, when you’re tunneling multicast traffic over OTV in the data plane. Needs to be SSM and thus igmp version 3*
  • Specify extend VLANs *Particular VLANs to be extended to other domain*
  • Enable jumbo frames (optional but highly recommended) 
OTV-East-Edge-1 OTV Configurations

 hostname OTV-East-Edge-1  
 !  
 vlan 56  
  name OTV_SITE_VLAN  
 otv site-vlan 56
 !  
 interface Overlay1  
  otv join-interface Ethernet2/1  
  otv control-group 233.1.1.1  
  otv data-group 233.1.1.2/32  !!-- Should be SSM
  otv extend-vlan 5  
  no shutdown  

OTV-West-Edge-1 OTV Configurations

 hostname OTV-West-Edge-1  
 !  
 vlan 55  
  name OTV_SITE_VLAN  
 otv site-vlan 55  
 !  
 interface Overlay1  
  otv join-interface Ethernet2/1  
  otv control-group 233.1.1.1  
  otv data-group 233.1.1.2/32  
  otv extend-vlan 5  
  no shutdown  

Verifying OTV

show otv site detail
show otv adjacency
show otv isis adjacency
show otv isis site
show otv route
show otv vlan-mapping
show otv internal adjacency

 OTV-West-Edge-1# show otv overlay 1  
 OTV Overlay Information  
 Overlay interface Overlay1  
  VPN name      : Overlay1  
  VPN state      : UP  
  Extended vlans   : 5 (Total:1)  
  Control group    : 233.1.1.1  
  Data group range(s) : 233.1.1.2/32  
  Join interface(s)  : Eth2/1 (1.1.1.1)  
  Site vlan      : 55 (up)  

 OTV-West-Edge-1# show otv site detail  
 Site Adjacency Information (Site-VLAN: 55) (* - this device)  
 Overlay1 Site-Local Adjacencies (Count: 1)  
  Hostname             System-ID   Up Time  Ordinal  
  -------------------------------- -------------- --------- ----------  
 * OTV-West-Edge-1         5000.0001.0006 04:48:30 0  

 OTV-West-Edge-1# show otv isis adjacency detail  
 OTV-IS-IS process: default VPN: Overlay1  
 OTV-IS-IS adjacency database:  
 System ID    SNPA      Level State Hold Time Interface  
 5000.0002.0006 5000.0002.0001 1   UP   00:00:06  Overlay1  
  Up/Down transitions: 1, Last transition: 00:05:24 ago  
  Circuit Type: L1  
  IPv4 Address: 3.3.3.1  
  IPv6 Address: 0::  
  Circuit ID: 5000.0002.0006.01, Priority: 64  
 !!--Unicast adjacency is possible by deploying an adjacency server 
 !!--but that is outside the scope of this blogtorial
  Adjacency Type: multicast  

 OTV-West-Edge-1# show otv isis ?  
  <CR>  
  >            Redirect it to a file  
  >>            Redirect it to a file in append mode  
  active-source      Display IS-IS Active-source information  
  adjacency        Display IS-IS adjacency information  
  database         Display IS-IS database information  
  default         Routing process tag  
  event-history      Display IS-IS event history  
  hostname         Display IS-IS hostname table information  
  interface        Display IS-IS interface information  
  internal         Show internal information  
  ip            Display IS-IS IPv4 information  
  mac           Display IS-IS MAC information  
  protocol         Display IS-IS process information  
  redistribute       Display IS-IS redistribute information  
  route          Display IS-IS route information  
  route-map        Display IS-IS route-map information  
  rrm           Display IS-IS Retransmit-Routing-Message information  
  site           Display IS-IS OTV site information  
  spf-log         Display IS-IS SPF information  
  srm           Display IS-IS Send-Routing-Message information  
  ssn           Display IS-IS Send-Sequence-Number information  
  statistics        Display IS-IS protocol statistics  

 OTV-West-Edge-1# show otv route  
 OTV Unicast MAC Routing Table For Overlay1  
 VLAN MAC-Address   Metric Uptime  Owner   Next-hop(s)  
 ---- -------------- ------ -------- --------- -----------  
   5 0050.0000.0500 11   00:00:53 site    Ethernet2/2       !!--Mac address of eth0 on ESXi-West-1
   5 0050.0000.0600 42   00:06:31 overlay  OTV-East-Edge-1  !!--Mac address of eth0 on ESXi-East-1
   5 aabb.cc00.3020 11   00:00:53 site    Ethernet2/2  
   5 aabb.cc00.4020 42   00:06:31 overlay  OTV-East-Edge-1  

 OTV-West-Edge-1# show otv internal state  
 OTV Server State:  
 Platform: Software  Restart type: Stateless  SUP state: Active  
 HA state:  
   Recovery           N/A  
   SDB URI            volatile:/dev/shm/otv_sdb_info  
   PSS URI            volatile:/dev/shm/otv_data  
 Server registration info:  
   clis orib ip ipv6 ifmgr pm udp l3vm igmp mrib  
 Servers UP:  
   clis orib ip ipv6 ifmgr pm udp l3vm igmp mrib  
 Global flags:  
   Recovery DONE flag      FALSE  
   Blackhole STP Max Config Set False  
   Pending messages count    0  
   ORIB buffer count       0  
   CLI processing        DONE  
   Have UFDM VLAN updates    Yes  
   Have UFDM adjacency updates  Yes  
   Worker thread         Idle  
 Timer info:  
   Next age time         00:10:51  
   Next ARP age time       00:10:51  
   Heartbeat timer  running, 00:00:02 seconds remaining  
   AED timer     Not Running  

Ping from ESXi-West-1 to ESXi-East-1


Wireshark Capture of the Ping taken on (ethernet 0/0) from ISN-West-1 and therefore the ICMP packets are OTV encapsulated.

As you can see the ICMP packet is the payload inside MPLS inside GRE inside IP.
  • The outer IP header contains the OTV-Edge devices. 
  • The MPLS label 37 corresponds to VLAN 5 that is being extended. Each OTV extended VLAN will have its own label. 
  • The original IP header contains the ESXI-West and ESXI-East IP for the ICMP 


External Resources

OTV Configuration Guide
OTV Verification Commands

Many more articles to come so ....

Please subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.