Thursday, December 18, 2014

CCIE v5 INE Home Lab - Part 3 - Console to Routers / Automating scripts

Don't you love it when it all comes together and all the puzzle pieces fall into place. Well I hope that this blogtorial does exactly that. In part 1 and part 2, we laid the groundwork and the foundation to successfully get a lab up and running and in this blogtorial we are going to finish it off with:
  • How to console into the virtual routers? 
  • How to get all initial configs prep'd so we can have concentrate on doing the labs rather than setting up the labs?
  • How to automate the loading of the initial configs? 
I know it's a lot to cover in one post, but I also didn't want to breakup this series into too many parts. So without further adieu, let's get started and I will try to streamline it and post as much screenshots as possible. Here is the overview and where we will be spending most of the blogtorial.


I normally would do it for free but I have had tons of requests and questions regarding the lab setup and scripts. So for a nominal fee I will configure your entire VMWare ESXi server / all the routers / the Linux VMs / auto loading scripts. More importantly it includes an easy to use WEB GUI to load the config files. Contact me via arwinr@gmail.com if you are interested. 

Screenshot of the WEB GUI. 

Note: Thomas Kjær pointed out in one of the comments below that ESXi version 6 recently released does not limit 4 serial per Linux VM. Therefore, instead of creating 4 linux VM you could just do it all in one Linux VM, but you would have to edit the scripts accordingly. 


How to console into the virtual routers?

Console access to the routers is accomplished by installing Linux virtual machines on the ESXi, adding serial adapters to the Linux VMs and redirecting the PIPEs (see part 2 on how to create the PIPE) to the Linux VM's serial adapters. Now this might sound complicated to some folks but even the novice Linux users can follow this tutorial and accomplish the task. Again, the reason I am doing it this way is because VMware ESXi "Serial-Over-Network" that most tutorials on the Internet talks about is only valid for 60 days as a demo or you have to get the enterprise license for ESXi which is over $2000.

Only draw back with PIPEs and Serial Adapters is that you can only add 4 Serial adapters to any particular Linux VM so you will have to create [ ceil ( total number of routers / 4) ] number of Linux VMs. So in our case ceil (10 routers / 4) is 3, so we would need a total of 3 Linux VMs to console into 10 routers. I'll briefly walk through on how to install a Linux machine as this is not very hard to do. As a side note you could use Windows VM and use HyperTerm or putty to serial into your routers, I am using Linux because it gives me more flexibility.

First download your favorite linux distro. I used CentOS 7 minimal install which can be downloaded from http://isoredirect.centos.org/centos/7/isos/x86_64/

Just go into any one of those those directories listed above and download the minimal install.

Create a new virtual machine.







Right click on the newly created Linux Machine and alter the settings.
  • Ram only needs to be 512MB
  • Name the VMs sniffer1, sniffer2, sniffer3 and so on as it will be important later-on in "automating scripts".
Add the serial adapters
This is how we connect to the PIPEs we created in PART 2 from the Linux VMs. So when you are all done you will have CSR1.pipe, CSR2.pipe, CSR3.pipe, CSR4.pipe connected to Linux VM sniffer1.  CSR5.pipe, CSR6.pipe, CSR7.pipe, and CSR8.pipe connected to Linux VM sniffer2. And finally CSR9.pipe, CSR10.pipe connected to Linux VM sniffer3. 

Screenshot below as an example on how to connect PIPEs to Linux VMs. 

Right click on the appropriate Linux VM and click Edit settings. For example, here I am adding CSR4.pipe to Linux VM sniffer1. 





Installing w/ Linux ISO

You need to SCP the Linux ISO file you downloaded to the VMware ESXi so you can install it as a Linux VM. I used WINSCP and connected to the ESXi and uploaded the ISO to the vmimages folder. Now I can use it to boot from it and install the VM. See below.

Right click on the Linux VM and go to Edit settings.

Now power on the Linux and follow the on-screen instructions to install Linux on the VM. 

Once the installation completes you can login into the first Linux VM as root/password you created during install and use minicom to console into routers.

First install minicom and expect by 'yum install -y minicom expect'.

minicom -D /dev/ttyS0 -- This will get you console to R1
minicom -D /dev/ttyS1 -- This will get you console to R2
minicom -D /dev/ttyS2 -- This will get you console to R3
minicom -D /dev/ttyS3 -- This will get you console to R4

To get out of console you can do CTRL A then let go of CTRL and press Z press X and Enter.

You can repeat the process above to create the rest of the Linux VM and make sure to modify things like the Serial Adapter pipe name for example CSR5.pipe, CSR6.pipe and so on.

Prepping the Config files

Why do we need to prep the config files? Well the initial config files from INE are just a template and it's not configured to fit exactly to what we are doing so we need to edit all the config files at once to get it working in our environment.

First thing we need to do is download the INE initial config files. If you have access to INE then you should know how to do this.

Then create a folder in the first Linux VM machine you created. I'll call my first Linux machine "sniffer 1".

mkdir /root/ine/

Copy all the INE initial config files into this directory using winscp or some other scp software. Since the files I downloaded and extracted are in the windows format I had to get rid of the ^M in all the .txt files. We can accomplish this in a sed one liner.

 cd /root/ine  
 find . -type f -exec sed -i 's/^M//' {} \;  

To enter ^M, type CTRL-V, then CTRL-M. That is, hold down the CTRL key then press V and M in succession.

Now we need to add one more thing to the config files and that's the 'platform console serial' command. So if we happen to load INE initial config file as a startup-configuration and reload the routers we won't loose console connection. Again, we can accomplish this in a sed one liner. I am also going to TAR up all the files to be SCP'd into the routers.

 cd /root/ine/  
 find . -type f -exec sed -i '/hostname*/a platform console serial' {} \;  
 tar -cvf ine.tar /root/ine/ 

Now the INE initial configuration are prepped to be loaded into the routers.

In order to get all the INE initial config files on the routers, we need to connect GigabitEthernet2 of all the routers into the management virtual switch and TFTP or SCP all the config files.

Right click on all the routers, click on 'Edit Settings' and change Network Adapter 2 to VM Network.


Now SCP or TFTP the entire the INE files (sniffer1) to bootflash of all the routers. I used SCP in my lab and on R1 - R10 do the command below. 172.20.37.233 is the IP of my SNIFFER1 Linux VM. 

R1# copy scp://root@172.20.37.233/root/ine/ine.tar/ bootflash:ine.tar

Now you have all the INE initial config files ready to load into the running config as needed.

Extract the tar file on all the routers. 


Router#archive tar /xtract bootflash:ine.tar bootflash:ine

Save a blank.cfg on all routers

When the routers initially come up, you should have no config on them. Save this state in the bootflash by typing "copy run bootflash:blank.cfg". For example,

Router#copy run bootflash:blank.cfg

Make sure to do this on ALL ROUTERS. 

Snapshot all the routers

CSR license level is only good for 60 days. After 60 days you won't have access to advanced features like crypto map (needed for DMVPN etc). So after you load/extract the ine files on the routers take a snapshot of your router.

Right click on the router on VMware -> Snapshot -> Take Snapshot.

When you are passed 60 days, you can just revert back to this initial snapshot.

Automatic Loading Initial Configs

As you are going through INE or any other workbooks you will have to load different initial config files are or reset the running config so you can work on a different task. Loading up the initial config files, reloading the routers and waiting for 5 minutes doesn't make much sense, so the best option I've found is to use "configure [replace initial config file] force". With these scripts you can fully automate the loading of the initial config files on all routers.

To accomplish this we are going to use a little bit of PHP, parallel processing scripts, and bash scripting using expect. Again, even a novice user can get this accomplished by following my steps and if you have any questions leave comments.

First install the packages needed.

 yum install -y php.x86_64 php-cli.x86_64 php-common.x86_64            

Next add these entries in the sniffer1 host files. Make sure to edit the IPs accordingly. You can do a ifconfig -a on the Linux VMs to figure out the machines IP.

 root@sniffer1# vim /etc/hosts  

 172.20.37.201 sniffer1  
 172.20.37.202 sniffer2  
 172.20.37.203 sniffer3  


Next create the PHP script to parallel process and run multiple scripts simultaneously.

 root@sniffer1# vim /root/autoload.php 
  
 <?php  

 #NAME::INE AUTO LOAD by Arwin R.  
 #DESC::Expect script to auto load INE initial config files.  
 #Version 1.1

 if ($argv[1] == "exec") {  
     $configfile = $argv[2];  
     $server = $argv[3];  
     if ($server == "sniffer1") {  
         $consolenum = 0;  
         for ($i = 1; $i <= 4; $i++) {  
             shell_exec ("ssh root@$server /root/minicom.sh $configfile" . "R$i.txt $consolenum");  
             $consolenum++;  
         }  
     }  
     if ($server == "sniffer2") {  
         $consolenum = 0;  
         for ($i = 5; $i <= 8; $i++) {  
             shell_exec ("ssh root@$server /root/minicom.sh $configfile" . "R$i.txt $consolenum");  
             $consolenum++;  
         }  
     }  
     if ($server == "sniffer3") {  
         $consolenum = 0;  
         for ($i = 9; $i <= 10; $i++) {  
             shell_exec ("ssh root@$server /root/minicom.sh $configfile" . "R$i.txt $consolenum");  
             $consolenum++;  
         }  
     }  
 }  
 if ($argv[1] == "load") {  
     $configfile = $argv[2];  
     shell_exec ("php autoload.php exec $configfile sniffer1 >> switch.log &");  
     shell_exec ("php autoload.php exec $configfile sniffer2 >> switch.log &");  
     shell_exec ("php autoload.php exec $configfile sniffer3 >> switch.log &");  
 }  
 ?>  

Save the file using :wq!

Put this script on all 3 Linux VMs (sniffer1, sniffer2, sniffer3) under /root/

 root@sniffer1: vim /root/minicom.sh

 #!/usr/bin/expect  
 #NAME::INE AUTO LOAD by Arwin R.  
 #DESC::Expect script to auto load INE initial config files.  
 #Version 1.1  
 set timeout 1  
 set configfile [lindex $argv 0];  
 set serialnumber [lindex $argv 1];  
 set fd [ open /dev/ttyS$serialnumber { RDWR NONBLOCK } ]  
 fconfigure $fd  
 spawn -open $fd  
 send "\x1A";  
 sleep 1;  
 send "\r";  
 sleep 1;  
 send "\032";  
 sleep 1;  
 send "\r";  
 sleep 1;  
 send "configure replace bootflash:blank.cfg force\r";  
 sleep 4;  
 send "\r";  
 sleep 1;  
 send "\x1A";  
 sleep 1;  
 send "\r";  
 sleep 1;  
 send "\032";  
 sleep 1;  
 send "configure replace bootflash:$configfile force\r";  
 send "\r";  
 sleep 2  
 send "\x1A";  
 sleep 1;  
 send "\r";  
 sleep 1;  
 send "\032";  
 sleep 1;  

Save the file :wq!

Sniffer 1 needs to be able to SSH without password to sniffer1, sniffer2, and sniffer3 to run commands so we need to setup 'SSH without password'.


 root@sniffer1# ssh-keygen -t rsa  
 Generating public/private rsa key pair.  
 Enter file in which to save the key (/home/sniffer1/.ssh/id_rsa):   
 Created directory '/home/sniffer1/.ssh'.  
 
 Make sure you just hit enter twice here 

 Enter passphrase (empty for no passphrase):   
 Enter same passphrase again:   
 Your identification has been saved in /home/sniffer1/.ssh/id_rsa.  
 Your public key has been saved in /home/sniffer1/.ssh/id_rsa.pub.  
 The key fingerprint is:  
 3e:41:01:71:31:92:91:7c:3b:ad:e9:58:37:bc:37:e4 root@sniffer1  
 
 Now we need to create a directory ~/.ssh as user root on sniffer2 and sniffer3. If the directory already exists don't worry about creating it.   

 root@sniffer1# ssh root@sniffer2 mkdir -p .ssh  
 root@sniffer2 password:   
 root@sniffer1# ssh root@sniffer3 mkdir -p .ssh  
 root@sniffer3 password:   
 
 Let's now append the public key to sniffer1, sniffer2 and sniffer3.   
 
 root@sniffer1# cat .ssh/id_rsa.pub | ssh root@sniffer1 'cat >> .ssh/authorized_keys'  
 root@sniffer1 password:   

 root@sniffer1# cat .ssh/id_rsa.pub | ssh root@sniffer2 'cat >> .ssh/authorized_keys'  
 root@sniffer2 password:   

 root@sniffer1# cat .ssh/id_rsa.pub | ssh root@sniffer3 'cat >> .ssh/authorized_keys'  
 root@sniffer3 password:   


Sniffer1 now should be able to SSH to sniffer2 and sniffer3 without having to enter a password so the scripts can run seamlessly.

On sniffer1 type

root@sniffer1# php /root/autoload.php load ine/advanced.technology.labs/basic.rip.routing/

will load basic.rip.routing initial config on R1 - R10.

Obviously if you want to change the number of routers or on what Linux machines to run the scripts and such you can edit the script.

There you have it ... a very customized CCIE lab with easy console into the multiple routers without restrictions or licenses, and easy loading of INE initial configs. If you follow part 1 - part 3 series, it might save you a few hours of searching the Internet on how to setup the lab for CCIE v5. I hope you've enjoyed the series, and as always if you have any questions just leave comments.

Here is my first youtube video ever showing you what the end result should look like ...

https://www.youtube.com/watch?v=zvLx_-ufEUk&feature=youtu.be

Many more articles to come so ....

Please subscribe/comment/+1 if you like my posts as it keeps me motivated to write more and spread the knowledge.

Happy labbing!!!

26 comments:

  1. I enjoyed reading all 3 articles and you certainly are creative. I am lost though on the 2nd and 3rd parts specific to console access. I understand to detail the exact steps is very time consuming and most likely, very long. But for those of us who have little ESXi experience and Linux, it's not easy to follow.

    I downloaded the Cent OS 7, minimum which was a very large file. I burned ISO image which created many different files on the CD. Creating the Sniffer 1, 2, 3, etc VM machines is easy, how to install Cent 7 as Guest has many like myself lost. Then the remaining steps about pipes it not clear either. It appears you have a perfect setup and it would be great if you could share more details. I'd donate some money to you for your time and efforts.

    ReplyDelete
    Replies
    1. I would be happy to go through the steps in detail. If you are interested you can add me to your Google chat arwinr@gmail.com and I can setup a time to go through the entire setup. No need to donate I will do it for free.

      I will also edit the blogs and add more details.

      Appreciate your comments.

      Delete
    2. interesting alternative, but how is this setup more efficient than serial over network?

      Delete
    3. Hi Arwin,

      Great tutorial. Thank you for creating it. I have one question. If I have securecrt and would like to use a windows VM can I do all the scripting etc. on this or will I still need the Cent OS boxes.

      Delete
    4. This way is better because VMware ESXi "Serial-Over-Network" that most tutorials on the Internet talks about is only valid for 60 days as a demo or you have to get the enterprise license for ESXi which is over $2000.

      You can use your windows VM to connect to console by using hyperterm or putty but the limitation still exists with 4 adapters per VM. Also I am not sure about the scripting would work on windows. I suppose you can use VB or something.

      Delete
  2. My sniffer is unable to get out to the internet - I am not sure what I missed. I cannot ping any internet IP address.

    ReplyDelete
    Replies
    1. Do route -n and post ... you might not have the default gateway. do you have the nic on the right network?

      Delete
  3. route -n and ifconfig are not found - i am going to try a different version of centos - the minimal install is pretty miminal. should i do netinstall?

    ReplyDelete
    Replies
    1. Try ip route add default via .. then yum install net-tools

      Delete
  4. use ip route show or ip route add default via dev etc man ip route then when you get the default gw set .. type yum install net-tools which will give you if config etc.

    ReplyDelete
  5. Arwin, Thanks for sharing your setup. I am getting an error when I try try to extract ine.tar on the router.

    Router#archive tar /xtract bootflash:ine.tar bootflash:ine
    %Tar checksum error in bootflash:ine.tar

    Router#dir
    Directory of bootflash:/

    11 drwx 16384 Feb 24 2015 17:04:20 +00:00 lost+found
    681409 drwx 4096 Feb 24 2015 17:05:20 +00:00 .super.iso.dir
    12 -rw- 45 Feb 25 2015 22:45:19 +00:00 .CsrLxc_LastInstall
    13 -rw- 83 Feb 24 2015 17:57:23 +00:00 virtual-instance.conf
    486721 drwx 4096 Feb 24 2015 17:07:43 +00:00 .ssh
    15 -rw- 161587200 Feb 24 2015 17:05:19 +00:00 iosxe-remote-mgmt.03.14.00.S.155-1.S-std.ova
    389380 -rw- 250571008 Feb 24 2015 17:06:18 +00:00 csr1000v-mono-universalk9.03.14.00.S.155-1.S-std.SPA.pkg
    389378 -rw- 4874 Feb 24 2015 17:06:17 +00:00 csr1000v-packages-universalk9.03.14.00.S.155-1.S-std.cof
    389379 -rw- 5662 Feb 24 2015 17:06:18 +00:00 packages.conf
    567841 drwx 4096 Feb 24 2015 17:07:45 +00:00 core
    470497 drwx 4096 Feb 24 2015 17:07:45 +00:00 .prst_sync
    275809 drwx 4096 Feb 24 2015 17:07:53 +00:00 .rollback_timer
    48674 -rw- 0 Feb 24 2015 17:08:02 +00:00 tracelogs.922
    584065 drwx 8192 Feb 25 2015 23:36:28 +00:00 tracelogs
    713857 drwx 4096 Feb 24 2015 17:08:18 +00:00 .installer
    48675 -rw- 0 Feb 25 2015 22:46:44 +00:00 cvac.log
    48676 -rw- 1552 Feb 25 2015 22:47:00 +00:00 csrlxc-cfg.log
    730081 drwx 4096 Feb 24 2015 17:57:34 +00:00 onep
    48673 -rw- 430331 Feb 26 2015 17:08:23 +00:00 ine.tar

    7835619328 bytes total (6611050496 bytes free)

    ReplyDelete
    Replies
    1. Seems like the ine.tar is corrupted as you were copying it to the router. Maybe you can do "copy /verify tftp://IPADDRESS/ine.tar bootflash:ine.tar" and make sure it transfers correctly to the routers. Also does the ine.tar open up regularly on your windows or linux machine?

      Delete
  6. Thanks Arwin. I have that fixed now.I am 99% done.
    The only problem I seem to be having now is with the script auto loading all the initial configs to the routers. The script seems to also remove the IPv4 and IPv6 address due to the removal of VLAN encapsulation. As a result, my sub-interfaces have no IP addresses, only the loopback and tunnel interfaces do. I observed this in your youtube video as well, starting at 1:39. Do you have a fix for this? Thanks in advance. Your posts have been very informative and educational. I visit everyday!!

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Well I actually resolved this issue a while back and forgot to update the post :)...well now this post is updated with that information. Basically when the routers first come up they should have no config. Save this state as blank.cfg to the bootflash. "copy run bootflash:blank.cfg" ... then in the minicom.sh replace the line where it first does the .. send "configure replace bootflash:$configfile force\r"; ... to load the blank.cfg first ... send "configure replace bootflash:blank.cfg force\r"; ... this will get rid of this problem :) .. again this post has been updated to include this information.

      Delete
  7. Hello Arwin, I hope all is well with you. Thanks for the update to the script. I apologize for not posting my success sooner but work and family life has had me really busy. I just got to my lab a few moments ago and got the scripts working. Thanks a million!!

    ReplyDelete
  8. You're welcome!! Glad I could help.

    ReplyDelete
  9. This is absolutely what I have been looking for. Thank you Arwin!

    ReplyDelete
  10. It seems like ESXi 6.0 has done away with the 4-serial port issue. I've just set up 10 CSR instances with named pipes and have minicom connected to all 10 at once from the same virtual machine.

    ReplyDelete
    Replies
    1. That is great news. I might just download ESXi 6.0 and give it a try. This just means we have to tweak the auto load config script a little bit and one Linux VM should be able to do the trick. Thanks for the comment.

      Delete
    2. I've added a note at the beginning of this blogtorial to let the readers know :) Thanks again.

      Delete
    3. Hi Arwin,

      First of all (since I forgot it the last time I commented) - Thank you so much for this write-up, it's really been extremely helpful in getting this up and running. :)

      A couple more notes:

      1. 20 wasn't a limit either, so I now have all 20 CSRs on the same ubuntu server. I load the configs all at once with your minicom.sh (with very minor modifications) and a short bash script:
      #!/bin/bash
      for i in `seq 1 20`; do
      let "t = $i - 1"
      ~/minicom.sh $1"R"$i.txt $t &
      done

      2. For IOS XE 3.13S and later, the ax license is already enabled at first boot with a throughput limit of 100Kbps. Higher throughput requires registering each CSR at the Cisco Software Licensing Portal:
      http://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/configuration/csr1000Vswcfg/licensing.html#77755

      BR,
      Thomas

      Delete
  11. Great series... can't thank you enough for it. A few notes from my mistakes, mostly caused by late nights:

    1) I had to only tar, and not tar/gzip the configs to get the archive xtract command to work for me. Otherwise, it gave me a checksum error every time.

    2) Don't create the scripts and ftp them to the sniffers or else windows formatting will destroy it. Just ssh and vi the files. Had to remember that, oh wait, these things are live on the network... doh!

    3) I had to install expect to get the minicom script to work (yum -y install expect). A "bad interpreter" error was the red flag in this case.

    Again, outstanding posts. Many thanks.

    ReplyDelete
    Replies
    1. Thanks for posting this. I've update the blog accordingly. You're very welcome :)

      Delete
  12. Hi Arwin
    Excellent Post !!
    I am also trying to mimic INE CCIE v5 lab setup with CSR 1000v
    However I am having one trouble.
    When I telnet into the CSR I get below console prompt.

    Router(diag)>en
    Password:
    Router(diag)#
    Router(diag)#

    Any Idea why I am ending up to diag mode. and how to get rid from this.

    FYI I am using ESXi 5.5 and the latest CSR ova file csr1000v-universalk9.03.13.01.S.154-3.S1-ext.ova

    I do appreciate your help. Thanks in advance

    ReplyDelete
    Replies
    1. http://www.cisco.com/c/en/us/td/docs/routers/asr1000/configuration/guide/chassis/asrswcfg/Using_CLI.html#pgfId-1040281

      There is a mention of this in the link above.

      Delete