Using Arista Telemetry to Monitor Network State

Arista’s Telemetry product allows you to stream network state in real-time from each piece of switching hardware to a central management point. Recently I’ve been waiting to prove out the visibility of the Telemetry product during a maintenance event. I recently had the opportunity when we were switching carriers on a transatlantic link between two sites.

The screenshot below shows a 30 minute slice of time where fiber optic links were brought down on Carrier A, traffic flow changed, connectivity was restored with Carrier B, and normal traffic flow resumed.

1 – Technicians disconnect fiber for the Primary Circuit at sites A and B.
2 – As the routing neighbor between the A and B sites drop, traffic is automatically moved to the Secondary Circuits.
3 – When optical connectivity has been restored between the two sites, the Primary Circuit re-establishes. Routing reconverges and traffic is shifted back onto the Primary Circuit.

Normal workflow for a change like this involves camping Syslog and the various switches involved, issuing commands to show network activity as the maintenance progressed. With Arista’s Telemetry product, I was able to see the state of various network components (light levels, interface state, bit rate, etc) all in a real-time display.

So far an impressive way to gain visibility into network state across a bunch of different metrics at a glance. I’m looking forward to test this product with my other future projects.


Race Report: Avenue of the Giants Half Marathon

Representing my home state of Illinois and Golden Gate Running Club

Elusive 90 Minute Half Marathon

After a disappointing attempt in February to run a Half in under 90 minutes, Xavier convinced me to sign up and train for the Avenue of the Giants Half in early May. It seemed really simple, but in hindsight this training cycle brought me up to a new level of running.

Trees Older Than Time

The event takes place in Humboldt Redwoods State Park in rural Northern California and follows an immaculately paved road through an old redwood forest. It took us about 5 hours to get to the race start given rest breaks, so not a quick drive out of San Francisco.

The entire race has you ensconced in a backdrop of redwoods so quiet that you can hear the birds chirping away in the forest. There isn’t much to do besides hit the turnaround point and double-back. Luckily for me, races in this scenery are a Zen-like experience and a welcomed break from busy city streets and intersections.

Doubling The Mileage

April was a big month for me, clocking in at almost 162 miles. Historically I’ve never run more than 85 miles in a month, so training for this race was a new experience.

Slave to the Spreadsheet

I’ve never followed a structured training plan before and Xavier helped guide me through the process. We used a slightly modified version of Brad Hudson’s Half-Marathon Level 1 plan with a few days switched around to follow GGRC‘s (Golden Gate Running Club) Wednesday Track night and Sunday runs.

Running ~40 miles/week was new to my body and the first three weeks of increased mileage had me in a rut. I needed to eat more, sleep more, and my water intake increased to around 2L/day. I had my general meals, but also started to eat when I was hungry. I successfully kept the snacking to whole wheat crackers with peanut butter, nuts, and banana & berry smoothies with yogurt. There was also a healthy amount of dark chocolate, which I basically consider energy.

The main weekly routine was (1) Tempo, (1) Long Run, (1) Speed Session, and (2-3) Recovery Sessions. I only missed two or three recovery runs due to work, injury, or travel; consistency was the name of the game for this plan and it shows in my Strava log.

Travel, Sickness, and Injury

Late March, when the miles started to add up, I had issues with my TFL and Hip Abductors. Every day, regardless of my workout, I was on the foam roller and Lacrosse ball for around 20 minutes a day to help loosen up tight areas. The diligence paid off and I was able to coast into the last month of training injury free.

The hardest block of time for me was the week of April 16-30th where I spend a week in Sydney Australia and also somehow contracted bronchitis. The 15 hour flight, jetlag, casual drinking, and inability to exchange air when running took a toll on me. Luckily I was able to find some unobstructed running loops in Sydney and log miles before the rest of my vacation crew wanted to head out of the apartment for the day. Additionally, my airway cleared up a few days before the race and my leg strength was unaffected. I knew at some point during the training block I was going to be sick, and I was just going to have to roll with it!

Race Day

I arrived at the start line injury free, rested, hydrated, and with a settled stomach — always a welcomed way to start any endurance event. It was slightly chilly and overcast, perfect running conditions. Since Xavier and I had the same race goal, we decided to run together with an agreement that if either of us was feeling strong and wanted to push pace, that we should go for it.

The limited GPS signaling meant that our watches were all over the place. My watch was generally beeping at each marker, but became more skewed with each mile. If you do this race, write out the mile times on your arm so you can see where you are at given clock time. The course it a basic out-and-back with a turnaround at the halfway mark. We clocked 44:45 at the turnaround, so we knew we had to maintain speed and start to try to negative split the race.

Around mile 10, Xavier started to pull ahead of me. My heart rate was increasing and my breathing became more labored. The rolling hills for the last third of the race started to wane on my effort. I found myself outputting the same effort as the first half (6:50 pace), but running slower (7:10-7:15). My heart rate was trending upward so I pulled my pace back by 2-3%.

Soon the sub-90 minute goal was out of reach and got weighed down with negative thoughts. When I accepted that I was going to clock in over my goal, I decided to enjoyed the scenery, cheered for other runners, and even ended up picking off a runner a head of me with a last bit of pain-cave-dizziness-effort at mile 12. As I finished, Xavier was standing right outside the chute and I could hear him starting to yell at me to kick my brains out – a welcomed voice after being in a pain cave for the last 20 minutes of the race.

I finished at 1:32:13, 26th overall, 8/71 in my age-group, and cut over 4 minutes off my last Half attempt. Overall a big gain despite not being able to clock in under 1:30.


Have You Tried Running More?

The post-race emotional high had Xavier and I recapping our training experiences and what I could do the next attempt to keep me from waning at around mile 10. For both of us, we needed more sustained efforts on longer runs. I was using GGRC‘s Sunday run as my long run, which meant running out to the Presidio (5 miles), idling for 10-15 minutes while the group gathers, running 6-7 miles with the group that included water/regroup breaks, and then circling back home.  The longer runs should have been constant efforts, which would have helped me dig deeper for the final third of the race.

This event was a big step for me, and I finally feel like I can brand myself as a runner. There is going to be another push to break the 90 minute mark, and I’m going to achieve it. This latest round of racing taught me more about my limits, what tools I need to be successful, and a few more tricks to improve the cardio system.

Cisco Live 2017

I’m back in San Francisco after a solid few days of conference sessions, heat, crowds, and getting to meet all sort of new faces. This year I concentrated on Nexus 9000 and VXLAN sessions as we are refreshing our TOR solution in the datacenter.

Attended Sessions

  • BRKARC-3222 – Cisco Nexus 9000 Architecture
  • BRKDCN-3020 – Network Analytics using Nexus 3000/9000 Switches
  • BRKDCN-3378 – Building Data Center Networks with VXLAN EVPN Overlays
  • BRKINI-2005 – Engineering Fast IO to the Network
  • BRKIPM-2264 – Multicast Troubleshooting
  • BRKRST-3320 – Troubleshooting BGP
  • BRKDCN-2015 – Nexus Standalone Container Networking

I also picked up a new addition to the library, Building Data Centers with VXLAN BGP EVPN.

Force-Directed Network Diagram with Arista eAPI and D3.js


The Arista eAPI give you the ability to interact with a switch over standard HTTPS and return structured JSON. Here a section of Python code to populate a database table to automatically generate a network diagram based on LLDP neighbor relationships.



Arista EOS
Python 2.7

Database Tables

CREATE TABLE report.control
  id serial NOT NULL,
  "switchName" text NOT NULL,
  "interfaceName" text NOT NULL,
  "interfaceType" text,
  monitor boolean,
  description text NOT NULL,
  "remoteSwitchName" text,
  "remoteSwitchPort" text,
  "lineProtocolStatus" text,
  "interfaceStatus" text,
  site text,
  CONSTRAINT pk_report_control PRIMARY KEY ("switchName", "interfaceName")

Sample Python Code

Populating our database table with Switch and Interface information.

import pyeapi

def control_insert_lldp(switchName, interfaceName, remoteSwitchName, remoteSwitchPort):
        conn = psycopg2.connect(conn_string)
        cursor = conn.cursor()
        sql = '''
        UPDATE report.control
        SET "remoteSwitchName" = %s, "remoteSwitchPort" = %s
        WHERE "switchName" = %s AND "interfaceName" = %s
        data = (remoteSwitchName, remoteSwitchPort, switchName, interfaceName, )
        cursor.execute(sql, data, )
    except psycopg2.IntegrityError:
    return 0

for switch in switches:
 node = pyeapi.connect_to(switch['deviceName'])
 output = node.enable('show lldp neighbors')
 neighbors = output[0]['result']['lldpNeighbors']
 for neighbor in neighbors:
 neighborDevice = removedomain(neighbor['neighborDevice'])
 control_insert_lldp(hostname, neighbor['port'], neighborDevice, neighbor['neighborPort'])
 except Exception as e:

Getting a list of switches from our database table.

def network_switches():
    conn = psycopg2.connect(conn_string)
    cursor = conn.cursor(cursor_factory=RealDictCursor)
    sql = '''
    DISTINCT(control."switchName") as "name",
      site."siteDescription" as group
      "remoteSwitchName" != ''
    cursor.execute(sql, )
    results = cursor.fetchall()
    return results

Returning a LLDP neighbor value if we have one for each switch interface.

def network_lldp_neighbors(switchName):
    conn = psycopg2.connect(conn_string)
    cursor = conn.cursor(cursor_factory=RealDictCursor)
    sql = '''
      DISTINCT(control."remoteSwitchName") as "remoteName"
      "switchName" = %s AND
      "remoteSwitchName" != ''
    ORDER BY "remoteSwitchName"
    data = (switchName, )
    cursor.execute(sql, data, )
    results = cursor.fetchall()
    return results

Create a JSON string for D3.js Force-Directed Graph.

def d3_lldp(element):
    links = []
    value = 1
    nodes = network_switches()
    idCount = 0
    for row in nodes:
        row['id'] = idCount
        # row['group'] = 1
        idCount += 1

    for node in nodes:
        lldpswitches = network_lldp_neighbors(node['name'])
        source = node['id']

        for connection in lldpswitches:
            for row in nodes:
                if row['name'] == connection['remoteName']:
                    target = row['id']
                    result = {'source': source, 'target': target, 'value': value}

    result = {'nodes': nodes, 'links': links}
    return json.dumps(result)

Arista ACE Training

Last week I had the opportunity to attend Arista ACE 2.1 training down at the HQ offices in Santa Clara. I was very fortunate to be in a class that was lead by Gary Donahue, the author of Arista Warrior. He is an excellent presenter and a extremely personable individual. If you ever have a chance to be in a class of his, sign up for it.

Photo Apr 18, 12 00 12 PM

The training was very hands on, with labs that covered Zero Touch Provisioning (ZTP), Multi-Chassis LAG (MLAG), Virtual Extensible LAN (VXLAN), and my favorite topic, the wonderful EAPI. Coming from a Cisco CCNA/CCNP background, these topics helped fill knowledge gaps on the Arista family of hardware.

At the end of the class, Gary was signing copies of his book so I left with a author signed copy of Arista Warrior. Not a bad addition to the growing collection of O’Reilly books around the house.

Photo Apr 25, 8 14 33 PM

rConfig on CentOS 6.6

I’ve used RANCID in the past, but I wanted to use a more modern configuration management tool at my current organization. I’ve been following the rConfig project for a while now and recently setup an instance of it on our network.

Here is a short guide on how to get rConfig up and running on a CentOS instance.

# CentOS not getting DHCP address on eth0 under VMware
edit and set to yes

# Install some tools
# I prefer vim over vi
sudo yum install net-tools
sudo yum install wget
sudo yum install zip unzip
sudo yum install vim-common vim-minimal vim-enhanced vim-X11

# Install Apache
sudo yum install httpd

#Install MySQL
yum install mysql mysql-server
service mysqld start

#Install PHP
yum install php php-common
yum install php-common php-cli php-mysql php-devel

# Service Restarts
service httpd restart
chkconfig httpd on
service mysqld restart
chkconfig mysqld on

# Adjust firewall to allow for inbound http
sudo iptables -I INPUT -p tcp --dport 80 -j ACCEPT
sudo service iptables save

# Use wget to get the rConfig zip from
download, unzip to /home/rconfig

# Change ownership
chown -R apache /home/rconfig
mv /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.original
cp /home/rconfig/www/install/ /etc/httpd/conf/httpd.conf

# Adjust permissions
vim /etc/selinux/config
"SELINUX=enforcing" to "SELINUX=disabled"

# Test that Apache and MySQL autostart

# Setup rConfig via the web interface

# Create your rConfig user
CREATE USER 'rconfig_user'@'localhost' IDENTIFIED BY 'some-password';
GRANT ALL PRIVILEGES ON * . * TO 'rconfig_user'@'localhost';

# Change rconfig_user password
SET PASSWORD FOR 'rconfig_user'@'localhost' = PASSWORD('some-password');

# Setup NTP
yum install ntp ntpdate ntp-doc
chkconfig ntpd on
/etc/init.d/ntpd start


GNS3 and VRRP Timers

While testing out a VRRP solution, I noticed that it was not performing as expected. The VRRP address was unresponsive so I started to investigate. Turning on console logging, I saw a large amount of flapping between Backup and Master states.

*Mar  1 02:37:23.739: VRRP: Grp 1 Event - Master down timer expired
*Mar  1 02:37:23.739: %VRRP-6-STATECHANGE: Vl20 Grp 1 state Backup -> Master
*Mar  1 02:37:25.095: %VRRP-6-STATECHANGE: Vl20 Grp 1 state Master -> Backup

It turns out that running 8 routers in GNS3 on my laptop was slightly under-powered platform and resulting in over a 2 second maximum response time from a VRRP peer.

Sending 8000, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 74 percent (611/818), round-trip min/avg/max = 4/705/1996 ms

After adjusting the advertise timers, everything started to perform as expected.

interface Vlan20
 ip address
 vrrp 1 ip
 vrrp 1 timers advertise 10
 vrrp 1 priority 110
interface Vlan20
 ip address
 vrrp 1 ip
 vrrp 1 timers advertise 10