Site Maintenance with Arista eAPI

I started a public GitHub Repository in order to share some of my Python automation tools.

I work remotely on a large number of Arista switches and I have developed a script that captures the state of the network in order to run a comparison before and after network maintenance events. We have a monitoring system in place that alerts on various SNMP traps and events, but often times I need to interpret a number of changes via my network engineer lens to confirm that a maintenance period has been completed successfully.

The first Python script I’m publishing interfaces with an Arista switch over HTTPS and captures various outputs via the eAPI: inventory with serial numbers, vlan states, mac address table, lldp neighbors, routing protocol states, and route entries to a text file. A diff on the files can be run post maintenance event to highlight any changes for review.

This type of granular record can show that an item such as a metric on a route has changed. Our monitoring system does not alert on this type of change as it is monitoring the BGP peer status.

 

Link github.com/brentnowak/arista-tools

Advertisements

Portfolio Building with iShares ETFs at Fidelity

My employer sponsored 401k is with Fidelity and I recently tasked myself with building a long-term, diversified portfolio on their platform.  I’ve built out a portfolio that has a heavy weight in US equities, but also captures a large amount of both developed and emerging markets.

At the time of this post, Fidelity offers 70 commission-free iShares ETFs of various categories and classes, so I was able to pick and choose from a large array of ETFs in order to build out a portfolio.

Portfolio Tenants

  • Overweight equities given that I am in my mid-30’s.
  • Limited bond exposure due to rising rates.
  • High international exposure (~38%) to potentially outsize risk/reward outside the US market.
  • Keep holdings to less than 12 to ease monthly re-balancing.
  • Increase US Small-Cap exposure due to a belief that they will outperform over the next 20 years.

General Portfolio 

 

Addition of IEUR (iShares Core MSCI Europe ETF)

I found that IXUS (iShares Core MSCI Total International Stock ETF) and IEMG (iShares Core MSCI Emerging Markets ETF) did not sufficiently provide enough weight for my liking towards major European countries.

Adding IEUR (iShares Core MSCI Europe ETF) to the portfolio brought the United Kingdom, France, and Germany to an exposure level that was more in my risk tolerance levels.

Exposure percentage summary of each ETF by country:

 

* iShares® is a registered trademark of BlackRock Fund Advisors and its affiliates

Using Arista Telemetry to Monitor Network State

Arista’s Telemetry product allows you to stream network state in real-time from each piece of switching hardware to a central management point. Recently I’ve been waiting to prove out the visibility of the Telemetry product during a maintenance event. I recently had the opportunity when we were switching carriers on a transatlantic link between two sites.

The screenshot below shows a 30 minute slice of time where fiber optic links were brought down on Carrier A, traffic flow changed, connectivity was restored with Carrier B, and normal traffic flow resumed.

1 – Technicians disconnect fiber for the Primary Circuit at sites A and B.
2 – As the routing neighbor between the A and B sites drop, traffic is automatically moved to the Secondary Circuits.
3 – When optical connectivity has been restored between the two sites, the Primary Circuit re-establishes. Routing reconverges and traffic is shifted back onto the Primary Circuit.

Normal workflow for a change like this involves camping Syslog and the various switches involved, issuing commands to show network activity as the maintenance progressed. With Arista’s Telemetry product, I was able to see the state of various network components (light levels, interface state, bit rate, etc) all in a real-time display.

So far an impressive way to gain visibility into network state across a bunch of different metrics at a glance. I’m looking forward to test this product with my other future projects.

Race Report: Avenue of the Giants Half Marathon

Representing my home state of Illinois and Golden Gate Running Club

Elusive 90 Minute Half Marathon

After a disappointing attempt in February to run a Half in under 90 minutes, Xavier convinced me to sign up and train for the Avenue of the Giants Half in early May. It seemed really simple, but in hindsight this training cycle brought me up to a new level of running.

Trees Older Than Time

The event takes place in Humboldt Redwoods State Park in rural Northern California and follows an immaculately paved road through an old redwood forest. It took us about 5 hours to get to the race start given rest breaks, so not a quick drive out of San Francisco.

The entire race has you ensconced in a backdrop of redwoods so quiet that you can hear the birds chirping away in the forest. There isn’t much to do besides hit the turnaround point and double-back. Luckily for me, races in this scenery are a Zen-like experience and a welcomed break from busy city streets and intersections.

Doubling The Mileage

April was a big month for me, clocking in at almost 162 miles. Historically I’ve never run more than 85 miles in a month, so training for this race was a new experience.

Slave to the Spreadsheet

I’ve never followed a structured training plan before and Xavier helped guide me through the process. We used a slightly modified version of Brad Hudson’s Half-Marathon Level 1 plan with a few days switched around to follow GGRC‘s (Golden Gate Running Club) Wednesday Track night and Sunday runs.

Running ~40 miles/week was new to my body and the first three weeks of increased mileage had me in a rut. I needed to eat more, sleep more, and my water intake increased to around 2L/day. I had my general meals, but also started to eat when I was hungry. I successfully kept the snacking to whole wheat crackers with peanut butter, nuts, and banana & berry smoothies with yogurt. There was also a healthy amount of dark chocolate, which I basically consider energy.

The main weekly routine was (1) Tempo, (1) Long Run, (1) Speed Session, and (2-3) Recovery Sessions. I only missed two or three recovery runs due to work, injury, or travel; consistency was the name of the game for this plan and it shows in my Strava log.

Travel, Sickness, and Injury

Late March, when the miles started to add up, I had issues with my TFL and Hip Abductors. Every day, regardless of my workout, I was on the foam roller and Lacrosse ball for around 20 minutes a day to help loosen up tight areas. The diligence paid off and I was able to coast into the last month of training injury free.

The hardest block of time for me was the week of April 16-30th where I spend a week in Sydney Australia and also somehow contracted bronchitis. The 15 hour flight, jetlag, casual drinking, and inability to exchange air when running took a toll on me. Luckily I was able to find some unobstructed running loops in Sydney and log miles before the rest of my vacation crew wanted to head out of the apartment for the day. Additionally, my airway cleared up a few days before the race and my leg strength was unaffected. I knew at some point during the training block I was going to be sick, and I was just going to have to roll with it!

Race Day

I arrived at the start line injury free, rested, hydrated, and with a settled stomach — always a welcomed way to start any endurance event. It was slightly chilly and overcast, perfect running conditions. Since Xavier and I had the same race goal, we decided to run together with an agreement that if either of us was feeling strong and wanted to push pace, that we should go for it.

The limited GPS signaling meant that our watches were all over the place. My watch was generally beeping at each marker, but became more skewed with each mile. If you do this race, write out the mile times on your arm so you can see where you are at given clock time. The course it a basic out-and-back with a turnaround at the halfway mark. We clocked 44:45 at the turnaround, so we knew we had to maintain speed and start to try to negative split the race.

Around mile 10, Xavier started to pull ahead of me. My heart rate was increasing and my breathing became more labored. The rolling hills for the last third of the race started to wane on my effort. I found myself outputting the same effort as the first half (6:50 pace), but running slower (7:10-7:15). My heart rate was trending upward so I pulled my pace back by 2-3%.

Soon the sub-90 minute goal was out of reach and got weighed down with negative thoughts. When I accepted that I was going to clock in over my goal, I decided to enjoyed the scenery, cheered for other runners, and even ended up picking off a runner a head of me with a last bit of pain-cave-dizziness-effort at mile 12. As I finished, Xavier was standing right outside the chute and I could hear him starting to yell at me to kick my brains out – a welcomed voice after being in a pain cave for the last 20 minutes of the race.

I finished at 1:32:13, 26th overall, 8/71 in my age-group, and cut over 4 minutes off my last Half attempt. Overall a big gain despite not being able to clock in under 1:30.

 

Have You Tried Running More?

The post-race emotional high had Xavier and I recapping our training experiences and what I could do the next attempt to keep me from waning at around mile 10. For both of us, we needed more sustained efforts on longer runs. I was using GGRC‘s Sunday run as my long run, which meant running out to the Presidio (5 miles), idling for 10-15 minutes while the group gathers, running 6-7 miles with the group that included water/regroup breaks, and then circling back home.  The longer runs should have been constant efforts, which would have helped me dig deeper for the final third of the race.

This event was a big step for me, and I finally feel like I can brand myself as a runner. There is going to be another push to break the 90 minute mark, and I’m going to achieve it. This latest round of racing taught me more about my limits, what tools I need to be successful, and a few more tricks to improve the cardio system.

Cisco Live 2017

I’m back in San Francisco after a solid few days of conference sessions, heat, crowds, and getting to meet all sort of new faces. This year I concentrated on Nexus 9000 and VXLAN sessions as we are refreshing our TOR solution in the datacenter.

Attended Sessions

  • BRKARC-3222 – Cisco Nexus 9000 Architecture
  • BRKDCN-3020 – Network Analytics using Nexus 3000/9000 Switches
  • BRKDCN-3378 – Building Data Center Networks with VXLAN EVPN Overlays
  • BRKINI-2005 – Engineering Fast IO to the Network
  • BRKIPM-2264 – Multicast Troubleshooting
  • BRKRST-3320 – Troubleshooting BGP
  • BRKDCN-2015 – Nexus Standalone Container Networking

I also picked up a new addition to the library, Building Data Centers with VXLAN BGP EVPN.

Force-Directed Network Diagram with Arista eAPI and D3.js

2016-05-23_force_direction_1Overview

The Arista eAPI give you the ability to interact with a switch over standard HTTPS and return structured JSON. Here a section of Python code to populate a database table to automatically generate a network diagram based on LLDP neighbor relationships.

 

Requirements

Arista EOS
Python 2.7
Postgresql
pyeapi

Database Tables

CREATE TABLE report.control
(
  id serial NOT NULL,
  "switchName" text NOT NULL,
  "interfaceName" text NOT NULL,
  "interfaceType" text,
  monitor boolean,
  description text NOT NULL,
  "remoteSwitchName" text,
  "remoteSwitchPort" text,
  "lineProtocolStatus" text,
  "interfaceStatus" text,
  site text,
  CONSTRAINT pk_report_control PRIMARY KEY ("switchName", "interfaceName")
)

Sample Python Code

Populating our database table with Switch and Interface information.

import pyeapi
pyeapi.load_config('nodes.conf')

def control_insert_lldp(switchName, interfaceName, remoteSwitchName, remoteSwitchPort):
    try:
        conn = psycopg2.connect(conn_string)
        cursor = conn.cursor()
        sql = '''
        UPDATE report.control
        SET "remoteSwitchName" = %s, "remoteSwitchPort" = %s
        WHERE "switchName" = %s AND "interfaceName" = %s
        '''
        data = (remoteSwitchName, remoteSwitchPort, switchName, interfaceName, )
        cursor.execute(sql, data, )
    except psycopg2.IntegrityError:
        conn.rollback()
    else:
        conn.commit()
    return 0

for switch in switches:
 node = pyeapi.connect_to(switch['deviceName'])
 try:
 output = node.enable('show lldp neighbors')
 neighbors = output[0]['result']['lldpNeighbors']
 for neighbor in neighbors:
 neighborDevice = removedomain(neighbor['neighborDevice'])
 control_insert_lldp(hostname, neighbor['port'], neighborDevice, neighbor['neighborPort'])
 except Exception as e:
 print(e)

Getting a list of switches from our database table.

def network_switches():
    conn = psycopg2.connect(conn_string)
    cursor = conn.cursor(cursor_factory=RealDictCursor)
    sql = '''
    SELECT
    DISTINCT(control."switchName") as "name",
      site."siteDescription" as group
    FROM
      report.control
    WHERE
      "remoteSwitchName" != ''
    ORDER BY
      control."switchName"
    '''
    cursor.execute(sql, )
    results = cursor.fetchall()
    return results

Returning a LLDP neighbor value if we have one for each switch interface.

def network_lldp_neighbors(switchName):
    conn = psycopg2.connect(conn_string)
    cursor = conn.cursor(cursor_factory=RealDictCursor)
    sql = '''
    SELECT
      DISTINCT(control."remoteSwitchName") as "remoteName"
    FROM
      report.control
    WHERE
      "switchName" = %s AND
      "remoteSwitchName" != ''
    ORDER BY "remoteSwitchName"
    '''
    data = (switchName, )
    cursor.execute(sql, data, )
    results = cursor.fetchall()
    return results

Create a JSON string for D3.js Force-Directed Graph.

def d3_lldp(element):
    links = []
    value = 1
    nodes = network_switches()
    idCount = 0
    for row in nodes:
        row['id'] = idCount
        # row['group'] = 1
        idCount += 1

    for node in nodes:
        lldpswitches = network_lldp_neighbors(node['name'])
        source = node['id']

        for connection in lldpswitches:
            for row in nodes:
                if row['name'] == connection['remoteName']:
                    target = row['id']
                    result = {'source': source, 'target': target, 'value': value}
                    links.append(result)

    result = {'nodes': nodes, 'links': links}
    return json.dumps(result)

Arista ACE Training

Last week I had the opportunity to attend Arista ACE 2.1 training down at the HQ offices in Santa Clara. I was very fortunate to be in a class that was lead by Gary Donahue, the author of Arista Warrior. He is an excellent presenter and a extremely personable individual. If you ever have a chance to be in a class of his, sign up for it.

Photo Apr 18, 12 00 12 PM

The training was very hands on, with labs that covered Zero Touch Provisioning (ZTP), Multi-Chassis LAG (MLAG), Virtual Extensible LAN (VXLAN), and my favorite topic, the wonderful EAPI. Coming from a Cisco CCNA/CCNP background, these topics helped fill knowledge gaps on the Arista family of hardware.

At the end of the class, Gary was signing copies of his book so I left with a author signed copy of Arista Warrior. Not a bad addition to the growing collection of O’Reilly books around the house.

Photo Apr 25, 8 14 33 PM