More Go concurrency using pipelines with eAPI

As a follow on to my previous post on using Go channels for concurrency, I thought I would try and use the pipeline pattern as well.  The idea is to create a series of goroutines that you can string together through channels.  This allows you to mix and match (compose) small functions to build the final result you want.  Its like using the ‘|’ operator in Unix.  For this example I’m going to take a few different show commands I want to run, create pipelined functions out of them, then string them together to pull down the final result I want.

For this example I will go grab the show version and show running-config of a series of Arista switches.  I’ve defined a json file to store the switch names and connection information.  Here is a short function to read in that file and parse the JSON data:

func readSwitches(filename string) []EosNode {
	var switches []EosNode
 
	file, err := os.Open("switches.json")
	if err != nil {
		panic(err)
	}
	decoder := json.NewDecoder(file)
	err = decoder.Decode(&switches)
	if err != nil {
		panic(err)
	}
	return switches
}

To store all the information I created a struct with fields for the relevant data (there are some extra fields here for future use):

type EosNode struct {
	Hostname      string
	MgmtIp        string
	Username      string
	Password      string
	Ssl           bool
	Reachable     bool
	ConfigCorrect bool
	Uptime        float64
	Version       string
	Config        string
	IntfConnected []string
	IpIntf        []string
	Vlans         []string
}

Now I start writing my functions.  There are three types of functions that we need.  First I will write a producer that starts the whole thing off by generating channels for each switch (in this case it will be EosNodes).  Then intermediate functions will take actions on those channels, and return a new channel with an EosNode.  Finally the consumer will take the channels and produce the final result.

The producer (or generator) will take a list of EosNodes, then kick off goroutines for each switch and tie them into the out channel, which I return from the function:

func genSwitches(nodes []EosNode) <-chan EosNode {
	out := make(chan EosNode)
	go func() {
		for _, node := range nodes {
			out <- node
		}
		close(out)
	}()
	return out
}

Now the intermediate functions that receive EosNodes from the channel, runs the eAPI call to fill in more data, then returns a new outbound channel with the new data populated in the EosNode:

func getConfigs(in <-chan EosNode) <-chan EosNode {
	out := make(chan EosNode)
	go func() {
		for n := range in {
			cmds := []string{"enable", "show running-config"}
			url := buildUrl(n)
			response := eapi.Call(url, cmds, "text")
			config := response.Result[1]["output"].(string)
			n.Config = config
			out <- n
		}
		close(out)
	}()
	return out
}
 
func getVersion(in <-chan EosNode) <-chan EosNode {
	out := make(chan EosNode)
	go func() {
		for n := range in {
			cmds := []string{"show version"}
			url := buildUrl(n)
			response := eapi.Call(url, cmds, "json")
			version := response.Result[0]["version"].(string)
			n.Version = version
			out <- n
		}
		close(out)
	}()
	return out
}

Note: I had a small helper function in there called buildUrl to create the eAPI URL.

Finally the consumer (or sink) in this case is just a for loop in main() that grabs the results from the channel:


	for i := 0; i < len(switches); i++ {
		node := <-out
		fmt.Println(node)
}

This comes after I call my functions, so the whole main() function looks like this:

func main() {
	swFilePtr := flag.String("swfile", "switches.json", "A JSON file with switches to fetch")
	flag.Parse() // command-line flag parsing
	switches := readSwitches(*swFilePtr)
 
	fmt.Println("############# Using Pipelines ###################")
	c1 := genSwitches(switches)
	c2 := getConfigs(c1)
	out := getVersion(c2)
	for i := 0; i < len(switches); i++ {
		node := <-out
		fmt.Println(node)
	}
}

In the above I start with the producer that creates a channel c1, then getConfigs takes that and produces a new channel c2 after processing. c2 is then fed into getVersion to produce yet another channel. Finally we consume it all. If I were to add more functions, I could keep chaining those channels together to grab all kinds of data from the switches. Here’s the complete program:

Advertisements

eAPI Python script to look at ARP entries per VRF

I needed to see all the different ARP entries in each VRF, so I wrote up this little script to do just that. The ‘show vrf’ command in eAPI has not yet been converted to JSON, so I had to do some text parsing to get the VRF names, then use those names to grab the ARP entries. On line 4 you’ll see that I use the ‘text’ option for the output of the JSON reply. That allows me to run a command that hasn’t been converted yet and get the raw text output:

response = switch.runCmds( 1, ["show vrf"], "text" )

The output looks like this:

"output": "   Vrf         RD            Protocols       State         Interfaces \n----------- ------------- --------------- ---------------- ---------- \n   test        100:100       ipv4            no routing               \n   test2       101:101       ipv4            no routing               \n   test3       102:102       ipv4            no routing               \n\n"

Or in a more familiar format:

   Vrf         RD            Protocols       State         Interfaces
----------- ------------- --------------- ---------------- ----------
   test        100:100       ipv4            no routing
   test2       101:101       ipv4            no routing
   test3       102:102       ipv4            no routing

Then I take the output and use splitlines() to take each line (separated by newline) and insert them into a list:

lines = response[0]['output'].splitlines()

Now I iterate through each entry of the ‘show ip vrf’ output and issue a ‘show ip arp vrf’ with the VRF name. I use the range() function, starting at the 3rd line (since the first two are just header lines), and go through the end of the list. Then I use the split() method to split each line on whitespace, taking the first entry which corresponds to the VRF name. Finally, I can use that VRF name in my command.

for i in range(2, len(lines) - 1):
  vrfname = lines[i].split()[0]
  command = "show ip arp vrf " + vrfname

Here’s the script in its entirety:

eAPI script to try different IP addresses

I’ve been trying to use code to solve more problems around the lab lately, and thought I’d start posting some of the little scripts I write. Today I had plugged a device (device A) into a switch and didn’t know what the device had set for its gateway (I did have the IP of the device itself, and it wasn’t .1 or .254). I didn’t have access to the configuration, so I thought I’d write a script to go through possible IP addresses and see if one of them would take until the owner got back to me. I had another machine trying to ping the IP address from a different subnet, so if the right gateway address was configured on the Ethernet port, I should start getting pings:

Device A — Switch — Test pinging machine
2.2.2.2 1.1.1.10

Now I started my constant ping from 1.1.1.1 to 2.2.2.2, then I created and ran this Python script to find the right address.

Arista eAPI from Microsoft PowerShell

I haven’t really played around with Windows in a while, but I’ve had a few people show me some cool things in PowerShell, so I thought I’d give it a try with eAPI. Here’s a really simple script that is able to fetch information from an Arista switch, and put it into an PowerShell object so that it can be used for whatever you’d like. Now I just need to find an excuse to buy a Surface Pro 2 🙂

I start off by just setting up some variables for the username, password, etc.. Variables in PowerShell start with a $ sign.

$username = "admin"
$password = "admin"
$switchIp = "172.22.28.157"

I’m able to insert variables directly into the string for the URL.

#URL
$eApiUrl = "https://$switchIp/command-api"

Now I create an array to hold the commands I want to send, and put that inside a hash table. Arrays are created with @() and Hash tables (Dictionaries in Python, Maps in Go) with @{}:

$cmds = @('show version')
$params = @{version= 1;cmds= $cmds; format="json"}

Now I create a new PowerShell object with all the required fields. PowerShell has this cool pipe operator (|) like Unix and Elixir. This allows you string together a bunch of stuff, in this case we end with piping the output to ConvertTo-Json to turn the object into a JSON string. Then I have to convert that string into an ASCII one to make it web friendly:

$command = (New-Object PSObject | Add-Member -PassThru NoteProperty jsonrpc '2.0' |
Add-Member -PassThru NoteProperty method 'runCmds' |
Add-Member -PassThru NoteProperty params $params |
Add-Member -PassThru NoteProperty id '1') | ConvertTo-Json
$bytes = [System.Text.Encoding]::ASCII.GetBytes($command)

After we have our command ready to go, we create the web connection and POST the JSON-RPC call. I also tell the system to ignore the web certificate since I haven’t installed the cert for the SSL connection.

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$web = [System.Net.WebRequest]::Create($eApiUrl)
$web.Method = "POST"
$web.ContentType = "application/json"
$web.Credentials = New-Object System.Net.NetworkCredential -ArgumentList $username, $password
$stream = $web.GetRequestStream()
$stream.Write($bytes, 0, $bytes.Length)
$stream.close()

Finally we take the response, and put it back into a PowerShell object using ConvertFrom-Json

$reader = New-Object System.IO.StreamReader -ArgumentList $web.GetResponse().GetResponseStream()
$response = $reader.ReadToEnd() | ConvertFrom-Json
$reader.Close()

Once we’ve got an object we can pull out pieces of the response:

$response.result
Write-Host "Model is: " + $response.result.modelName

This is what the output looks like:
Screen Shot 2014-08-20 at 4.44.14 PM

Here’s the full script from start to finish:

Arista eAPI (JSON-RPC over HTTP) in Go

I’ve been wanting to try out Go for a while, and finally decided to give it a try. This is a first stab at using Go to communicate with Arista eAPI via JSON-RPC over HTTP. There is a standard JSON-RPC library in Go, but unfortunately it doesn’t work over HTTP. Here is the code:

I start out by defining a package. Since I’m just using this standalone at this point, we’re using package main.
Next I import a few libraries. One to note is the "github.com/mitchellh/mapstructure" library. This is a handy tool for decoding map structures into Go structures, which we’ll make use of when taking in the ‘unknown’ JSON data and putting it into a struct. To install this library Go has a handy tool that can fetch directly from Github:

go get github.com/mitchellh/mapstructure

Now I move on to defining structs to hold our data in. The first three: Parameters, Request, and JsonRpcResponse are used to decode the initial JSON-RPC stuff. Just as a refresher here’s what JSON-RPC request will look like:

{
   "jsonrpc": "2.0",
   "method": "runCmds",
   "params": {
      "version": 1,
      "cmds": [
         "show version"
      ],
      "format": "json"
   },
   "id": "1"
}

Some things to note when looking at the structs for newcomers to Go like myself:
* The `json:"jsonrpc"` tags tell the JSON library to use that as the actual JSON object name instead of the name given in the struct. I ended up having to do that a lot due to the need for having the struct variable name be capitalized.
* I use the type []map[string]interface{} whenever I’m dealing with data that is not known beforehand, such as the result from the RPC.

Starting a line 54, I create a function to call eAPI via HTTP. I start by filling out the JSON Request struct fields, then marshaling them into a JSON object. After we have the JSON object, I execute an HTTP POST to send the command over and return the response as the return value of the function. Before returning the response I run it through decodeEapiResponse to take the raw response data and put it inside a JsonRpcResponse struct. All of this gets called down in the main function in lines 107-110.

Next you’ll see two functions I implemented to decode the result further into more usable structs. The first was my initial attempt for ShowVersion, which I wrote before I found the mapstructure library. This takes the response and puts everything into the appropriate fields of the struct and returns a ShowVersion struct.

The second decoding function makes use of the mapstructure library and is much cleaner. Now mapstructure takes care of all the manual mapping I did in the previous function for me.

I hope to take this and use goroutines for the calls, that will allow these to happen concurrently. The upside of this will be that I could have commands sent to a number of switches more efficiently. Go isn’t quite as easy as Python, but has some definite advantages in speed, concurrency, and static type checking that make it a useful alternative when performance matters. Any comments or suggestions for improvements are certainly welcome! The code can be cloned from github:
https://github.com/fredhsu/go-eapi.git

Arista JSON eAPI example

One of the great things about working with EOS is the ability to script with JSON-RPC.  No longer does a network admin need to do screen scraping, you can get clean, machine-friendly data from the switch using CLI syntax you’re familiar with.  I’ll outline a simple example using Python.

First add jsonrpclib to your Python environment:

sudo easy_install jsonrpclib

Now we can use that library to make scripting to EOS pretty easy:

from jsonrpclib import Server
switches = ["172.22.28.156", "172.22.28.157", "172.22.28.158"]
username = "admin"
password = "admin"

So far I’ve setup a list of switch IP addresses, and a username/password to use to login to each of them. Now let’s do something useful:

# Going through all the switch IP addresses listed above
for switch in switches:
    urlString = "https://{}:{}@{}/command-api".format(username, password, switch) #1
    switchReq = Server( urlString ) #2
    # Display the current vlan list
    response = switchReq.runCmds( 1, ["show vlan"] ) #3
    print "Switch : " + switch + " VLANs: " 
    print response[0]["vlans"].keys() #4

Now I iterate through each of the switches in the list. On each iteration the script does the following:
1) Creates a string that defines the url to reach the API
2) Start creating a JSON-RPC request with the url
3) Finish building the JSON-RPC request and send the HTTP POST with the commands I want to run on the switch. The JSON response is stored in response. The JSON-RPC library returns the “result” field automatically, so there is no need to parse through the boilerplate JSON-RPC reply.
4) Print out each of the VLANs configured on the switch. The response from the switch is a list, so first I grab the first (in this case only) item indexed by 0. This gives me a dictionary. Next I use the vlans key to select an object from the dictionary. This returns another dictionary, which has the VLAN names as the keys (and details as the values). Since I want to print a list of all the VLANs, I use the keys() method which returns a list of all the keys in the dictionary. Here is the JSON that is being parsed:

{
   "jsonrpc": "2.0",
   "result": [
      {
         "sourceDetail": "",
         "vlans": {
            "1": {
               "status": "active",
               "name": "default",
               "interfaces": {
                  "Ethernet14": {
                     "privatePromoted": false
                  },
                  "Ethernet15": {
                     "privatePromoted": false
                  },
                  "Ethernet16": {
                     "privatePromoted": false
                  },
                  "Ethernet17": {
                     "privatePromoted": false
                  },
                  "Ethernet13": {
                     "privatePromoted": false
                  }
               },
               "dynamic": false
            },
            "51": {
               "status": "active",
               "name": "VLAN0051",
               "interfaces": {
                  "Vxlan1": {
                     "privatePromoted": false
                  }
               },
               "dynamic": false
            },
            "61": {
               "status": "active",
               "name": "VLAN0061",
               "interfaces": {
                  "Vxlan1": {
                     "privatePromoted": false
                  }
               },
               "dynamic": false
            }
         }
      }
   ],
   "id": "CapiExplorer-123"
}

Here’s the full script that also adds a few lines to configure a vlan:

JSON RPC in Objective C

I am working on a side project for work using JSON RPC from an iPhone (Arista eAPI calls from an iPhone app), so I had to get familiar with using JSON RPC using the iOS Objective C libraries. Here’s a quick rundown of how it works:

There are two major components:
NSJSONSerialization – Converts JSON Data to Dictionaries/Arrays, and vice versa
NSURLConnection – Handles HTTP connection to the API server

First I create a NSMutableDictionary with all the keys/values that I want to encode into JSON to send to the server (or switch in this case). In this example, commands is an array of strings, specific to eAPI calls, and paramsDict puts the other eAPI wrappers around the API call. Then I wrap paramsDict inside the JSON RPC Request format, which is rpcDict:

    NSMutableDictionary *paramsDict = [NSMutableDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithInt:1], @"version",
                                       commands, @"cmds",
                                       @"json", @"format", nil];
    NSMutableDictionary *rpcDict = [NSMutableDictionary dictionaryWithObjectsAndKeys:@"2.0", @"jsonrpc",
                                    @"runCmds", @"method",
                                    paramsDict, @"params",
                                    @"CapiExplorer-123", @"id", nil];

Now I take rpcDict, and serialize it into a NSData JSON object, the resulting jsonData is ready to be sent to the API server:

    NSError *error;
    NSData *jsonData = [NSJSONSerialization dataWithJSONObject:rpcDict options:NSJSONWritingPrettyPrinted error:&error];

Now I create the URL request that the application will use to make the RPC request. In this case I’m providing my username and password in the URL string, and _aristaSwitch.switchIP is an NSString containing the IP address of the switch. Also note that I needed to set the content type, and used the jsonData data object as the body:

    NSString *urlString = [NSString stringWithFormat:@"http://user:pass@%@/command-api",
                           _aristaSwitch.switchIP];
    NSURL *url = [NSURL URLWithString:urlString];
    NSMutableURLRequest *request = [[NSMutableURLRequest alloc] init];
    [request setURL:url];
    [request setHTTPMethod:@"POST"];
    [request setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
    [request setValue:@"application/json" forHTTPHeaderField:@"Accept"];
    [request setHTTPBody:jsonData];

Now I can make the connection, send the request, and start receiving a reply:

    NSURLConnection *connection = [[NSURLConnection alloc]initWithRequest:request delegate:self];
    [connection start];

Once I’ve sent the request, I need to then handle the data coming back from the server. To do this, I have the class implement NSURLConnectionDelegate and NSURLConnectionDataDelegate. These delegates handle the connection response, and downloading data. Now I implement the necessary functions, the first two handle the initial response and data stream. I’m using receivedData as a global variable to hold the data coming from the server:

- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
{
    // It can be called multiple times, for example in the case of a
    // redirect, so each time we reset the data.
    [self.receivedData setLength:0];
}

- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
    // Append the new data to receivedData.
    [self.receivedData appendData:data];
}

The final method to implement takes care of the data once the connection has finished. Here I use NSJSONSerialization again to turn the received JSON data back into an NSDictionary that I can use.

- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
    NSError *error;
    
    NSDictionary *myDictionary = [NSJSONSerialization JSONObjectWithData:self.receivedData options:NSJSONReadingMutableContainers error:&error];
    NSLog(@"myDict %@ %@", myDictionary, error);
    connection = nil;
    self.receivedData = nil;
}