More Go concurrency using pipelines with eAPI

As a follow on to my previous post on using Go channels for concurrency, I thought I would try and use the pipeline pattern as well.  The idea is to create a series of goroutines that you can string together through channels.  This allows you to mix and match (compose) small functions to build the final result you want.  Its like using the ‘|’ operator in Unix.  For this example I’m going to take a few different show commands I want to run, create pipelined functions out of them, then string them together to pull down the final result I want.

For this example I will go grab the show version and show running-config of a series of Arista switches.  I’ve defined a json file to store the switch names and connection information.  Here is a short function to read in that file and parse the JSON data:

func readSwitches(filename string) []EosNode {
	var switches []EosNode
 
	file, err := os.Open("switches.json")
	if err != nil {
		panic(err)
	}
	decoder := json.NewDecoder(file)
	err = decoder.Decode(&switches)
	if err != nil {
		panic(err)
	}
	return switches
}

To store all the information I created a struct with fields for the relevant data (there are some extra fields here for future use):

type EosNode struct {
	Hostname      string
	MgmtIp        string
	Username      string
	Password      string
	Ssl           bool
	Reachable     bool
	ConfigCorrect bool
	Uptime        float64
	Version       string
	Config        string
	IntfConnected []string
	IpIntf        []string
	Vlans         []string
}

Now I start writing my functions.  There are three types of functions that we need.  First I will write a producer that starts the whole thing off by generating channels for each switch (in this case it will be EosNodes).  Then intermediate functions will take actions on those channels, and return a new channel with an EosNode.  Finally the consumer will take the channels and produce the final result.

The producer (or generator) will take a list of EosNodes, then kick off goroutines for each switch and tie them into the out channel, which I return from the function:

func genSwitches(nodes []EosNode) <-chan EosNode {
	out := make(chan EosNode)
	go func() {
		for _, node := range nodes {
			out <- node
		}
		close(out)
	}()
	return out
}

Now the intermediate functions that receive EosNodes from the channel, runs the eAPI call to fill in more data, then returns a new outbound channel with the new data populated in the EosNode:

func getConfigs(in <-chan EosNode) <-chan EosNode {
	out := make(chan EosNode)
	go func() {
		for n := range in {
			cmds := []string{"enable", "show running-config"}
			url := buildUrl(n)
			response := eapi.Call(url, cmds, "text")
			config := response.Result[1]["output"].(string)
			n.Config = config
			out <- n
		}
		close(out)
	}()
	return out
}
 
func getVersion(in <-chan EosNode) <-chan EosNode {
	out := make(chan EosNode)
	go func() {
		for n := range in {
			cmds := []string{"show version"}
			url := buildUrl(n)
			response := eapi.Call(url, cmds, "json")
			version := response.Result[0]["version"].(string)
			n.Version = version
			out <- n
		}
		close(out)
	}()
	return out
}

Note: I had a small helper function in there called buildUrl to create the eAPI URL.

Finally the consumer (or sink) in this case is just a for loop in main() that grabs the results from the channel:


	for i := 0; i < len(switches); i++ {
		node := <-out
		fmt.Println(node)
}

This comes after I call my functions, so the whole main() function looks like this:

func main() {
	swFilePtr := flag.String("swfile", "switches.json", "A JSON file with switches to fetch")
	flag.Parse() // command-line flag parsing
	switches := readSwitches(*swFilePtr)
 
	fmt.Println("############# Using Pipelines ###################")
	c1 := genSwitches(switches)
	c2 := getConfigs(c1)
	out := getVersion(c2)
	for i := 0; i < len(switches); i++ {
		node := <-out
		fmt.Println(node)
	}
}

In the above I start with the producer that creates a channel c1, then getConfigs takes that and produces a new channel c2 after processing. c2 is then fed into getVersion to produce yet another channel. Finally we consume it all. If I were to add more functions, I could keep chaining those channels together to grab all kinds of data from the switches. Here’s the complete program:

Arista eAPI (JSON-RPC over HTTP) in Go

I’ve been wanting to try out Go for a while, and finally decided to give it a try. This is a first stab at using Go to communicate with Arista eAPI via JSON-RPC over HTTP. There is a standard JSON-RPC library in Go, but unfortunately it doesn’t work over HTTP. Here is the code:

I start out by defining a package. Since I’m just using this standalone at this point, we’re using package main.
Next I import a few libraries. One to note is the "github.com/mitchellh/mapstructure" library. This is a handy tool for decoding map structures into Go structures, which we’ll make use of when taking in the ‘unknown’ JSON data and putting it into a struct. To install this library Go has a handy tool that can fetch directly from Github:

go get github.com/mitchellh/mapstructure

Now I move on to defining structs to hold our data in. The first three: Parameters, Request, and JsonRpcResponse are used to decode the initial JSON-RPC stuff. Just as a refresher here’s what JSON-RPC request will look like:

{
   "jsonrpc": "2.0",
   "method": "runCmds",
   "params": {
      "version": 1,
      "cmds": [
         "show version"
      ],
      "format": "json"
   },
   "id": "1"
}

Some things to note when looking at the structs for newcomers to Go like myself:
* The `json:"jsonrpc"` tags tell the JSON library to use that as the actual JSON object name instead of the name given in the struct. I ended up having to do that a lot due to the need for having the struct variable name be capitalized.
* I use the type []map[string]interface{} whenever I’m dealing with data that is not known beforehand, such as the result from the RPC.

Starting a line 54, I create a function to call eAPI via HTTP. I start by filling out the JSON Request struct fields, then marshaling them into a JSON object. After we have the JSON object, I execute an HTTP POST to send the command over and return the response as the return value of the function. Before returning the response I run it through decodeEapiResponse to take the raw response data and put it inside a JsonRpcResponse struct. All of this gets called down in the main function in lines 107-110.

Next you’ll see two functions I implemented to decode the result further into more usable structs. The first was my initial attempt for ShowVersion, which I wrote before I found the mapstructure library. This takes the response and puts everything into the appropriate fields of the struct and returns a ShowVersion struct.

The second decoding function makes use of the mapstructure library and is much cleaner. Now mapstructure takes care of all the manual mapping I did in the previous function for me.

I hope to take this and use goroutines for the calls, that will allow these to happen concurrently. The upside of this will be that I could have commands sent to a number of switches more efficiently. Go isn’t quite as easy as Python, but has some definite advantages in speed, concurrency, and static type checking that make it a useful alternative when performance matters. Any comments or suggestions for improvements are certainly welcome! The code can be cloned from github:
https://github.com/fredhsu/go-eapi.git