Arista eAPI (JSON-RPC over HTTP) in Go

I’ve been wanting to try out Go for a while, and finally decided to give it a try. This is a first stab at using Go to communicate with Arista eAPI via JSON-RPC over HTTP. There is a standard JSON-RPC library in Go, but unfortunately it doesn’t work over HTTP. Here is the code:

I start out by defining a package. Since I’m just using this standalone at this point, we’re using package main.
Next I import a few libraries. One to note is the "github.com/mitchellh/mapstructure" library. This is a handy tool for decoding map structures into Go structures, which we’ll make use of when taking in the ‘unknown’ JSON data and putting it into a struct. To install this library Go has a handy tool that can fetch directly from Github:

go get github.com/mitchellh/mapstructure

Now I move on to defining structs to hold our data in. The first three: Parameters, Request, and JsonRpcResponse are used to decode the initial JSON-RPC stuff. Just as a refresher here’s what JSON-RPC request will look like:

{
   "jsonrpc": "2.0",
   "method": "runCmds",
   "params": {
      "version": 1,
      "cmds": [
         "show version"
      ],
      "format": "json"
   },
   "id": "1"
}

Some things to note when looking at the structs for newcomers to Go like myself:
* The `json:"jsonrpc"` tags tell the JSON library to use that as the actual JSON object name instead of the name given in the struct. I ended up having to do that a lot due to the need for having the struct variable name be capitalized.
* I use the type []map[string]interface{} whenever I’m dealing with data that is not known beforehand, such as the result from the RPC.

Starting a line 54, I create a function to call eAPI via HTTP. I start by filling out the JSON Request struct fields, then marshaling them into a JSON object. After we have the JSON object, I execute an HTTP POST to send the command over and return the response as the return value of the function. Before returning the response I run it through decodeEapiResponse to take the raw response data and put it inside a JsonRpcResponse struct. All of this gets called down in the main function in lines 107-110.

Next you’ll see two functions I implemented to decode the result further into more usable structs. The first was my initial attempt for ShowVersion, which I wrote before I found the mapstructure library. This takes the response and puts everything into the appropriate fields of the struct and returns a ShowVersion struct.

The second decoding function makes use of the mapstructure library and is much cleaner. Now mapstructure takes care of all the manual mapping I did in the previous function for me.

I hope to take this and use goroutines for the calls, that will allow these to happen concurrently. The upside of this will be that I could have commands sent to a number of switches more efficiently. Go isn’t quite as easy as Python, but has some definite advantages in speed, concurrency, and static type checking that make it a useful alternative when performance matters. Any comments or suggestions for improvements are certainly welcome! The code can be cloned from github:
https://github.com/fredhsu/go-eapi.git

Arista JSON eAPI example

One of the great things about working with EOS is the ability to script with JSON-RPC.  No longer does a network admin need to do screen scraping, you can get clean, machine-friendly data from the switch using CLI syntax you’re familiar with.  I’ll outline a simple example using Python.

First add jsonrpclib to your Python environment:

sudo easy_install jsonrpclib

Now we can use that library to make scripting to EOS pretty easy:

from jsonrpclib import Server
switches = ["172.22.28.156", "172.22.28.157", "172.22.28.158"]
username = "admin"
password = "admin"

So far I’ve setup a list of switch IP addresses, and a username/password to use to login to each of them. Now let’s do something useful:

# Going through all the switch IP addresses listed above
for switch in switches:
    urlString = "https://{}:{}@{}/command-api".format(username, password, switch) #1
    switchReq = Server( urlString ) #2
    # Display the current vlan list
    response = switchReq.runCmds( 1, ["show vlan"] ) #3
    print "Switch : " + switch + " VLANs: " 
    print response[0]["vlans"].keys() #4

Now I iterate through each of the switches in the list. On each iteration the script does the following:
1) Creates a string that defines the url to reach the API
2) Start creating a JSON-RPC request with the url
3) Finish building the JSON-RPC request and send the HTTP POST with the commands I want to run on the switch. The JSON response is stored in response. The JSON-RPC library returns the “result” field automatically, so there is no need to parse through the boilerplate JSON-RPC reply.
4) Print out each of the VLANs configured on the switch. The response from the switch is a list, so first I grab the first (in this case only) item indexed by 0. This gives me a dictionary. Next I use the vlans key to select an object from the dictionary. This returns another dictionary, which has the VLAN names as the keys (and details as the values). Since I want to print a list of all the VLANs, I use the keys() method which returns a list of all the keys in the dictionary. Here is the JSON that is being parsed:

{
   "jsonrpc": "2.0",
   "result": [
      {
         "sourceDetail": "",
         "vlans": {
            "1": {
               "status": "active",
               "name": "default",
               "interfaces": {
                  "Ethernet14": {
                     "privatePromoted": false
                  },
                  "Ethernet15": {
                     "privatePromoted": false
                  },
                  "Ethernet16": {
                     "privatePromoted": false
                  },
                  "Ethernet17": {
                     "privatePromoted": false
                  },
                  "Ethernet13": {
                     "privatePromoted": false
                  }
               },
               "dynamic": false
            },
            "51": {
               "status": "active",
               "name": "VLAN0051",
               "interfaces": {
                  "Vxlan1": {
                     "privatePromoted": false
                  }
               },
               "dynamic": false
            },
            "61": {
               "status": "active",
               "name": "VLAN0061",
               "interfaces": {
                  "Vxlan1": {
                     "privatePromoted": false
                  }
               },
               "dynamic": false
            }
         }
      }
   ],
   "id": "CapiExplorer-123"
}

Here’s the full script that also adds a few lines to configure a vlan:

OVSDB Client in Python Part 2

In the last post I left off simply connecting to the OVSDB, and sending/receiving messages. Now I want to add in a couple things:
1) Handling of Echo messages – There should be periodic Echo messages sent back and forth between my client and the ovsdb-server
2) Handling of updates – When we issue a command like `monitor`, not only do we get a snapshot of the database at that time, but we also register ourselves for updates. So I’d like the client to keep listening for updates to OVSDB and allow me to see them

Using the `recv` socket command like we did in the beginning would require the client to keep looping around waiting for a message and that doesn’t seem very efficient. Instead I’m going to leverage select. What `select` will let me do is to wait for incoming data, then take action when we hear something, or take action when there is outgoing data waiting to be processed. Here’s an example:

def listen_for_messages(sock, message_queues):
    # To send something, add a message to queue and append sock to outputs
    inputs = [sock, sys.stdin]
    outputs = []
    while sock:
        readable, writable, exceptional = select(inputs, outputs, [])
        for s in readable:
            if s is sock:
                data = sock.recv(4096)
                message_queues[sock].put(data)
                outputs.append(sock)
                print "recv:" + data
            elif s is sys.stdin:
                print sys.stdin.readline()
                sock.close()
                return
            else:
                print "error"
        for w in writable:
            if w is sock:
                sock.send(message_queues[sock].get_nowait())
                outputs.remove(sock)
            else:
                print "error"

To start things off, I create a few lists that will hold the inputs and outputs I plan to monitor. In this function I initialize the inputs to be the socket connection and stdin, and outputs are empty. Next I create a while loop that will run as long as the socket is open: `while sock:`. Now we get to the select statement. I pass in the inputs and outputs that will be monitored, and the select will return whatever inputs or outputs have data that need to be processed: `readable, writable, exceptional = select(inputs, outputs, [])`
If there are any inputs, they will be assigned to the variable `readable` and we will enter the first for loop. There we check to see if there is socket data or stdin data. When we get data on a socket, we will simply copy that data into our message queue (this is a quick and dirty way to reply to an echo), and put the socket into the output list. If its something from stdin (i.e. user entering stuff on the keyboard), I will just close the socket for now. As you can imagine, one could easily use this to allow the user to send commands to the ovsdb-server with some extra modification.
Now I handle the output list, much the same way I did the input list. In this case, if there is socket data in the output list, I pull the data off the message queue, and send it out the socket. Then I remove the socket from the output queue to leave it empty again.
That should take care of our basic message handler. Using select, we’re able to handle various inputs and outputs in a efficient manner, and in this case, receive and send echo messages. This will keep the connection to the ovsdb-server alive for us. Also, if we had issued a monitor command, it will receive those updates from the server and print them to screen.

If another function wants to send messages out to the server, we would need to slightly modify things so that we can add messages to the queue and put the socket into the output list. I’ve posted a modified version of this code on github where you can have a look if interested: https://github.com/fredhsu/py-ovsdb-client

ODL subnet REST API, Python, and some error handling

Its been a while since I’ve had a chance to play around on OpenDaylight for a while, so I thought I’d warm up with some Python and API calls. One thing I haven’t done much of with my code so far is handling errors, so in this post I’m going to access the Subnet API, and try to do some error handling as well from Python.

I recently found a nice HTTP client called Requests that simplifies the HTTP requests. You can add it using pip:

pip install requests

And then import it into your Python code:

import requests
from requests.auth import HTTPBasicauth

I’ve also imported the auth stuff for logging into the controller.
Now making a GET requests is simple, here is an example of querying the controller for all the configured subnets:

user = 'admin'
password = 'admin'
servierIP = '10.55.17.20'
container = 'default'
allSubnets = '/controller/nb/v2/subnet/' + container + '/subnet/all'
url = 'http://' + serverIP + ':' + port + allSubnets
r = requests.get(url, auth=(user, password))
print r.json()

Requests can automatically take the response from the controller and give you back the JSON data. Now lets add some error handling by wrapping everything with try/except/else:

errorcodes = {
    400: 'Invalid data',
    401: 'User not authorized',
    409: 'Name conflict',
    404: 'Container name not found',
    500: 'Internal error',
    503: 'Service unavailable'
    }

try:
    r = requests.get(url, auth=(user, password))
    r.raise_for_status()
except requests.exceptions.HTTPError as e:
    print e
    print "Reason : %s" % errorcodes[r.status_code]
else:
    # No errors loading URL
    result = find_subnet(r.json()['subnetConfig'], subnetquery)
    print result

The first thing we do is use r.raise_for_status() which returns null if all goes well, otherwise it will raise an exception. One of the exceptions that can be raised is an HTTPError, in which case we’ll print out the error.

If everything is ok, then I pass the resulting JSON to a find_subnet function which just searches the JSON list for a particular subnet:

# given a list of subnets and a subnet to find, will return the subnet if found
# or None if not found
def find_subnet(subnets, subnetName):
    for subnet in subnets:
        if subnet['subnet'] == subnetName:
            return subnet
    return None

Nothing too crazy here, but just another Python example for those who are interested. Also, please note that the APIs have changed here and there, so be sure to check the URLs you are calling to make sure they are correct. I’ve found that the API docs on the OpenDaylight wiki are not always in sync with the version of controller I’m using. You can find the full script here:
https://github.com/fredhsu/odl-scripts/tree/master/python/subnets

Handling packets on the OpenDaylight controller

One of the actions that an OpenFlow switch can take is to punt a packet to the controller.  This example will take a look at how we can see those packets, and do something with them.  I hope to follow this up with another post that does something more exciting, but for now I’ll just try to print out what type of packet it is.  This is one of the things that (as far as I know)  you would only be able to do with an OSGi module, and is not available via the REST API.

First we create our Maven pom.xml with the required imports. In this case we’ll need some parts of the SAL and switchmanager:

 
Now we can create our activator. The key ingredient here is to register for callbacks from the Data Packet Service via OSGi in our public void configureInstance method:

            c.add(createContainerServiceDependency(containerName).setService(
                    IDataPacketService.class).setCallbacks(
                    "setDataPacketService", "unsetDataPacketService")
                    .setRequired(true));

This ties into methods that we implement in our GetPackets class:

    void setDataPacketService(IDataPacketService s) {
        this.dataPacketService = s;
    }

    void unsetDataPacketService(IDataPacketService s) {
        if (this.dataPacketService == s) {
            this.dataPacketService = null;
        }
    }

We make the class implement the IListenDataPacket interface to get notified of packets received on the controller:

public class GetPackets implements IListenDataPacket

And we override the public PacketResult receiveDataPacket(RawPacket inPkt) method:

    @Override
    public PacketResult receiveDataPacket(RawPacket inPkt) {
        if (inPkt == null) {
            return PacketResult.IGNORED;
        }
        log.trace("Received a frame of size: {}",
                        inPkt.getPacketData().length);
        Packet formattedPak = this.dataPacketService.decodeDataPacket(inPkt);
        System.out.println("packet");
        System.out.println(formattedPak);        
        if (formattedPak instanceof Ethernet) {
            System.out.println(formattedPak);
            Object nextPak = formattedPak.getPayload();
            if (nextPak instanceof IPv4) {
                IPv4 ipPak = (IPv4)nextPak;
                System.out.println("IP");
                log.trace("Handled IP packet");
                int sipAddr = ipPak.getSourceAddress();
                InetAddress sip = NetUtils.getInetAddress(sipAddr);
                int dipAddr = ipPak.getDestinationAddress();
                InetAddress dip = NetUtils.getInetAddress(dipAddr);
                System.out.println("SRC IP:");
                System.out.println(sip);
                System.out.println("DST IP:");
                System.out.println(dip);

                Object frame = ipPak.getPayload();
                if (frame instanceof ICMP) {
                    System.out.println("ICMP from instance");
                }
                String protocol = IPProtocols.getProtocolName(ipPak.getProtocol());
                if (protocol == IPProtocols.ICMP.toString()) {
                    ICMP icmpPak = (ICMP)ipPak.getPayload();
                    System.out.println("ICMP from checking protocol");
                    handleICMPPacket((Ethernet) formattedPak, icmpPak, inPkt.getIncomingNodeConnector());
                }
            }
        }
        return PacketResult.IGNORED;
    }

You’ll notice that we can keep going into the different payloads of the frame/packet to get to the next network layer. However, using instanceof can be slow, so an alternative is to pull out the protocol field, and do a comparison. In my example I’ve specifically handled ICMP packets, and used both methods for determining if the IP packet is ICMP.

Unit Testing OpenDaylight code with Mininet and Python

I recently got pinged by Dale Carder from the University of Wisconsin regarding a python API he is developing for ODL.  The API is a nice step in relieving some of the tedium when dealing with the ODL REST API.
One of the cool things he’s done as part of his project is create some unit tests for his code using the python API for mininet, coupled with his Python API code.  Unit tests are a great way to make sure the code you’re creating does what it should do, and keeps doing the right things when you make changes. They are key to practices such as Test Driven Development(TDD). Combining Mininet API calls with ODL API calls could be a powerful tool for creating network applications. Let’s take a closer look and see what he’s done to leverage that Mininet API:

First he creates a class for to define the topology:

class SingleSwitchTopo(Topo):
    "Single switch connected to n hosts."
    def __init__(self, n=2, **opts):
        # Initialize topology and default options
        Topo.__init__(self, **opts)
        # mininet/ovswitch does not want ':'s in the dpid
        switch_id = SWITCH_1.translate(None, ':')
        switch = self.addSwitch('s1', dpid=switch_id)
        for h in range(n):
            host = self.addHost('h%s' % (h + 1))
            self.addLink(host, switch)

This gives us a Mininet instance with a switch with hosts that can be used to test the API calls.

Next he starts up the test network:

def setup_mininet_simpleTest():
    "Create and test a simple network"
    topo = SingleSwitchTopo(n=4)
    #net = Mininet(topo)
    net = Mininet( topo=topo, controller=lambda name: RemoteController( 
                   name, ip=CONTROLLER ) )
    net.start()

Then finally he setups the tests and tests the API calls that he makes. Here is the setup and one of the test cases:

class TestSequenceFunctions(unittest.TestCase):
    """Tests for OpenDaylight

       At this point, tests for OpenDaylightFlow and OpenDaylightNode
       are intermingled.  These could be seperated out into seperate
       suites.
    """

    def setUp(self):
        odl = OpenDaylight()
        odl.setup['hostname'] = CONTROLLER
        odl.setup['username'] = USERNAME
        odl.setup['password'] = PASSWORD
        self.flow = OpenDaylightFlow(odl)
        self.node = OpenDaylightNode(odl)

        self.switch_id_1 = SWITCH_1

        self.odl_test_flow_1 = {u'actions': u'DROP',
           u'etherType': u'0x800',
           u'ingressPort': u'1',
           u'installInHw': u'true',
           u'name': u'odl-test-flow1',
           u'node': {u'@id': self.switch_id_1, u'@type': u'OF'},
           u'priority': u'500'}

        self.odl_test_flow_2 = {u'actions': u'DROP',
           u'etherType': u'0x800',
           u'ingressPort': u'2',
           u'installInHw': u'true',
           u'name': u'odl-test-flow2',
           u'node': {u'@id': self.switch_id_1, u'@type': u'OF'},
           u'priority': u'500'}


    def test_01_delete_flows(self):
        """Clean up from any previous test run, just delete these
            flows if they exist.
        """
        try:
            self.flow.delete(self.odl_test_flow_1['node']['@id'],
                             self.odl_test_flow_1['name'])
        except:
            pass

        try:
            self.flow.delete(self.odl_test_flow_2['node']['@id'],
                             self.odl_test_flow_2['name'])
        except:
            pass

You can find the beginnings of his API here:

http://net.doit.wisc.edu/~dwcarder/scripts/opendaylight/
please keep in mind its still very early goings, and a work in progress. Thanks goes out to Dale for letting me borrow his code for this post.

Aside

Overlay and Underlay networks

I just read an article from Greg Ferro about Integerating Overlay and Physical Networks http://etherealmind.com/integrating-overlay-networking-and-the-physical-network/?utm_source=feedly&utm_medium=feed&utm_campaign=Feed%3A+etherealmind+%28My+Etherealmind+-+Network+design%2C+architecture%2C+thinking%2C+working.+Tech.%29 which is a nice compliment to an interesting demo I just saw at Cisco Live. The demo showed how the XNC can be used to manage an Overlay/Underlay network using OpenFlow to manage the traffic flows between OVS instances with GRE tunnels (overlay), and onePK to manage the physical network (underlay).  

For instance, you may have traffic that needs low latency, and other traffic that needs maximum bandwidth.  You could specify these two types of flows using the TIF (Topology Indpenednet Forwading) feature on the controller.  Then you could tie them to “IPSLA” tags that will indicate to the controller what characteristics the traffic wants.  The controller is aware of both the overlay and underlay networks, and can coordinate the configuration between them. OnePK is used to inject “application routes” into the underlay network that dictate which path the GRE tunnels take, based on the same tags used in the OpenFlow configuration.

I hope to grab some screenshots and add them to this post, but this seems like a cool use case for the controller. I also hope that some of this functionality will make its way into OpenDaylight.