Monday, July 15, 2013

Building a better internet exchange with OpenFlow

Internet exchanges

The internet is a collection of cables and routers - cables join the routers together, and routers decide where your data should go. Internet exchanges perform a crucial job in the middle of this - they provide a way for different (often competing) providers to exchange internet traffic. An internet exchange is typically implemented using some number of switches that are federated together somehow so that every port can exchange traffic with every other port as fast as they need to.

Problems with internet exchanges

Many internet exchanges are implemented as a layer-2 broadcast domain, and often include a route server to make it easier for participants to exchange routes with everyone over a single peering. Layer-2 broadcast domains need to be well policed though, because people can easily mess with other peoples' traffic, either accidentally, or maliciously.

Things that you need to watch out for on a layer-2 broadcast domain:
  • Broadcast traffic - ARP requests are necessary, others probably aren't
  • MAC spoofing
  • ARP spoofing/proxy ARP
  • STP & other lovely protocols
Things that you need at an IPv4/IPv6 peering exchange
  • Unicast IPv4 and IPv6 traffic
  • ARP requests (not necessarily broadcast)
  • Legitimate ARP replies

Where's the button to turn this all on?

You'll often build an internet exchange with cheap and big switches, so you're not likely to have a huge amount of functionality to do clever stuff like this - and I've never seen a switch config before that lets me check the validity of ARP replies. To do this in software, however, is pretty easy.

Software Peering Exchange

Here's something I put together last night:

It's an extension of my OF1.2 learning switch, with an extra table to handle ARP entries. Here's how the tables are laid out:

The controller takes some simple JSON config of which IP is expected on each port, and loads these into the switch. Table 0 forwards all ARP packets to table 1, and table 1 checks the packets - if they're ARP requests, then it forwards them out the appropriate port (so ARP requests are effectively unicast across the internet exchange), and for ARP replies, it only allows each port to reply for its own IP address. Any packets that don't get forwarded out a port get sent to the controller (including valid ARP replies before we learn any MAC addresses, but the controller deals with this smartly & learns the MAC address properly).

This is combined with the learning switch that I posted earlier in the week, with one alteration - our source-MAC matches also have to match the ethertype, and this can be constrained in the controller to limit it to IPv4 or IPV6 (or either or - this could be done on a per port basis if you wanted to). Packets with known source-MACs are send to table 2, and then forwarded out the right port if we've learned their MAC - otherwise, it goes to the controller for processing.

What else can we do?

We can lock down the controller a bit to only allow IPv4, IPv6 and ARP - this would stop CDP, STP and the like leaking into the exchange - only valid traffic, and no spoofing.

We could go a step further though - we could build in ACLs on each port based on the routes that they announce for basic RPF, so that a participant can only route traffic from an IP range if they advertise that route to the exchange already - this would prevent asymmetric routing.

So go, have a play - see if you can break it or find something else to add!

Sunday, July 14, 2013

OpenFlow 1.2 switch app with multiple tables

Another OpenFlow switch app?

I'm sorry - there's quite a few of these, but this one is cool. I've been part of a couple of OpenFlow bootcamps in the last year and a bit, and an example that I've used is the pyswitch app that came stock with NOX. It's only a few lines of functional code, but it's a nice example to show what OpenFlow controller code looks like, and how you can easily build a learning switch from scratch. We'd go through a couple of scenarios on the whiteboard, but then we'd find a problem that OpenFlow 1.0 couldn't solve.

Why we need multiple flow tables

Let's say we have two nodes connected to the OpenFlow switch. Node 1 sends a packet to node 2, and the switch doesn't know this MAC address, so it passes the packet up to the controller. The controller learns the MAC address of node 1 (where the packet came from), and assigns it to port 1. It then pushes a flow to the controller to make sure that traffic for node 1 goes out the right port, and then sends the packet on its way.

What happens when node 2 replies?

The switch gets the packet for node 1, has a flow for it, and forwards it. There's a problem here though - because the packet never went to the controller, the switch never learns the MAC address of node 2 - so all traffic to node 2 will go via the controller and clog things up.

The OpenFlow learning switch apps that I've seen deal with this by storing pairs - they match on a destination MAC address *and* on the source MAC address - this way, whenever we get a packet from a new MAC address, we make sure the controller knows and pushes out the right flow. This works perfectly, but it's inefficient - we need O(n^2) flows, meaning 100 MAC address on a switch requires 20,000 flows (one flow in each direction = 2 flows for each pair of mac addresses).

This is where later versions of OpenFlow start to shine. We can create multiple flow tables, so we have an initial table to check the source MAC (and forward to the controller if we don't know it), and a second table to check the destination MAC (and forward to controller, or just flood, if we don't know it). This way we need to record each source MAC once, and each destination MAC once - so 100 MAC addresses on a switch only needs 200 flows - only O(n).

What it looks like

I've spent the whole weekend getting OpenVSwitch working so I could test it, but this is the final product. My good friend Josh suggested I use the Ryu controller as it has OpenFlow 1.2 and 1.3 support. It comes with a simple_switch application for OpenFlow 1.0, so I modified this to make it work with OpenFlow 1.2. There's a couple of subtle differences between the protocols, but once it was refactored for 1.2, it was pretty easy to set up a second table. The code is here if you want it, and you can just clone this repo when you want to use Ryu - it's up to date with the current github version (as of 14 July 2013), but I'm hoping they'll accept my pull request and just add my app into the main build.

Next steps

I really want to get BIRD working with an OpenFlow controller, so we'll see if that happens - I've got BIRD pushing out a stream of JSON routes that anything can pick up, and I just need this to get turned into flows. It really does feel like we're getting closer to a production OpenFlow controller that plugs into a cheap switch and takes a whole internet route table. Now is a very good time to be a network engineer!

How to build an OpenFlow testbed on an Ubuntu 12.04 VM in VirtualBox

Installing Ubuntu

I started with an Ubuntu Server 12.04.2 64 bit iso, and a VirtualBox VM with 1024MB of RAM and 8GB of hard disk. My version of VirtualBox is 4.0.10r72479 on Windows 7 x64 Professional. The install is pretty normal - if you've never installed Ubuntu Server before, you shouldn't find this too hard - just follow the prompts and keep pressing enter.

This would be a good time to pour yourself a single malt coffee

Don't be too fussy about packages, but as a rule I tend to want to install the OpenSSH server just because it's a good habit to get into - and something that'll trip you up if you forget to just that one time when it's really important.

Pro tip ™

This will take a couple of minutes to finish, and then you'll have an Ubuntu install ready to go. Remember to eject the ISO, and then turn the machine off. Time for the ugly stuff.

Configuring the network stuff

I've set up my testbed with 3x Ubuntu servers, 2 of which were set up like this and left, and a third that we did some special stuff with. For all of them, we'll need to set up extra adaptors on Internal Networks - the two client machines each get a single new adaptor with their own intnet, and the OVS machine (the third one) gets two new adaptors - the first one goes onto intnet1 (to connect to client 1), and the second goes onto intnet2. I've left the original adaptor untouched on all of the machines so we can add packages later without having to break networking.

Edit our new OFSwitch2 VM

Keep Adapter 1 as is so we can download stuff

Intnet1 matches up with client VM 1

Intnet2 to client VM 2

Once you've set this up, find the vbox file for your VM and open it up in your text editor, Make sure you close VirtualBox first - otherwise it won't take your changes. You'll want to add the following lines:

<ExtraDataItem name="VBoxInternal/Devices/e1000/1/LUN#0/Config/IfPolicyPromisc" value="allow-all"/>
<ExtraDataItem name="VBoxInternal/Devices/e1000/2/LUN#0/Config/IfPolicyPromisc" value="allow-all"/>

The secret sauce

That last part is super important - I spend a few hours today and last night trying to figure out why some packets would hit the bridge and others wouldn't - VirtualBox by default will accept broadcasts and unicasts to your address, but not other MAC addresses. Being a switch, you generally want to accept every MAC address except your own, so this is fairly important.

Installing OpenVSwitch

I've used version 1.10 because it's the coolest. Download it to a folder on your VM, untar, and read the INSTALL file because that's what cool kids do. In actual fact, there's a INSTALL.Debian, but that didn't work for me, so I just built it the generic way.

Packages to install (so you don't spend the next hour chasing dependencies):

  • build-essential
  • pkg-config
  • autoconf
  • automake
  • python-qt-dev
  • python-dev
  • python-twisted-conch
  • libtool
Then run the install
sudo make install

I'm pleasantly surprised to say that this all worked the first time - just make sure you install all of those packages in one go and it'll work perfectly from the start :)

Running OpenVSwitch

Now is a good time to start up OpenVSwitch to test that everything is working as you should expect - if we do this right, then the OpenFlow part will be easy. Fire up your two client machines, and set up eth1 on both of them to IPs in the same range - I've used and, but use something else if this would clash with your other network.

Once you have them up, start up OpenVSwitch with the following stuff - I've kept them in separate screens to make it easier

Start a screen (screen)

Screen 0:
sudo modprobe openvswitch
sudo ovsdb-tool create

sudo ovsdb-server --remote=ptcp:9999:

New screen(CTRL+A, C)

Screen 1
sudo ovs-vswitchd tcp:

Screen 2
ovs-vsctl --db=tcp: add-br br0
ovs-vsctl --db=tcp: add-port br0 eth1
ovs-vsctl --db=tcp: add-port br0 eth2
ovs-vsctl --db=tcp: set bridge br0 protocols=OpenFlow12
sudo ifconfig eth1 up
sudo ifconfig eth2 up

If you bring up your client VMs you should be able to ping between them now. If you can, then great - we'll move onto getting OpenFlow working. You need one more line of code, assuming the controller is (or will be) on the same machine:

ovs-vsctl --db=tcp: set-controller br0 tcp:

Getting OpenFlow going

We're on the home straight here. You can install the controller of your choosing, or you can install Ryu with the following instructions:

sudo apt-get install git python-setuptools
git clone
cd ryu
sudo python install

You can then sit and watch as it downloads its dependencies from pypi. When it's done, fire up the controller with an app, and you're ready to go.

ryu-manager ryu/app/

You can check the flow tables (in another screen) with the following command:

sudo ovs-ofctl dump-flows br0

Check out the manpages if you want to learn more:

You've got an OpenFlow testbed now, you can do what you want with it. Play with different controllers, or different versions of OpenFlow - it's all up to you.

OpenFlow 1.2 with Ryu PREVIEW

I've been playing with the Ryu OpenFlow controller this weekend, and I've got something that's nearly ready for you - here's a sneak preview

Tuesday, July 2, 2013

SDN plugin for the BIRD software router

A BGP router for SDN and OpenFlow

I've been playing with the BIRD software router for a couple of weeks to make it pipe out routes that I can play with. The reason for this is so that we can leverage the years of development time spent on the BGP side of things, and then simply translate the routes into flows to make a BGP router. There are already projects using RouteFlow for this, but we can simplify things a lot further - RouteFlow relies on VMs that run Quagga instances, and I feel it would be much better if the software router just talked directly to the OpenFlow controller.

What does it look like?

Every time a route update comes through to BIRD, BIRD pipes it out to a file. We can then get SDN controllers to follow this file and pull out the routes in JSON format and process as they wish to.

What's next?

Make an OpenFlow controller that polls the file and pushes out flows to your OF switch(es). This stuff isn't rocket science - just download and install and see it work for yourself!