How to build an OpenFlow testbed on an Ubuntu 12.04 VM in VirtualBox

Installing Ubuntu

I started with an Ubuntu Server 12.04.2 64 bit iso, and a VirtualBox VM with 1024MB of RAM and 8GB of hard disk. My version of VirtualBox is 4.0.10r72479 on Windows 7 x64 Professional. The install is pretty normal - if you've never installed Ubuntu Server before, you shouldn't find this too hard - just follow the prompts and keep pressing enter.

This would be a good time to pour yourself a single malt coffee

Don't be too fussy about packages, but as a rule I tend to want to install the OpenSSH server just because it's a good habit to get into - and something that'll trip you up if you forget to just that one time when it's really important.

Pro tip ™

This will take a couple of minutes to finish, and then you'll have an Ubuntu install ready to go. Remember to eject the ISO, and then turn the machine off. Time for the ugly stuff.

Configuring the network stuff

I've set up my testbed with 3x Ubuntu servers, 2 of which were set up like this and left, and a third that we did some special stuff with. For all of them, we'll need to set up extra adaptors on Internal Networks - the two client machines each get a single new adaptor with their own intnet, and the OVS machine (the third one) gets two new adaptors - the first one goes onto intnet1 (to connect to client 1), and the second goes onto intnet2. I've left the original adaptor untouched on all of the machines so we can add packages later without having to break networking.

Edit our new OFSwitch2 VM

Keep Adapter 1 as is so we can download stuff

Intnet1 matches up with client VM 1

Intnet2 to client VM 2

Once you've set this up, find the vbox file for your VM and open it up in your text editor, Make sure you close VirtualBox first - otherwise it won't take your changes. You'll want to add the following lines:

<ExtraDataItem name="VBoxInternal/Devices/e1000/1/LUN#0/Config/IfPolicyPromisc" value="allow-all"/>
<ExtraDataItem name="VBoxInternal/Devices/e1000/2/LUN#0/Config/IfPolicyPromisc" value="allow-all"/>

The secret sauce

That last part is super important - I spend a few hours today and last night trying to figure out why some packets would hit the bridge and others wouldn't - VirtualBox by default will accept broadcasts and unicasts to your address, but not other MAC addresses. Being a switch, you generally want to accept every MAC address except your own, so this is fairly important.

Installing OpenVSwitch

I've used version 1.10 because it's the coolest. Download it to a folder on your VM, untar, and read the INSTALL file because that's what cool kids do. In actual fact, there's a INSTALL.Debian, but that didn't work for me, so I just built it the generic way.

Packages to install (so you don't spend the next hour chasing dependencies):

• build-essential
• pkg-config
• autoconf
• automake
• python-qt-dev
• python-dev
• python-twisted-conch
• libtool

Then run the install

./boot.sh

./configure

make

sudo make install

I'm pleasantly surprised to say that this all worked the first time - just make sure you install all of those packages in one go and it'll work perfectly from the start :)

Running OpenVSwitch

Now is a good time to start up OpenVSwitch to test that everything is working as you should expect - if we do this right, then the OpenFlow part will be easy. Fire up your two client machines, and set up eth1 on both of them to IPs in the same range - I've used 10.1.1.1/24 and 10.1.1.2/24, but use something else if this would clash with your other network.

Once you have them up, start up OpenVSwitch with the following stuff - I've kept them in separate screens to make it easier

Start a screen (screen)

Screen 0:

sudo modprobe openvswitch

sudo ovsdb-tool create

sudo ovsdb-server --remote=ptcp:9999:127.0.0.1

New screen(CTRL+A, C)

Screen 1

sudo ovs-vswitchd tcp:127.0.0.1:9999

Screen 2

ovs-vsctl --db=tcp:127.0.0.1:9999 add-br br0

ovs-vsctl --db=tcp:127.0.0.1:9999 add-port br0 eth1

ovs-vsctl --db=tcp:127.0.0.1:9999 add-port br0 eth2
ovs-vsctl --db=tcp:127.0.0.1:9999 set bridge br0 protocols=OpenFlow12

sudo ifconfig eth1 up

sudo ifconfig eth2 up

If you bring up your client VMs you should be able to ping between them now. If you can, then great - we'll move onto getting OpenFlow working. You need one more line of code, assuming the controller is (or will be) on the same machine:

ovs-vsctl --db=tcp:127.0.0.1:9999 set-controller br0 tcp:127.0.0.1:6633

Getting OpenFlow going

We're on the home straight here. You can install the controller of your choosing, or you can install Ryu with the following instructions:

sudo apt-get install git python-setuptools

git clone http://github.com/osrg/ryu

cd ryu

sudo python setup.py install

You can then sit and watch as it downloads its dependencies from pypi. When it's done, fire up the controller with an app, and you're ready to go.

ryu-manager ryu/app/simple_switch.py

You can check the flow tables (in another screen) with the following command:

sudo ovs-ofctl dump-flows br0

Check out the manpages if you want to learn more:

http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities%2Fovs-ofctl.8

http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities%2Fovs-vsctl.8

What next?

You've got an OpenFlow testbed now, you can do what you want with it. Play with different controllers, or different versions of OpenFlow - it's all up to you.

DjangoFlow - Web UI for the POX OpenFlow controller

The Plan

OpenFlow is a simple concept - an open interface to switch and router hardware. Can we tie this into an open web framework to create a foundation for a really simple Web UI?

The tools

I've been learning how to use Django recently, and it seems like a perfect choice for this task. It's written in Python, so it integrates easily with the POX controller. I've also used virtualenv to make it more portable - this means the project is largely self-contained and can be copied to a new system by simply copying the folder.

Implementation

To start, I created a virtualenv called "djangoflow" on an Ubuntu machine and installed django in it. There are lots of ways to do this - do a google for "django setup virtualenv ubuntu" and you'll get tons of hits.

I made a project called mysite, and an app called flew (NZ english for "flow"), and left that bit for the time being.

The folder structure then looks something like this:

djangoflow

- bin

- include

- lib

- local

- mysite

--- flew

--- mysite

I then did a git clone of the latest build of POX from the NOXREPO github, into the djangoflow folder, and then copied everything from the base mysite folder into the pox folder. The folder structure then looks like:

djangoflow

- bin

- include

- lib

- local

- pox

--- .git

--- flew

--- mysite

--- pox

This is great - now back to Django.

The model that I used was really simple, here's what I've got so far:

from django.db import models

# Create your models here.

class Flow(models.Model):

internalip = models.CharField(max_length=200)

externalip = models.CharField(max_length=200)

idletime = models.IntegerField()

hardtime = models.IntegerField()

def __unicode__(self):

return "Internal: " + self.internalip + ", External: " + self.externalip

class User(models.Model):

name = models.CharField(max_length=200)

class Device(models.Model):

dpid = models.CharField(max_length=200)

We'll only use the Flow model today, the others are for expansion later. To make this work, we'll need to edit the settings in mysite.settings - set up the databases and installed_apps sections as follows:

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.

        'NAME': 'flew.db',                      # Or path to database file if using sqlite3.

        'USER': '',                      # Not used with sqlite3.

        'PASSWORD': '',                  # Not used with sqlite3.

        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.

        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.

    }

}

INSTALLED_APPS = (

    'django.contrib.auth',

    'django.contrib.contenttypes',

    'django.contrib.sessions',

    # 'django.contrib.sites',

    'django.contrib.messages',

    'django.contrib.staticfiles',

    # Uncomment the next line to enable the admin:

    'django.contrib.admin',

    # Uncomment the next line to enable admin documentation:

    'django.contrib.admindocs',

    'flew',

)

Awesome, now run syncdb (check against whatever tutorial you use to see how this is done) and populate the database. Now, fire up the shell and create your first flow:

python manage.py shell

from flew.models import Flow

f = Flow(internalip = "10.1.10.2", externalip = "10.1.20.2", idletime = 300, hardtime = 3600)

f.save()

If that all worked, then you'll have a new entry in your database. We are never going to access the database directly though - we can access all the Django goodness from POX.

Open up pox/forwarding/l2_learning.py and have a look through - if you've used this before, then good, if not, then see what it all does.

I've hijacked this just for this example, but it sets a starting point for any POX-Django integrated tools. Make sure your imports section looks like this:

# import django stuff

from django.core.management import setup_environ

from mysite import settings

setup_environ(settings)

from flew.models import Flow

from pox.core import core

import pox.openflow.libopenflow_01 as of

from pox.lib.revent import *

from pox.lib.util import dpidToStr

from pox.lib.util import str_to_bool

from pox.lib.packet import ethernet

from pox.lib.packet import ipv4

import time

Then go down to the __init__ function for LearningSwitch and add this to the end:

    # add new flow by default

    flow1 = Flow.objects.all()[0]

    msg = of.ofp_flow_mod()

    msg.match = of.ofp_match()

    msg.match.dl_type = ethernet.IP_TYPE

    msg.match.nw_src = str(flow1.internalip)

    msg.match.nw_dst = str(flow1.externalip)

    msg.idle_timeout = flow1.idletime

    msg.hard_timeout = flow1.hardtime

    msg.actions.append(of.ofp_action_output(port = 1))

    #msg.buffer_id = event.ofp.buffer_id # 6a

    self.connection.send(msg)

What does this do? It takes the first Flow out of our database, creates a flow to allow traffic from internalip, going to externalip, to go out port 1, which should be pointing at the outside world. This flow will stay in the switch for an hour, or 5 minutes without being triggered - whichever happens first.

Question time

Q: Is it really this easy?

A: Yes. Python is easy, Django is easy, Virtualenv is easy, POX is easy. It's just a little tricky making them all work together - that's what this guide is for.

Q: Why add flows this way when we already have a CLI?

A: Django has a clean admin interface that I haven't covered here (check the setup tutorial on the Django website) - you can set flows in there, and every time a switch connects, it will use those flows.

Q: Could I start using this right now?

A: Totally. If you want your switches to retrieve their configuration from the controller automatically when they start up, you can set a bunch of super specific flows here and roll it out now. If you want something a bit more clever, then you can jump into the Django and POX API and make it happen yourself.

Q: What next?

A: There are two extra models that we didn't use - one for user authentication, and one for managing devices. If your OpenFlow devices connect by SSL (which they should in the real world) then you can create models for them that hold particular flows - this can all be managed via a web interface. As for user authentication, there are millions of ways to do this - you can set privileges for who can set certain flows, make requests that admins can approve - the possibilities are endless!

Creating an Olive with JunOS 12.1 on VirtualBox

First, download jinstall-12.1R1.9-domestic-signed.tgz from the Juniper website.




You’ll need to unpack it and play around with it a little. Unpack the tgz

sam@laptop:~/Downloads$ mkdir jinst-signed
sam@laptop:~/Downloads$ cd jinst-signed/
sam@laptop:~/Downloads/jinst-signed$ tar -xzf ../jinstall-12.1R1.9-domestic-signed.tgz
sam@laptop:~/Downloads/jinst-signed$ ls -l
razem 437104
-rw-r--r-- 1 sam sam      7153 2012-03-24 19:57 certs.pem
-rw-r--r-- 1 sam sam        50 2012-03-25 03:31 +COMMENT
-rw-r--r-- 1 sam sam      1154 2012-03-25 03:31 +CONTENTS
-rw-r--r-- 1 sam sam       195 2012-03-25 03:31 +DESC
-rw-r--r-- 1 sam sam     87216 2012-03-25 03:31 +INSTALL
-rw-r--r-- 1 sam sam      6267 2012-03-25 02:25 issu-indb.tgz
-rw-r--r-- 1 sam sam 447458263 2012-03-25 02:40 jinstall-12.1R1.9-domestic.tgz
-rw-r--r-- 1 sam sam        33 2012-03-25 03:19 jinstall-12.1R1.9-domestic.tgz.md5
-rw-r--r-- 1 sam sam        41 2012-03-25 03:19 jinstall-12.1R1.9-domestic.tgz.sha1
-rw-r--r-- 1 sam sam       525 2012-03-25 03:31 jinstall-12.1R1.9-domestic.tgz.sig


There are tons of of files in this archive, and lots of guides say to try and play with it and pack it back up, but it’s actually a lot easier to just use the unsigned archive jinstall-12.1R1.9-domestic.tgz


If we unpack this, we get the following files:



-rw-r--r-- 1 sam sam  11673600 2012-03-25 02:38 bootstrap-install-12.1R1.9.tar
-rw-r--r-- 1 sam sam        39 2012-03-25 02:39 +COMMENT
-rw-r--r-- 1 sam sam       702 2012-03-25 02:39 +CONTENTS
-rw-r--r-- 1 sam sam    106121 2012-03-25 02:39 +DEINSTALL
-rw-r--r-- 1 sam sam       244 2012-03-25 02:39 +DESC
-rw-r--r-- 1 sam sam    107634 2012-04-02 19:57 +INSTALL
-rw-r--r-- 1 sam sam 440390207 2012-03-25 02:26 jbundle-12.1R1.9-domestic.tgz
-rw-r--r-- 1 sam sam      5933 2012-04-02 19:56 pkgtools.tgz
-rw-r--r-- 1 sam sam    106669 2012-04-02 19:57 +REQUIRE


We need to do edit the +INSTALL and +REQUIRE files here. Do a search for "re_name" and you'll find the following line:

check_arch_compatibility()

{

    re_name=`/sbin/sysctl -n hw.re.name 2>/dev/null`

    if [ -z "$re_name" ]; then

        Error "hw.re.name sysctl not supported."

    fi

This will make the install fail, so we need to replace it with the following:

check_arch_compatibility() { #re_name=`/sbin/sysctl -n hw.re.name 2>/dev/null` re_name="olive" if [ -z "$re_name" ]; then Error "hw.re.name sysctl not supported." fi


Once you've updated both of these files, unpack the pkgtools.tgz into a new directory


sam@laptop:~/Downloads/jinst-signed/jinst$ mkdir pkgtools
sam@laptop:~/Downloads/jinst-signed/jinst$ cd pkgtools/
sam@laptop:~/Downloads/jinst-signed/jinst/pkgtools$ tar -xzf ../pkgtools.tgz
sam@laptop:~/Downloads/jinst-signed/jinst/pkgtools$ ls -l
razem 8
drwxrwxr-x 2 sam sam 4096 2012-04-02 18:52 bin
drwxrwxr-x 2 sam sam 4096 2012-04-02 18:52 pkg
sam@laptop:~/Downloads/jinst-signed/jinst/pkgtools$ ls -l bin/
razem 4
-rwxr-xr-x 1 sam sam 2980 2012-04-02 19:55 checkpic


The checkpic file needs to be replaced with the "true" program from FreeBSD, so now is a good time to install it on our virtual machine. I've used FreeBSD 7.3, because FreeBSD 8 and higher use gpart instead of bsdlabel for changing the boot partition, but if you're keen to play with it, look in the +INSTALL file for the line "bsdlabel -B -b /boot/boot $labeldrive"


Anyway, get the i386 ISO for FreeBSD (the NZ mirror is on Citylink, so if your ISP zero-rates traffic through WIX/APE then it won't add to your data cap), and make up a virtual machine. I've used VirtualBox, but it should work in any virtual machine software

I used 512MB at first, but it wasn't enough for JunOS 12.1 - bumping it up to 640MB for the install made it work though. As you'll see in the later screenshots, I did this on my netbook with 990MB of RAM, so I couldn't bump it up too much further.

 An 8GB virtual hard drive gives you lots of room for new versions of software in your /var/tmp directory

This part is important - when booting, you can only see the output from JunOS through the console port. The easiest way to get this set up is to install socat and screen, and use the following command:

socat UNIX-CONNECT:/tmp/olivecom1 PTY,link=olivescreen

This will hang (only works once the virtual machine is running and the pipe is created) - so in a second terminal use this command:

screen olivescreen

 Boot up the installer, and go for a Standard install.

At the partition screen, push C to create a new slice, and allocate all the disk space to it. Then press Q and go to the next screen

 Choose a Standard boot loader

 The next screen lets you create partitions. Create the first partition with 1024M, as a file system partition, and mount point /

 Once this is done, create a 1024M swap partition, then a 16sM file system partition mounted at /dummy (this makes the labels line up so the installer doesn't fail), a 1024M file system partition at /config, and the last partition (with all the left over space) on /var

 Looking good so far? If the drive labels don't match up (i.e. no partition ad0s1f) then the JunOS install will fail

 Do a minimum install, install from CD/DVD, say no to all the questions, and you're done!

 Now you can exit the installer and reboot (don't forget to eject the virtual DVD!)

 When it reboots, use "dhclient em0" to get an address (probably 10.0.2.15), and then copy the file /usr/bin/true to your computer. The command "scp /usr/bin/true name@10.0.2.2:~/" should work for some computers, but under lubuntu it fails, so I had to use my GuruPlug instead ;)

Let's leave that virtual machine cooking for a bit. Now you have this "true" file, copy it in place of checkpic in your pkgtools/bin directory, then pack it all up

sam@laptop:~/Downloads/jinst-signed/jinst/pkgtools$ tar -xzf ../pkgtools.tgz *


Delete the pkgtools directory, and pack this all up into another .tgz

sam@laptop:~/Downloads/jinst-signed/jinst$ tar -xzf ../jinstall-12.1R1.9-domestic-olive.tgz *


Then you're good to go. Use scp to copy this onto your virtual machine, and use the pkg_add command to install JunOS 12.1

 When it's all done, rebooot, and watch it all in your terminal

And here is your JunOS 12.1 Olive. Haven't tested mine yet, but will end up using it as a testbed for big changes at work. Make sure you use the "cli" command to get the JunOS commandline, then run it like normal - use "request system power-off" to shut it down before turning it off. Check my older posts for how to network them, and remember - ethernet multicast fails on these, but you can make it work over GRE or IP-IP tunnels (older posts cover this too).

Openflow with NOX & Pronto/Pica8

We've got a Pronto 3290 at work, and with Josh Bailey's help I've been getting it talking Openflow to a NOX controller running pyswitch.

I figure the more I write about it, the more sense it'll make, so here's a summary of how far I've come:

• The pronto runs Open vSwitch, which lets you add your own flows manually - makes it easy to see what flows your controller has added too. They're supposedly going to add Openflow v1.2 support soon, which means IPv6!
• NOX doesn't find the Python bindings for OpenSSL on Ubuntu 11.10 (oneiric) in its current branch, but the destiny branch does - a bit of Git skill will sort this out for you
• Wireshark has an OpenFlow dissector which is part of the OpenFlow code, but it doesn't work with newer versions of Wireshark, you'll need this patch to make it build - confirmed working on Ubuntu 11.10
• Pyswitch (included as part of NOX) doesn't send back the right flows to the pronto - it sets the ethertype as 0x8100, so the flows look like this: idle_timeout=5,priority=65535,in_port=8,dl_vlan=1,dl_vlan_pcp=0,dl_src=00:XX:XX:XX:XX:XX,dl_dst=00:YY:YY:YY:YY:YY,dl_type=0x8100 actions=output:3 - this is where I'm going to start modding pyswitch

And this is where I am now. The plan for the next few weeks (which will probably change) is going to be something like this:

1. Make pyswitch send correct Openflow data
2. Mod pyswitch (or a demo router app) to do some basic routing and ACL
3. Hope that someone has written a BGP Openflow app so that I don't have to - otherwise, look at options for this

I'll be back with more details