Monday, February 18, 2013

SamShares - Parsing financial data out of annual report PDFs

What's up

I've been doing a lot of financial research, and a big chunk of that is looking through financial reports, manually copying the fields for assets, liabilities, equity, EBIT etc. It's boring as hell, and takes a long time. Why can't we automate this?

Parsing PDFs

I started by forking PyPDF2 to give me better access to the underlying objects. It's a fairly good start for working with PDFs, but just blurts out (some of) the text in a random order, which isn't what I want. This lead me down a bit of a rabbit hole and lead to me downloading a copy of the PDF 1.7 reference and browsing through this, sections 5.2 and 5.3 in particular

What's the plan?

  • Find the pages with assets/liabilites and income
  • Render them such that it's obvious where the columns and rows line up
  • Convert this to a spreadsheet
  • ???
  • PROFIT
For example, above is a screenshot from the annual report of New Zealand's largest NZX company, Fletcher Building. The PDF displays like lovely rows and columns, but can't be easily accessed in this way. If we can parse the PDF and render all the text in place, we can then make fairly accurate guesses at which rows and columns the values fall into.

Quick primer to text in PDF

Here are some of the operators you'll find for manipulating text in a PDF

BT, ET - Start and end a text object. This initialises the text matrix to the identify matrix - i.e. positioned at the top left of the document
Td, TD, T* - Operators to move the cursor to the next line
TM - Sets the text matrix. This is an affine transform, with 6 parameters - the first 4 matter for manipulating the text itself (scaling, warping, italics), and the last two essentially just set the start point for the text. This is enough for us to cheat and guess which way the text will go
Tc, Tw, Tf and lots more - Spacing and font settings

Tj, TJ - Display a text string - Tj does this simply, TJ has options after each character/substring for spacing information

Putting it all together

To parse a table out of a PDF, here's the rough idea:
  1. Locate all the strings on a page (BT/ET and TJ/Tj operators)
  2. Create a structure which ties the strings to locations (probably just Tm)
  3. Assign values row and column IDs
Once this is done, just check what is at the leftmost and topmost of each table, and use these as keys to the data. For the above image, the field "total assets" lined up with "June 2012" gives two results, so these just need to be referenced to the headers at the top, OR we can cheat and use the leftmost as this is generally the convention.

Next steps

Assuming I can make all this work, the data will then just be stored in a DB of some sort, keyed by year and company. Once this is automated enough to just pull PDFs out of NZX announcements, it'll be left in the background accumulating data, eventually building a corpus of financial data from NZX companies that can be used to make financial analysis much, much quicker and more versatile than it currently is.


Tuesday, February 12, 2013

OpenFlow 1.0 support on Juniper MX240 with JunOS 12.3

Juniper have added OpenFlow to JunOS 12.3

Do you have a spare MX240 lying around? Chuck a copy of JunOS 12.3 on it and you can get Openflow 1.0 up and running and have a play.

Details

  • Fairly full OF1.0 implementation. I don't have a spare MX240 to test, but it would appear that everything is handled in hardware (not sure how Junipers could do otherwise tbh)
  • Supports multiple VLANs - if these can be turned on and off from the controller then this would be awesome (let me know if you find this out)
  • Doesn't handle buffered packets - make sure your controller can handle OFPT_PACKET_IN messages that don't send a buffer ID (current version of POX doesn't do this?, but the betta branch does)
  • Doesn't handle TLS connectivity to the controller - not the end of the world, but I'm curious as to why this was done
  • Doesn't do anything related to STP... who cares?
  • Only supports MX240s...
This looks like a great start, well done Juniper! Here's my list of requests for the next iteration:
  • Support more than one device :) MX80's would be great, also looking to see what the EX series implementation looks like
  • Buffered packets! Everyone else does this, and it greatly speeds up the flows-per-second bottleneck between the switch and controller
That's pretty much all from me. OF1.1 support (or 1.3 as this is where everyone is going) would be awesome so we can drive MPLS, but other than that, this is fantastic news.

Update

It looks like it's not quite ready for RouteFlow - Joe Stringer pointed this out in the notes:

• If the controller pushes a flow with a set source MAC address action, the router cannot
   program the corresponding filter term. However, CLI show commands still display the
  flow with the associated action, and the device sends an OFPET_FLOW_MOD_FAILED
 error message with an OFPMFC_UNSUPPORTED code to the controller. [PR 838699]
• If the controller pushes a flow with a set destination MAC address action, the router
   cannot program the corresponding filter term. However, CLI show commands still
  display the flow with the associated action, and the device sends an
 OFPET_FLOW_MOD_FAILED error message with an OFPMFC_UNSUPPORTED code
to the controller. [PR 838709]
• If a flow contains a set IP source address action or a set IP destination address action,
   the device rejects the flow and sends an OFPET_FLOW_MOD_FAILED error m

In other words, no MAC/IP address rewrites = no routing :(

Disclaimer

I've been told that this info and the linked documents are public... If Juniper isn't happy with this, please get in touch and I'll fix it.

Friday, January 25, 2013

Thimble - Secure, high-speed connectivity with OpenFlow & Science DMZ

I've been busy

Last week I had the pleasure of spending a week in Honolulu at TIP2013, going to workshops, watching presentations, and socialising with some of the most talented network engineers in the world, and I felt incredibly fortunate to do so. I also had the opportunity to present some of my OpenFlow work with them, and was amazed at the feedback that I got from everybody. The recording of my presentation is now available online. I gave the same presentation this week at NZNOG, and I'll link to that when the on-demand video is available.

Thimble

For scientists to move big data in reasonable time frames, they need to have data transfer nodes outside of their campus firewalls, inside a Science DMZ. This is a proven way to optimise file transfers, but exposes your file transfer nodes to the whole internet. Thimble is a way of using OpenFlow to easily program ACLs into your edge switch, and I've covered it briefly here and here. The variation that I presented at TIP2013 and NZNOG positions the Science DMZ between an edge router and campus firewall, allowing a subset of routes to be sent to the OpenFlow switch. This means we only need to send experimental traffic to the Thimble, allowing implementers to test this without risking production traffic.

Clever stuff

You can federate a few Thimbles with a single controller, allowing a file transfer logged on the web application to trigger multiple switches around the world to be reconfigured. We're not just tied to a web interface either - there's nothing to stop us implementing uPnP at the edge and letting file transfer programs communicate with the network to arrange ACLs. The end goal is to take existing concepts and apply modern software design principles, allowing us to do things easily that used to be out of the question - network doesn't need to be this hard.

If you're interested in building a Thimble, I'd love to hear from you - leave a comment or email me and I'll be more than happy to hear your stories and help wherever I can.

RouteFlow in New Zealand

We've had some cool demos up and running this week - a distributed router was deployed at WIX and another data centre here in Wellington, and a RouteFlow deployment powered one of the internet feeds for the NZNOG conference this week. We had a screen up showing the flows present in the switch as people joined and left the wifi, and counters for data plane and control plane traffic. It was a nice visual demo - we could connect a cell phone, show the MAC and IP addresses in the new flow, and start a download to show the data plane traffic going up without changing the control plane traffic - just to prove that we don't need to send every packet via the controller!

What's coming in 2013

We're off to a really good start with OpenFlow this year, and I reckon the killer app is a year or two off at tops. Here's what I want to see happen:
  • Distributed routers - a mesh of 5-10, all controlled centrally
  • MPLS!!! OpenVSwitch now does OpenFlow1.2, and the Pica8 switches have implemented this, so there's no excuse for us not having an OpenFlow replacement for LDP/RSVP
  • Breaking OSI. Flows can match on any combination of ethernet, IP, and TCP/UDP, so I want to see more clever stuff happening with that. OpenFlow will let you do longest-prefix-matching on MAC addresses if you want, or route based on TCP/UDP ports, or some other weird combination. You'd want to find a way to do this that would be valuable, but people just need to jump in and see what new ideas they can come up with.
  • ???? - you tell me what you'd like to see in OpenFlow this year.

Friday, September 7, 2012

Tunneling traffic through your OpenFlow controller - Building a POX-based OpenFlow router

Why would you do this?

If we want to make an OpenFlow router, we need to be able to communicate with other non-OpenFlow routers. Normally, you would assign an IP address to your router, turn on BGP/OSPF, and then configure these protocols to talk to other routers using this IP address. With OpenFlow, the controller has the brains, but no obvious way to talk to other network devices. If only we could pretend that the controller was in the router somehow...

Can't we just look at the OpenFlow messages?

Sure, and we looked at this last week, but it's clumsy and means we need to reinvent the wheel to make software routers talk to POX. RouteFlow abstracts this by loading software routers in virtual machines, last week's demonstration hardcodes everything into the controller, but tunnelling gives us a middle-of-the-road solution: no virtual machines needed, but we can still bind stuff to a network interface on the controller and let the linux network stack handle already-solved problems like TCP and the like.

Building a tunnel

Linux has a fantastic tool called TUN/TAP, which lets you create virtual network interfaces. One end talks to the Linux network stack and lets any application use it, and the other end talks to our program. In the spirit of keeping things modular, and minimising opportunities for me to write bad code, I've used the PyTap library to set this up. PyTap has a PIP package, which means we can easily add it to a virtualenv and continue to keep everything self-contained.

Protip: TUN interfaces take IP packets, TAP interfaces take Ethernet packets

If you haven't used virtualenvs, here's the basic idea:

virtualenv tundemo
cd tundemo
source bin/activate
pip install pytap
git clone http://github.com/noxrepo/pox

This will set you up with a virtualenv that has POX and PyTap ready to go. Despite being in a virtualenv, PyTap still needs root privileges, so you'll need to be root before source'ing into your virtualenv to make this work. If anyone can show me how to make this work without root privileges I'll be happy to hear (presumably some trickery with the /dev/net/tun device)

As with my other modules, I've hacked code into a copy of forwarding.l2_learning - this time I've renamed it to tundemo, and changed the name of the class all through the source.

Here are all my imports, add these at the top:

from pytun import TunTapDevice, IFF_TAP
from pox.lib.addresses import *
from pox.lib.packet import *
from threading import Thread
import subprocess

In the __init__() function, I've put the following code to make the TAP device:

    # Our table
    self.macToPort = {}
    
    # TAP device
    self.tap = TunTapDevice(flags=IFF_TAP)
    self.tap.addr = '10.1.1.13'
    self.tap.netmask = '255.255.255.0'
    self.tap.mtu=1300
    print "hwaddr for " + self.tap.name + ": " + str(EthAddr(self.tap.hwaddr))
    
    # Bring tap interface up
    subprocess.check_call("ifconfig " + self.tap.name + " up", shell=True)

PyTap chooses a random MAC address when it creates the interface, so printing it out lets us debug things a bit easier.

Tunneling fron TAP to switch

Once we have our TAP interface up, we need to handle packets that we receive on it. Let's set up a thread to handle this

# Create thread to read from tap and send to switch
    self.th = Thread(target=handle_tap_in, args=(self))
    self.th.daemon = True
    self.th.start()

    # Set max packet size to 1400 bytes
    self.connection.send(of.ofp_set_config(miss_send_len=1400))

Our handler function is fairly straightforward

def handle_tap_in(switch):
  while True:
    packettap = switch.tap.read(switch.tap.mtu+24)
    print "Packet read from tap"
    e = ethernet()
    e.parse(packettap[4:])
    
    port = of.OFPP_ALL
    if e.dst in switch.macToPort:
        port = switch.macToPort[e.dst]
    
    msg = of.ofp_packet_out()
    msg.data = packettap[4:]
    msg.actions.append(of.ofp_action_output(port =
                                          port))
    switch.connection.send(msg)

This will send all packets that come up on the tap0 interface to the switch, and either floods them or sends them on the right port, depending on what MAC addresses we've already learned.

Tunneling from switch to TAP

We already get sent packets from the switch by default, and these go to the _handle_PacketIn() function. We just need to get the raw data out and send this to the TAP interface

My switch always sends VLAN-tagged packets, so if yours doesn't then you'll want to change this a bit. Here is the SendToTap() function:

def SendToTap():
     # remove vlan header and rebuild
      print "Forwarding packet"
      v = packet.next
      i = v.next
      eth = ethernet(src=packet.src, dst=packet.dst, type=v.eth_type)
      print type(i)
      eth.set_payload(i)
      # first 4 bytes are 00 00 08 00 (null short, then IPv4 ethertype)
      totap = struct.pack('!bbbb', 0, 0, 8, 0) + eth.pack()
      #print totap.encode('hex')
      self.tap.write(totap)

And we call this when a packet comes to us with a multicast MAC or our MAC:

if packet.dst == EthAddr(self.tap.hwaddr):
      print "Packet for us!"
      SendToTap()
      return

if packet.dst.isMulticast():
      SendToTap()
      flood() # 3a

Now the tunnel is all good to go. Just make sure any devices plugged into the switch have an MTU of 1300, and you can talk to the controller, transfer files off with SCP (30 minutes to copy an Ubuntu ISO at around 4Mb/s)

A couple of hiccups


Packet sizes

My switch doesn't seem to handle having the packet-size value changed. POX by default tells the switch to send the first 128 bytes of packets, and while we can send messages to increase this, they're ignored. The work-around is to change DEFAULT_MISS_SEND_LEN to 1400 in pox/openflow/libopenflow01.py

Jitter

Latency varies from 1ms to 50ms, and TCP really, really doesn't like this. UDP routing protocols like OSPF shouldn't notice this, and even TCP-based routing protocols like BGP should be fine - but TCP gets really confused and this means you shouldn't expect any large data flows to work well with this.

MTU sizes

This stuff confuses me. I'm a network engineer, and I'm supposed to know this stuff, but I don't. When we read from the TAP device, we read the MTU + 24 bytes. There's 14 bytes for the Ethernet header, 4 bytes for the TAP header, and another 6 bytes in there for no obvious reason. 24 bytes just seems to work, and I have no idea why.

TAP device

Two things bug me about this - there doesn't seem to be a nice way to bring it up (apart from using ifconfig), and you need root to create it in the first place - I'd want to fix both of these for a nicer solution

Next steps

  • TAP devices could be created for each physical port on an OpenFlow device, or as routed interfaces for each VLAN - limitless opportunities here
  • BIRD or Quagga could bind to a TAP device, and the controller could turn routes into flows. BIRD has a python interface, but since both use standard routing protocols, you could easily sniff the traffic and build routing tables out of these. Sniffing BGP updates is still way easier than trying to build a Python TCP stack
  • VRFs? Traffic injection? Just another example of how easy it is to grab POX and do novel things with inexpensive hardware

Friday, August 31, 2012

ARP and ping in POX - Building a POX-based OpenFlow router

What are we doing?

Today, we're going to look at how to handle ARP and ICMP ping messages in the OpenFlow controller POX. The results aren't amazing - latency is between 5 and 50 milliseconds (using pypy makes no difference) - but it's an important feature for any layer 3 device.

Why is it important?

If we want to make native router modules in OpenFlow, we need to be able to assign IP addresses to interfaces on our device. This means the router can talk IP to other devices on the network, a vital step towards building an OpenFlow router.

What about RouteFlow?

RouteFlow is a fully-functional OpenFlow router that you can use today, that translates the physical ports on your OpenFlow device to interfaces on a virtual machine. This virtual machine runs a software router daemon like Quagga or BIRD, meaning you can leverage a mature software router instead of making your own.

RouteFlow represents an important step in OpenFlow routing, but I think we can do better. RouteFlow polls the RIB on a virtual machine and translates that to OpenFlow, which means the router daemons don't know that they're talking to the controller.

If we build a clean interface, we can write POX modules for OSPF, IS-IS, BGP and the like, and let them talk directly to the controller.

How to make packets in POX

I love the packet library in POX, it's clean and easy to use. To make your own packets, just do what your network stack normally does - create the payload, wrap that in the layer below, then the layer below that, and once you're at Ethernet you're finished.

Step 1: ARP replies

I've started with the forwarding.l2_learning module from POX, and added some code to the _handle_PacketIn function, just under self.macToPort[packet.src] = event.port (so that MAC addresses are still stored for each new port).

match = of.ofp_match.from_packet(packet)
if ( match.dl_type == packet.ARP_TYPE and
match.nw_proto == arp.REQUEST and
match.nw_dst == IPAddr("10.1.1.253")):
  self.RespondToARP(packet, match, event)
  return

This checks for ARP requests for our hardcoded IP 10.1.1.253, and responds. The code to respond is as follows:

  def RespondToARP(self, packet, match, event):
    # reply to ARP request
    r = arp()
    r.opcode = arp.REPLY
    r.hwdst = match.dl_src
    r.protosrc = IPAddr("10.1.1.253")
    r.protodst = match.nw_src
    r.hwsrc = EthAddr("00:12:34:56:78:90")
    e = ethernet(type=packet.ARP_TYPE, src=r.hwsrc, dst=r.hwdst)
    e.set_payload(r)
    log.debug("%i %i answering ARP for %s" %
     ( event.dpid, event.port,
       str(r.protosrc)))
    msg = of.ofp_packet_out()
    msg.data = e.pack()
    msg.actions.append(of.ofp_action_output(port =
                                          of.OFPP_IN_PORT))
    msg.in_port = event.port
    event.connection.send(msg)

We build an ARP packet by calling the arp() function from pox.lib.packet, and it initialises the packet as follows:

def __init__(self, raw=None, prev=None, **kw):
        packet_base.__init__(self)

        self.prev = prev

        self.hwtype     = arp.HW_TYPE_ETHERNET
        self.prototype  = arp.PROTO_TYPE_IP
        self.hwsrc      = ETHER_ANY
        self.hwdst      = ETHER_ANY
        self.hwlen      = 6
        self.opcode     = 0
        self.protolen   = 4
        self.protosrc   = IP_ANY 
        self.protodst   = IP_ANY
        self.next       = b''

        if raw is not None:
            self.parse(raw)

        self._init(kw)

We just need to set the OPCODE, HWSRC, HWDST, PROTOSRC and PROTODST fields of this. I've done this in the body of the code, but we can simplify it by passing extra arguments as follows:

r = arp( opcode=arp.REPLY, 
         hwsrc=EthAddr("00:12:34:56:78:90"),
         hwdst=match.dl_src,
         protosrc = IPAddr("10.1.1.253"),
         protodst = match.nw_src)

Once we've created the ARP packet, we need to create an Ethernet packet to put this into. This isn't perfect (we should check for VLAN tags and add them, or steal the body of the original packet and modify that), but it works if we're just dealing with a straight Ethernet network.

e = ethernet(type=packet.ARP_TYPE, src=r.hwsrc, dst=r.hwdst)
e.set_payload(r)

Then we send this off to the controller, which sends it out the port it came through. Now we have an IP address that people can find, let's make it respond to something.

Step 2: Ping replies

If we can reply to ARP requests, we can reply to pings. This has a few more layers, but that just makes the code a little longer, not any more complicated.

The ARP reply is easy - we make an ARP packet, then put that in an Ethernet packet. For ping reply, this is what we do:
  1. Get the payload from the echo request (ping)
  2. Create an echo reply packet, insert the old payload
  3. Create an ICMP packet, insert the echo reply
  4. Create an IPv4 packet, insert the ICMP
  5. Create an Ethernet packet, insert the IPv4
Here's what the code looks like:

  def RespondToPing(self, ping, match, event):
    p = ping
    # we know this is an ICMP Echo packet, so loop through
    # maybe this needs a try... except?
    while not isinstance(p, echo):
      p = p.next
    
    r = echo(id=p.id, seq=p.seq)
    r.set_payload(p.next)
    i = icmp(type=0, code=0)
    i.set_payload(r)
    ip = ipv4(protocol=ipv4.ICMP_PROTOCOL,
              srcip=IPAddr("10.1.1.253"),
              dstip=match.nw_src)
    ip.set_payload(i)
    e = ethernet(type=ping.IP_TYPE,
                 src=match.dl_dst,
                 dst=match.dl_src)
    e.set_payload(ip)
    log.debug("%i %i answering PING for %s" % (
              event.dpid, event.port,
              str(match.nw_src)))
    msg = of.ofp_packet_out()
    msg.data = e.pack()
    msg.actions.append(of.ofp_action_output(port =
                                          of.OFPP_IN_PORT))
    msg.in_port = event.port
    event.connection.send(msg)

Simple, just slightly longer than the ARP code.

Pictures

Here's a look at the controller output, and the view from Wireshark.

The OpenFlow dissector for Wireshark is part of the OpenFlow reference switch. It's a few years old, and uses an obselete API call - I can put up a patch if anyone gets stuck.

Next steps

  • ARP tables - if we're going to route traffic, we need to find the MAC addresses of destination IPs so that we send traffic to them
  • Routing protocol - RIP and OSPF will be fairly easy, BGP will be a bit harder due to relying on TCP. These can all be added to POX as modules
  • TUN/TAP support - we can create TUN/TAP interfaces and let the linux TCP stack do the hard work for us. This means a BGP module would create the TUN/TAP interface and handle OpenFlow encapsulation/decapsulation, but could offload the TCP to the Linux stack.

Tuesday, August 21, 2012

DjangoFlow part two: Quick and simple UI

In the last episode...

My previous blog post showed how to integrate POX with Django, and didn't have too much colour. It took a bit of playing to get it to integrate for the first time, but now it's done, it is incredibly easy to do cool stuff with this software combo, and make new apps for easy OpenFlow access.

Part 2: An actual User Interface

I put in about 10 minutes extra just so I could give you all some lovely pictures, and here's what it all looks like:

This is all Django - I've just turned on the admin interface. Here we can add and remove flows from the database.

Here's a list of the two flows that I put in for testing

And here's the output from my OpenFlow switch.


It doesn't update in real time, but you can manipulate the database via the interweb and push those flows directly out to your switch. Making this update in real time is going to be another half-hour's work, tops.

What's new?

I changed the model a bit - now we can choose ports as well. Remember to delete your old database before syncing again or else it'll get upset

class Flow(models.Model):
internalip = models.CharField(max_length=200)
externalip = models.CharField(max_length=200)
internalport = models.IntegerField()
externalport = models.IntegerField()
idletime = models.IntegerField()
hardtime = models.IntegerField()
def __unicode__(self):
return "Internal: IP=" + self.internalip + " port=" + str(self.internalport) +", External: IP=" + self.externalip + " port=" + str(self.externalport)

The code in l2_learning.py got a bit of a birthday too:

class LearningSwitch (EventMixin):
  def __init__ (self, connection, transparent):
    # Switch we'll be adding L2 learning switch capabilities to
    self.connection = connection
    self.transparent = transparent

    # Our table
    self.macToPort = {}

    # We want to hear PacketIn messages, so we listen
    self.listenTo(connection)

    #log.debug("Initializing LearningSwitch, transparent=%s",
    #          str(self.transparent))
    
    # add new flows by default
    for flow in Flow.objects.all():
self.AddFlowFromModel(flow)
    
  
  def AddFlowFromModel(self, flow):
    # add outgoing flow
    msg = of.ofp_flow_mod()
    msg.match = of.ofp_match()
    msg.match.dl_type = ethernet.IP_TYPE
    msg.match.nw_src = str(flow.internalip)
    msg.match.nw_dst = str(flow.externalip)
    msg.match.in_port = flow.internalport
    msg.idle_timeout = flow.idletime
    msg.hard_timeout = flow.hardtime
    msg.actions.append(of.ofp_action_output(port = flow.externalport))
    self.connection.send(msg)
    
    # add incoming flow
    msg = of.ofp_flow_mod()
    msg.match = of.ofp_match()
    msg.match.dl_type = ethernet.IP_TYPE
    msg.match.nw_src = str(flow.externalip)
    msg.match.nw_dst = str(flow.internalip)
    msg.match.in_port = flow.externalport
    msg.idle_timeout = flow.idletime
    msg.hard_timeout = flow.hardtime
    msg.actions.append(of.ofp_action_output(port = flow.internalport))
    self.connection.send(msg)

After this, it was just a case of enabling the admin interface as per https://docs.djangoproject.com/en/dev/intro/tutorial02/ - this is our admin.py:

from django.contrib import admin
from flew.models import Flow

admin.site.register(Flow)

And for the sake of completeness, here's how to start them - the web interface is

python manage.py runserver

and the controller is

python pox.py forwarding.l2_learning

What next?

I don't know... I thought this would be much harder? Authentication will be fun, some way to dynamically check the database and update flows in real time (and remove them maybe) - this is left as an exercise for the reader though

DjangoFlow - Web UI for the POX OpenFlow controller

The Plan

OpenFlow is a simple concept - an open interface to switch and router hardware. Can we tie this into an open web framework to create a foundation for a really simple Web UI?

The tools

I've been learning how to use Django recently, and it seems like a perfect choice for this task. It's written in Python, so it integrates easily with the POX controller. I've also used virtualenv to make it more portable - this means the project is largely self-contained and can be copied to a new system by simply copying the folder.

Implementation

To start, I created a virtualenv called "djangoflow" on an Ubuntu machine and installed django in it. There are lots of ways to do this - do a google for "django setup virtualenv ubuntu" and you'll get tons of hits.

I made a project called mysite, and an app called flew (NZ english for "flow"), and left that bit for the time being.

The folder structure then looks something like this:

djangoflow
- bin
- include
- lib
- local
- mysite
--- flew
--- mysite

I then did a git clone of the latest build of POX from the NOXREPO github, into the djangoflow folder, and then copied everything from the base mysite folder into the pox folder. The folder structure then looks like:

djangoflow
- bin
- include
- lib
- local
- pox
--- .git
--- flew
--- mysite
--- pox

This is great - now back to Django.

The model that I used was really simple, here's what I've got so far:

from django.db import models

# Create your models here.

class Flow(models.Model):
internalip = models.CharField(max_length=200)
externalip = models.CharField(max_length=200)
idletime = models.IntegerField()
hardtime = models.IntegerField()
def __unicode__(self):
return "Internal: " + self.internalip + ", External: " + self.externalip

class User(models.Model):
name = models.CharField(max_length=200)

class Device(models.Model):
dpid = models.CharField(max_length=200)

We'll only use the Flow model today, the others are for expansion later. To make this work, we'll need to edit the settings in mysite.settings - set up the databases and installed_apps sections as follows:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'flew.db',                      # Or path to database file if using sqlite3.
        'USER': '',                      # Not used with sqlite3.
        'PASSWORD': '',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
    }
}

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    # 'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    'django.contrib.admindocs',
    'flew',
)

Awesome, now run syncdb (check against whatever tutorial you use to see how this is done) and populate the database. Now, fire up the shell and create your first flow:

python manage.py shell
from flew.models import Flow
f = Flow(internalip = "10.1.10.2", externalip = "10.1.20.2", idletime = 300, hardtime = 3600)
f.save()

If that all worked, then you'll have a new entry in your database. We are never going to access the database directly though - we can access all the Django goodness from POX.

Open up pox/forwarding/l2_learning.py and have a look through - if you've used this before, then good, if not, then see what it all does.

I've hijacked this just for this example, but it sets a starting point for any POX-Django integrated tools. Make sure your imports section looks like this:

# import django stuff
from django.core.management import setup_environ
from mysite import settings
setup_environ(settings)
from flew.models import Flow

from pox.core import core
import pox.openflow.libopenflow_01 as of
from pox.lib.revent import *
from pox.lib.util import dpidToStr
from pox.lib.util import str_to_bool
from pox.lib.packet import ethernet
from pox.lib.packet import ipv4
import time

Then go down to the __init__ function for LearningSwitch and add this to the end:

    # add new flow by default
    flow1 = Flow.objects.all()[0]
    msg = of.ofp_flow_mod()
    msg.match = of.ofp_match()
    msg.match.dl_type = ethernet.IP_TYPE
    msg.match.nw_src = str(flow1.internalip)
    msg.match.nw_dst = str(flow1.externalip)
    msg.idle_timeout = flow1.idletime
    msg.hard_timeout = flow1.hardtime
    msg.actions.append(of.ofp_action_output(port = 1))
    #msg.buffer_id = event.ofp.buffer_id # 6a
    self.connection.send(msg)

What does this do? It takes the first Flow out of our database, creates a flow to allow traffic from internalip, going to externalip, to go out port 1, which should be pointing at the outside world. This flow will stay in the switch for an hour, or 5 minutes without being triggered - whichever happens first.

Question time

Q: Is it really this easy?
A: Yes. Python is easy, Django is easy, Virtualenv is easy, POX is easy. It's just a little tricky making them all work together - that's what this guide is for.

Q: Why add flows this way when we already have a CLI?
A: Django has a clean admin interface that I haven't covered here (check the setup tutorial on the Django website) - you can set flows in there, and every time a switch connects, it will use those flows.

Q: Could I start using this right now?
A: Totally. If you want your switches to retrieve their configuration from the controller automatically when they start up, you can set a bunch of super specific flows here and roll it out now. If you want something a bit more clever, then you can jump into the Django and POX API and make it happen yourself.

Q: What next?
A: There are two extra models that we didn't use - one for user authentication, and one for managing devices. If your OpenFlow devices connect by SSL (which they should in the real world) then you can create models for them that hold particular flows - this can all be managed via a web interface. As for user authentication, there are millions of ways to do this - you can set privileges for who can set certain flows, make requests that admins can approve - the possibilities are endless!