Why would you do this?
If we want to make an OpenFlow router, we need to be able to communicate with other non-OpenFlow routers. Normally, you would assign an IP address to your router, turn on BGP/OSPF, and then configure these protocols to talk to other routers using this IP address. With OpenFlow, the controller has the brains, but no obvious way to talk to other network devices. If only we could pretend that the controller was in the router somehow...
Can't we just look at the OpenFlow messages?
Sure, and we looked at this last week, but it's clumsy and means we need to reinvent the wheel to make software routers talk to POX. RouteFlow abstracts this by loading software routers in virtual machines, last week's demonstration hardcodes everything into the controller, but tunnelling gives us a middle-of-the-road solution: no virtual machines needed, but we can still bind stuff to a network interface on the controller and let the linux network stack handle already-solved problems like TCP and the like.
Building a tunnel
Linux has a fantastic tool called TUN/TAP, which lets you create virtual network interfaces. One end talks to the Linux network stack and lets any application use it, and the other end talks to our program. In the spirit of keeping things modular, and minimising opportunities for me to write bad code, I've used the PyTap library to set this up. PyTap has a PIP package, which means we can easily add it to a virtualenv and continue to keep everything self-contained.
Protip: TUN interfaces take IP packets, TAP interfaces take Ethernet packets
If you haven't used virtualenvs, here's the basic idea:
virtualenv tundemo
cd tundemo
source bin/activate
pip install pytap
git clone http://github.com/noxrepo/pox
This will set you up with a virtualenv that has POX and PyTap ready to go. Despite being in a virtualenv, PyTap still needs root privileges, so you'll need to be root before source'ing into your virtualenv to make this work. If anyone can show me how to make this work without root privileges I'll be happy to hear (presumably some trickery with the /dev/net/tun device)
As with my other modules, I've hacked code into a copy of forwarding.l2_learning - this time I've renamed it to tundemo, and changed the name of the class all through the source.
Here are all my imports, add these at the top:
from pytun import TunTapDevice, IFF_TAP
from pox.lib.addresses import *
from pox.lib.packet import *
from threading import Thread
import subprocess
In the __init__() function, I've put the following code to make the TAP device:
# Our table
self.macToPort = {}
# TAP device
self.tap = TunTapDevice(flags=IFF_TAP)
self.tap.addr = '10.1.1.13'
self.tap.netmask = '255.255.255.0'
self.tap.mtu=1300
print "hwaddr for " + self.tap.name + ": " + str(EthAddr(self.tap.hwaddr))
# Bring tap interface up
subprocess.check_call("ifconfig " + self.tap.name + " up", shell=True)
PyTap chooses a random MAC address when it creates the interface, so printing it out lets us debug things a bit easier.
Tunneling fron TAP to switch
Once we have our TAP interface up, we need to handle packets that we receive on it. Let's set up a thread to handle this
# Create thread to read from tap and send to switch
self.th = Thread(target=handle_tap_in, args=(self))
self.th.daemon = True
self.th.start()
# Set max packet size to 1400 bytes
self.connection.send(of.ofp_set_config(miss_send_len=1400))
Our handler function is fairly straightforward
def handle_tap_in(switch):
while True:
packettap = switch.tap.read(switch.tap.mtu+24)
print "Packet read from tap"
e = ethernet()
e.parse(packettap[4:])
port = of.OFPP_ALL
if e.dst in switch.macToPort:
port = switch.macToPort[e.dst]
msg = of.ofp_packet_out()
msg.data = packettap[4:]
msg.actions.append(of.ofp_action_output(port =
port))
switch.connection.send(msg)
This will send all packets that come up on the tap0 interface to the switch, and either floods them or sends them on the right port, depending on what MAC addresses we've already learned.
Tunneling from switch to TAP
We already get sent packets from the switch by default, and these go to the _handle_PacketIn() function. We just need to get the raw data out and send this to the TAP interface
My switch always sends VLAN-tagged packets, so if yours doesn't then you'll want to change this a bit. Here is the SendToTap() function:
def SendToTap():
# remove vlan header and rebuild
print "Forwarding packet"
v = packet.next
i = v.next
eth = ethernet(src=packet.src, dst=packet.dst, type=v.eth_type)
print type(i)
eth.set_payload(i)
# first 4 bytes are 00 00 08 00 (null short, then IPv4 ethertype)
totap = struct.pack('!bbbb', 0, 0, 8, 0) + eth.pack()
#print totap.encode('hex')
self.tap.write(totap)
And we call this when a packet comes to us with a multicast MAC or our MAC:
if packet.dst == EthAddr(self.tap.hwaddr):
print "Packet for us!"
SendToTap()
return
if packet.dst.isMulticast():
SendToTap()
flood() # 3a
Now the tunnel is all good to go. Just make sure any devices plugged into the switch have an MTU of 1300, and you can talk to the controller, transfer files off with SCP (30 minutes to copy an Ubuntu ISO at around 4Mb/s)
A couple of hiccups
Packet sizes
My switch doesn't seem to handle having the packet-size value changed. POX by default tells the switch to send the first 128 bytes of packets, and while we can send messages to increase this, they're ignored. The work-around is to change DEFAULT_MISS_SEND_LEN to 1400 in pox/openflow/libopenflow01.py
Jitter
Latency varies from 1ms to 50ms, and TCP really, really doesn't like this. UDP routing protocols like OSPF shouldn't notice this, and even TCP-based routing protocols like BGP should be fine - but TCP gets really confused and this means you shouldn't expect any large data flows to work well with this.
MTU sizes
This stuff confuses me. I'm a network engineer, and I'm supposed to know this stuff, but I don't. When we read from the TAP device, we read the MTU + 24 bytes. There's 14 bytes for the Ethernet header, 4 bytes for the TAP header, and another 6 bytes in there for no obvious reason. 24 bytes just seems to work, and I have no idea why.
TAP device
Two things bug me about this - there doesn't seem to be a nice way to bring it up (apart from using ifconfig), and you need root to create it in the first place - I'd want to fix both of these for a nicer solution
Next steps
- TAP devices could be created for each physical port on an OpenFlow device, or as routed interfaces for each VLAN - limitless opportunities here
- BIRD or Quagga could bind to a TAP device, and the controller could turn routes into flows. BIRD has a python interface, but since both use standard routing protocols, you could easily sniff the traffic and build routing tables out of these. Sniffing BGP updates is still way easier than trying to build a Python TCP stack
- VRFs? Traffic injection? Just another example of how easy it is to grab POX and do novel things with inexpensive hardware
