Bootstrapping LXCs behind an APT proxy

Had some fun getting lxc-create to do its stuff this morning, the /etc/default/lxc config file has half a hint but not a whole one:

# MIRROR to be used by ubuntu template at container creation:
# Leaving it undefined is fine
#MIRROR="http://archive.ubuntu.com/ubuntu"
# or 
#MIRROR="http://<host-ip-addr>:3142/archive.ubuntu.com/ubuntu"

# LXC_AUTO - whether or not to start containers symlinked under
# /etc/lxc/auto
LXC_AUTO="true"

# Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your
# containers.  Set to "false" if you'll use virbr0 or another existing
# bridge, or mavlan to your host's NIC.
USE_LXC_BRIDGE="true"

# If you change the LXC_BRIDGE to something other than lxcbr0, then
# you will also need to update your /etc/lxc/lxc.conf as well as the
# configuration (/var/lib/lxc/<container>/config) for any containers
# already created using the default config to reflect the new bridge
# name.
# If you have the dnsmasq daemon installed, you'll also have to update
# /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon.
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"

LXC_SHUTDOWN_TIMEOUT=120

The line that it's missing is "you'll also need to set the SECURITY_MIRROR field or else it dies halfway" - so add the following to your /etc/default/lxc and you're good to go:

MIRROR="http://IP-OF-PROXY:3142/archive.ubuntu.com/ubuntu"
SECURITY_MIRROR="http://IP-OF-PROXY:3142/security.ubuntu.com/ubuntu"

How to install Ubuntu Server 14.04 on a Mac

We have an old Mac tower in the office that I've been using as a headless server, and last time around I couldn't even get 12.04 server to install - I needed to install 12.04 desktop.

This time around, the Ubuntu server 14.04 install works fine... except it hangs at boot

Looking at the output, it has an issue with plymouth-upstart-bridge, then tries to mount swap, and then it does nothing...

I came across this post that had the answer - booting will work if you boot in recovery mode (grub -> advanced options -> ubuntu recovery) and then resume the boot. The permanent fix is to add "nomodeset" to grub.

Instructions from the beginning

1. Copy ubuntu 14.04 amd64 server iso onto a USB drive (sudo dd if=ubuntu-14.04-server-amd64.iso of=/dev/sdX bs=1M) (*don't blame me* if you accidentally override your main harddrive)
2. Plug usb key into server, boot up (if it doesn't boot off USB you may need to hold down ALT to get the magic screen and then choose EFI USB boot)
3. Run install like you normally would
4. Find that it hangs on boot
5. Boot again but in recovery mode (at GRUB choose "advanced options" and boot into recovery mode)
6. Log in
7. sudo vim /etc/default/grub
8. Edit the line GRUB_CMDLINE_LINUX_DEFAULT="" to say GRUB_CMDLINE_LINUX_DEFAULT="nomodeset"
9. Save file
10. sudo update-grub
11. Reboot
12. *magic it works yay*

Private servers on EC2

Build all the servers (or, how to setup a secure-ish private minecraft server on EC2)

At the time of writing, Amazon EC2 gives you a free virtual machine for the first 12 months of a new account. It's not huge, but it has a CPU and 1GB of RAM, and that gives you a few possibilities. A VM in the cloud is cool, but there's an obvious downside to not having it in your own house/office/parents' basement

Open servers on the internet

If you want to do anything cool with your VM, chance are you'll end up opening ports, and introducing security holes. You may say that you're not worried about this, that your server isn't super important, but if someone can own your VM, then they're one step closer to breaking into your own computer, and that's a bad thing.

In short, you need a way to lock things down, and that's what we're going to do here. But first, let's spin up an EC2 VM

Ubuntu VMs on EC2

Go to the Compute menu, and click on Launch Instance

Check the Free tier only box, and click on the Ubuntu 64 bit VM

You'll see the green highlighted free tier eligible text next to the smallest VM, select that and click Next: Configure Instance Details

Leave as is and click next

Nothing to see here, click next

You can add tags if you want, but we don't need them for anything in particular right now

You'll want to make some adjustments to the security groups - click Add Rule and make a custom TCP rule opening up port 1723 - this will allow PPTP VPN connections.

Your config should look like mine here

And now you're good to go, click Launch and head to the final step

Make a name for your keypair and download it

Annnnnnnd you're all good to go!

To log in (instructions for OSX/Linux, windows users can figure out how to do this in PuTTY themselves), use the -i switch to load the private key (make sure you chmod it to 600), and log in with the user ubuntu. On the "compute" screen you'll be able to see the IP for your server, log in as follows:

ssh -i keyfile.pem ubuntu@IP

And you're in! Feel free to have a play with this as is, or move onto the next step - setting up containers.

Linux containers (LXC)

Making them work

LXCs are the secret sauce that'll let you make faux-VMs to put things in. Install LXC with the following:

sudo apt-get update
sudo apt-get install lxc

This will load up everything you need to build an LXC. To build your LXC, use the following. We'll call our container "minecraft", for no particular reason:

sudo lxc-create -t ubuntu -n minecraft

This will take a long time, so get yourself a coffee or three. This only happens the first time, as it sets up the template - subsequent LXCs will take seconds when you make them.

Once this finishes, start it up and log in (CTRL+A followed by the Q key will get you out of the console)

sudo lxc-start -d -n minecraft
sudo lxc-console -n minecraft

Here's your container! If for some reason, you were wanting to install minecraft, you would install the JDK and download the minecraft JAR file from the appropriate website. The below lines will install version 1.7.10, you'll probably want whatever the latest version is.

sudo apt-get install default-jdk
wget https://s3.amazonaws.com/Minecraft.Download/versions/1.7.10/minecraft_server.1.7.10.jar

Making things a little nicer

Your LXCs will use DHCP when you spin them up, which is great for getting things going at first, but we want a little bit more control. Go back to the host machine (CTRL+A, then Q, if you aren't in it already) and open up /etc/default/lxc-net.conf and change the DHCP settings so we can open up some space - change the settings in there to the following:

LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.200,10.0.3.240"
LXC_DHCP_MAX="40"

We'll also want to give our LXC a static IP, so get back into your LXC

sudo lxc-console -n minecraft

and edit /etc/network/interfaces (if you're still in the host you can edit this at /var/lib/lxc/minecraft/rootfs/etc/network/interfaces if you like). Remove the line

iface eth0 inet dhcp

and replace with the following

iface eth0 inet static
        address 10.0.3.250
        netmask 255.255.255.0
        gateway 10.0.3.1

Save and close, and power off your container for now

sudo poweroff

This is all we need from the LXC for now, let's set up the VPN.

PPTP VPNs

We're using PPTP today because you can get it working in minutes, and it has native clients in Ubuntu and OSX. As per the instructions on the Ubuntu website, we'll install as follows:

sudo apt-get install pptpd

Then edit /etc/pptpd.conf, scroll to the bottom, and set up addresses for clients. I've done this as follows:

localip 10.0.3.1

remoteip 10.0.3.2-22

Open /etc/ppp/pptpd-options, and add DNS

ms-dns 8.8.8.8

ms-dns 8.8.4.4

Then edit /etc/ppp/chap-secrets and set up accounts. This shouldn't need to be said, but just in case security isn't clear, DO NOT USE THESE LOGINS AS EVERYONE ON THE INTERNET KNOWS WHAT THEY ARE NOW

user1 * password1 *

user2 * password2 *

Now restart the PPTP server

/etc/init.d/pptpd restart

And give it a shot!

Putting it all together

Weird stuff can happen if you haven't shut everything down, so the safest way to apply your LXC settings is reboot the whole VM

reboot

Now log back in, start up your LXC, and then try to log in from the outside. Connect to your server with your PPTP account, then once the VPN is up, try to ping your LXC on 10.0.3.250. If you've installed a minecraft server on it, log into the LXC and fire it up with the following:

java -Xms512M -Xmx1G -jar minecraft_server.1.7.10.jar nogui

Edit the newly-generated eula.txt to say true, start up again, and test it out... remember to use the LXC's internal IP (10.0.3.250) when connecting though!

Troubleshooting

• Try turning it off and on again. I am not joking.
• Make sure you're fully upgraded (apt-get upgrade/apt-get dist-upgrade) - LXC has dependencies on newer versions of certain code, but doesn't automatically upgrade them for you.

What have we done?!

What's the point in putting a VM inside a VM in the cloud? The reason you might consider this, is because while cloud resources are cheap and plentiful, they're also hard to configure properly to allow full access to you, but completely lock out everyone else. I can't make any guarantees on the security of this - LXC is still new, and there are further things we can do to lock down the internal bridge - but I hope this has been a useful start to get people thinking about bridging out into private servers in public clouds such as EC2.

10G > 1G

10Gb/s ain't what it used to be

It was only a few years ago that 10Gb/s kit cost 10's of thousands of dollars and needed massive XenPaks to plug in as optics. It's now 2014, 10Gb/s SFPs cost about $200 each, and the closest I get to a XenPak is the broken one I use as a bottle opener.

Because it's so cheap, it's a no-brainer to put 10Gb/s NICs in your servers, but there's no guarantee that your network can support 10Gb/s the whole way through. You might think that a 1Gb/s bottleneck in your network isn't a big deal, and that TCP can fumble around and find the top speed for your connection, but you might be disappointed to hear that it's not that easy.

TCP is dumb

TCP doesn't have any parameters, internal or external, for how fast it's sending data. It has a window of how much unaccounted data has been sent, and this window moves along as acknowledgements are received. The size of this window sets an upper bound on the average speed (based on latency and packet loss, feel free to explore this), but not on the maximum speed. This becomes a problem as bandwidth and latency both increase.

Long fat networks

The TCP window keeps track of every byte "in flight" - in other words, all data that has been sent and not acknowledged. It can't send any more data until the first lot of data has been acknowledged, and it needs a buffer (window) to track this. The smallest this buffer can be is latency x bandwidth, and this number can get very big very quickly. If you're trying to send at 1Gb/s to a destination 160ms away, you need a window of 20MB - if you want to do this at 10Gb/s, you need a window of 200MB! Compared to the 128GB flash drives that clip onto your keyring, this doesn't seem like a huge amount, but to the switches and routers in your network, this is a lot to soak up if your traffic has to go across a slow part of the network

How 10 goes into 1

If your sender and receiver have 10Gb/s connections, but the network has a 1Gb/s segment in the middle, you can run into interesting problems. With 160ms of latency in the way, your sender dumps 20MB onto the network at 10Gb/s - in the time it takes to arrive at the start of the 1Gb/s segment, the 1Gb/s segment can only send 2MB - leaving 18MB to be dealt with. If you have big enough buffers, then this will eke out into the network at 1Gb/s and everything will be fine!

However, 20MB is a big buffer - it holds 160ms of data. We've all seen buffer bloat (when buffers fill up and stay full and add extra latency to the network), and hear about it being a bad thing, but this is an instance where buffers are *very* important. If you have no buffers, your TCP stream starts up and immediately drops 90% of its packets, and things go very bad, very fast.

Labbing this up

You can simulate this yourself between two Linux machines with tc and iperf. First, plug them into each other at 10Gb/s, make sure they're tuned (net.ipv4.tcp_rmem/wmem need to have the last parameter set to about 64MB), and test between them. Assuming sub-millisecond latency, you should see very close to 10Gb/s of TCP throughput (if not, the servers are mistuned, or underpowered).

box1: iperf -i1 -s

box2: iperf -i1 -c box1 -t300

Looking good? If not, you're out of luck in this instance - TCP tuning is outside the scope of this post, go and ask ESnet what to do.

Assuming this is all working, we'll add some latency as an egress filter on box1, and see what happens

sudo tc qdisc add dev eth0 netem delay 50ms limit 10000

Try the iperf again - is it still working well? If you're not averaging 7-8Gb/s then you might want to do some tuning, and come back when it's looking better.

Now we've got a known-good at 50ms, let's try simulating a 1Gb/s bottleneck in the network. Apply an ingress policer to box2 as follows:

sudo tc qdisc add dev eth0 handle ffff: ingress

sudo tc filter add dev eth0 parent ffff: protocol ip prio 1 u32 match ip src 0.0.0.0/0 police rate 1000mbit burst 100k mtu 100k drop flowid :1

Try your iperf again - how does it look? When I tested this, I couldn't get more than 10Mb/s - something is seriously wrong! Let's try and send some UDP through to see what's happening

box1: iperf -i1 -s -u -l45k

box2: iperf -i1 -u -l45k -c box1 -t300 -b800m

Odd... we can send 800Mb/s of UDP with little or no packet loss, but can't get more than 20Mb/s of TCP?!

The fix for this is adding a shaper to make sure nothing gets dropped by the policer. We can add this on box1 in this instance as follows:

sudo tc qdisc del dev eth0 root

sudo tc qdisc add dev eth0 root handle 1: tbf rate 900mbit burst 12000 limit 500m mtu 100000

sudo tc qdisc add dev eth0 parent 1: netem delay 50ms limit 10000

You'll notice we deleted and then re-added the latency - this is just a limitation of how we chain these qdiscs together. But give it a shot - try an iperf between the two boxes with TCP, and magic - you can get 850Mb/s now!

A sustainable fix

We're not going to add shapers everywhere just because we know that some parts of our network have bad bottlenecks. It's okay though - smart people are working on a fix. By adding a knob to TCP where we explicitly say how fast we want it to go, we can make TCP pace itself, instead of emptying out its window onto the network, and then getting upset when it doesn't get magically taken care of. This is still experimental, but I'm keen to hear if anyone has had any luck with it - this is a very important step forward for TCP, and will become gradually more important as our networks get longer and fatter.

Building a better internet exchange with OpenFlow

Internet exchanges

The internet is a collection of cables and routers - cables join the routers together, and routers decide where your data should go. Internet exchanges perform a crucial job in the middle of this - they provide a way for different (often competing) providers to exchange internet traffic. An internet exchange is typically implemented using some number of switches that are federated together somehow so that every port can exchange traffic with every other port as fast as they need to.

Problems with internet exchanges

Many internet exchanges are implemented as a layer-2 broadcast domain, and often include a route server to make it easier for participants to exchange routes with everyone over a single peering. Layer-2 broadcast domains need to be well policed though, because people can easily mess with other peoples' traffic, either accidentally, or maliciously.

Things that you need to watch out for on a layer-2 broadcast domain:

• Broadcast traffic - ARP requests are necessary, others probably aren't
• MAC spoofing
• ARP spoofing/proxy ARP
• STP & other lovely protocols

Things that you need at an IPv4/IPv6 peering exchange

• Unicast IPv4 and IPv6 traffic
• ARP requests (not necessarily broadcast)
• Legitimate ARP replies

Where's the button to turn this all on?

You'll often build an internet exchange with cheap and big switches, so you're not likely to have a huge amount of functionality to do clever stuff like this - and I've never seen a switch config before that lets me check the validity of ARP replies. To do this in software, however, is pretty easy.

Software Peering Exchange

Here's something I put together last night: https://github.com/samrussell/ryu/blob/master/ryu/app/spe.py

It's an extension of my OF1.2 learning switch, with an extra table to handle ARP entries. Here's how the tables are laid out:

The controller takes some simple JSON config of which IP is expected on each port, and loads these into the switch. Table 0 forwards all ARP packets to table 1, and table 1 checks the packets - if they're ARP requests, then it forwards them out the appropriate port (so ARP requests are effectively unicast across the internet exchange), and for ARP replies, it only allows each port to reply for its own IP address. Any packets that don't get forwarded out a port get sent to the controller (including valid ARP replies before we learn any MAC addresses, but the controller deals with this smartly & learns the MAC address properly).

This is combined with the learning switch that I posted earlier in the week, with one alteration - our source-MAC matches also have to match the ethertype, and this can be constrained in the controller to limit it to IPv4 or IPV6 (or either or - this could be done on a per port basis if you wanted to). Packets with known source-MACs are send to table 2, and then forwarded out the right port if we've learned their MAC - otherwise, it goes to the controller for processing.

What else can we do?

We can lock down the controller a bit to only allow IPv4, IPv6 and ARP - this would stop CDP, STP and the like leaking into the exchange - only valid traffic, and no spoofing.

We could go a step further though - we could build in ACLs on each port based on the routes that they announce for basic RPF, so that a participant can only route traffic from an IP range if they advertise that route to the exchange already - this would prevent asymmetric routing.

So go, have a play - see if you can break it or find something else to add!

OpenFlow 1.2 switch app with multiple tables

Another OpenFlow switch app?

I'm sorry - there's quite a few of these, but this one is cool. I've been part of a couple of OpenFlow bootcamps in the last year and a bit, and an example that I've used is the pyswitch app that came stock with NOX. It's only a few lines of functional code, but it's a nice example to show what OpenFlow controller code looks like, and how you can easily build a learning switch from scratch. We'd go through a couple of scenarios on the whiteboard, but then we'd find a problem that OpenFlow 1.0 couldn't solve.

Why we need multiple flow tables

Let's say we have two nodes connected to the OpenFlow switch. Node 1 sends a packet to node 2, and the switch doesn't know this MAC address, so it passes the packet up to the controller. The controller learns the MAC address of node 1 (where the packet came from), and assigns it to port 1. It then pushes a flow to the controller to make sure that traffic for node 1 goes out the right port, and then sends the packet on its way.

What happens when node 2 replies?

The switch gets the packet for node 1, has a flow for it, and forwards it. There's a problem here though - because the packet never went to the controller, the switch never learns the MAC address of node 2 - so all traffic to node 2 will go via the controller and clog things up.

The OpenFlow learning switch apps that I've seen deal with this by storing pairs - they match on a destination MAC address *and* on the source MAC address - this way, whenever we get a packet from a new MAC address, we make sure the controller knows and pushes out the right flow. This works perfectly, but it's inefficient - we need O(n^2) flows, meaning 100 MAC address on a switch requires 20,000 flows (one flow in each direction = 2 flows for each pair of mac addresses).

This is where later versions of OpenFlow start to shine. We can create multiple flow tables, so we have an initial table to check the source MAC (and forward to the controller if we don't know it), and a second table to check the destination MAC (and forward to controller, or just flood, if we don't know it). This way we need to record each source MAC once, and each destination MAC once - so 100 MAC addresses on a switch only needs 200 flows - only O(n).

What it looks like

I've spent the whole weekend getting OpenVSwitch working so I could test it, but this is the final product. My good friend Josh suggested I use the Ryu controller as it has OpenFlow 1.2 and 1.3 support. It comes with a simple_switch application for OpenFlow 1.0, so I modified this to make it work with OpenFlow 1.2. There's a couple of subtle differences between the protocols, but once it was refactored for 1.2, it was pretty easy to set up a second table. The code is here if you want it, and you can just clone this repo when you want to use Ryu - it's up to date with the current github version (as of 14 July 2013), but I'm hoping they'll accept my pull request and just add my app into the main build.

Next steps

I really want to get BIRD working with an OpenFlow controller, so we'll see if that happens - I've got BIRD pushing out a stream of JSON routes that anything can pick up, and I just need this to get turned into flows. It really does feel like we're getting closer to a production OpenFlow controller that plugs into a cheap switch and takes a whole internet route table. Now is a very good time to be a network engineer!

How to build an OpenFlow testbed on an Ubuntu 12.04 VM in VirtualBox

Installing Ubuntu

I started with an Ubuntu Server 12.04.2 64 bit iso, and a VirtualBox VM with 1024MB of RAM and 8GB of hard disk. My version of VirtualBox is 4.0.10r72479 on Windows 7 x64 Professional. The install is pretty normal - if you've never installed Ubuntu Server before, you shouldn't find this too hard - just follow the prompts and keep pressing enter.

This would be a good time to pour yourself a single malt coffee

Don't be too fussy about packages, but as a rule I tend to want to install the OpenSSH server just because it's a good habit to get into - and something that'll trip you up if you forget to just that one time when it's really important.

Pro tip ™

This will take a couple of minutes to finish, and then you'll have an Ubuntu install ready to go. Remember to eject the ISO, and then turn the machine off. Time for the ugly stuff.

Configuring the network stuff

I've set up my testbed with 3x Ubuntu servers, 2 of which were set up like this and left, and a third that we did some special stuff with. For all of them, we'll need to set up extra adaptors on Internal Networks - the two client machines each get a single new adaptor with their own intnet, and the OVS machine (the third one) gets two new adaptors - the first one goes onto intnet1 (to connect to client 1), and the second goes onto intnet2. I've left the original adaptor untouched on all of the machines so we can add packages later without having to break networking.

Edit our new OFSwitch2 VM

Keep Adapter 1 as is so we can download stuff

Intnet1 matches up with client VM 1

Intnet2 to client VM 2

Once you've set this up, find the vbox file for your VM and open it up in your text editor, Make sure you close VirtualBox first - otherwise it won't take your changes. You'll want to add the following lines:

<ExtraDataItem name="VBoxInternal/Devices/e1000/1/LUN#0/Config/IfPolicyPromisc" value="allow-all"/>
<ExtraDataItem name="VBoxInternal/Devices/e1000/2/LUN#0/Config/IfPolicyPromisc" value="allow-all"/>

The secret sauce

That last part is super important - I spend a few hours today and last night trying to figure out why some packets would hit the bridge and others wouldn't - VirtualBox by default will accept broadcasts and unicasts to your address, but not other MAC addresses. Being a switch, you generally want to accept every MAC address except your own, so this is fairly important.

Installing OpenVSwitch

I've used version 1.10 because it's the coolest. Download it to a folder on your VM, untar, and read the INSTALL file because that's what cool kids do. In actual fact, there's a INSTALL.Debian, but that didn't work for me, so I just built it the generic way.

Packages to install (so you don't spend the next hour chasing dependencies):

• build-essential
• pkg-config
• autoconf
• automake
• python-qt-dev
• python-dev
• python-twisted-conch
• libtool

Then run the install

./boot.sh

./configure

make

sudo make install

I'm pleasantly surprised to say that this all worked the first time - just make sure you install all of those packages in one go and it'll work perfectly from the start :)

Running OpenVSwitch

Now is a good time to start up OpenVSwitch to test that everything is working as you should expect - if we do this right, then the OpenFlow part will be easy. Fire up your two client machines, and set up eth1 on both of them to IPs in the same range - I've used 10.1.1.1/24 and 10.1.1.2/24, but use something else if this would clash with your other network.

Once you have them up, start up OpenVSwitch with the following stuff - I've kept them in separate screens to make it easier

Start a screen (screen)

Screen 0:

sudo modprobe openvswitch

sudo ovsdb-tool create

sudo ovsdb-server --remote=ptcp:9999:127.0.0.1

New screen(CTRL+A, C)

Screen 1

sudo ovs-vswitchd tcp:127.0.0.1:9999

Screen 2

ovs-vsctl --db=tcp:127.0.0.1:9999 add-br br0

ovs-vsctl --db=tcp:127.0.0.1:9999 add-port br0 eth1

ovs-vsctl --db=tcp:127.0.0.1:9999 add-port br0 eth2
ovs-vsctl --db=tcp:127.0.0.1:9999 set bridge br0 protocols=OpenFlow12

sudo ifconfig eth1 up

sudo ifconfig eth2 up

If you bring up your client VMs you should be able to ping between them now. If you can, then great - we'll move onto getting OpenFlow working. You need one more line of code, assuming the controller is (or will be) on the same machine:

ovs-vsctl --db=tcp:127.0.0.1:9999 set-controller br0 tcp:127.0.0.1:6633

Getting OpenFlow going

We're on the home straight here. You can install the controller of your choosing, or you can install Ryu with the following instructions:

sudo apt-get install git python-setuptools

git clone http://github.com/osrg/ryu

cd ryu

sudo python setup.py install

You can then sit and watch as it downloads its dependencies from pypi. When it's done, fire up the controller with an app, and you're ready to go.

ryu-manager ryu/app/simple_switch.py

You can check the flow tables (in another screen) with the following command:

sudo ovs-ofctl dump-flows br0

Check out the manpages if you want to learn more:

http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities%2Fovs-ofctl.8

http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities%2Fovs-vsctl.8

What next?

You've got an OpenFlow testbed now, you can do what you want with it. Play with different controllers, or different versions of OpenFlow - it's all up to you.