From the start
We run a mix of Brocade MLX and Juniper MX80's at work, and I've spent the last week trying to make them talk to each other properly. You'd think that by 2013, a multi-vendor network would work fine using standardised protocols, but it's still quite time consuming finding which ways work and which ways most certainly don't. Oddly enough, I'm not the only person who's been working on this recently -
Nick Buraglio has done a bit in the last couple of weeks too (thanks for the help on this)
As SDN starts to take over, this will become much less of a problem, but until then, here's how to do the following with Juniper MX-series routers and Brocade MLX/XMR chassis:
- Jumbo frames
- OSPF
- MPLS
- VPLS
- VLL/l2circuit
- Tagging VPLS/VLL/l2circuit into a VRF on Juniper
Disclaimer - the MX80 chassis has lots of stuff built in, including a tunnel services PIC - we need this for some of the stuff below, not sure how it works with bigger chassis.
Jumbo frames
This should be easy, but there's a couple of things that will trip you up if you aren't careful. The maximum frame size you can have on the Brocades is 9216 bytes, and on the Junipers it's 9192 bytes. I tried to set the frame size on the Brocades down to 9192 bytes and found a weird quirk - I could send 9146 byte pings from the Juniper, but the Brocade would only respond to 9142 byte pings - it appears the Brocades include the FCS in their count of frame size.
In the end, the best solution was to just leave both routers at their maximum values, set VPLS/VLL MTUs to 9100 bytes (or some number with a bit of headroom over 9000), and IP MTUs to 9000 (except for OSPF interfaces, but we'll get to that soon).
The main thing to remember is that on the Junipers, you can set MTU on the physical interface, or inside a "family XXX" stanza, but not directly on a logical interface. If you want to set the IP MTU for a logical interface, it sits in "interface XXX -> unit Y -> family inet -> mtu 9000"
OSPF
You *can* stand up OSPF with Jumbo frames, but it's fine with 1500 byte frames. We're in the position of introducing Junipers into our Brocade OSPF cloud, and since the defaults for Brocade are already 1500 bytes, it's easier to step the Junipers down than bring the Brocades up. I've set up the Brocade-facing interface like this:
interfaces {
ge-1/0/0 {
vlan-tagging;
mtu 9192;
unit 1000 {
vlan-id 1000;
family inet {
mtu 1500
address 10.1.2.1/30;
}
family mpls;
}
}
}
The Brocade end looks like this:
interface ve 100
bfd interval 100 min-rx 100 multiplier 3
ip ospf area 0
ip ospf cost 100
ip ospf dead-interval 40
ip ospf hello-interval 10
ip address 10.2.3.2/30
ip mtu 1500
!
The only tricky bit is making sure the IP MTU is the same for each end - if you get a huge route update going into an interface that can't take the whole packet then you'll end up blackholing routes. Junipers are supposed to not stand up OSPF when there's an IP MTU mismatch, but it doesn't always work - it pays to test with ping packets to confirm - you should be able to ping in either direction with a 1472 bytes of data (1500 - 20 byte IP header - 8 byte ICMP header).
MPLS
This is pretty straightforward - set up loopback interfaces on each end, and enable RSVP and LDP. We'll use RSVP for the outer tags, and LDP for the inner tags - no clever BGP signalling here.
Juniper:
protocols {
rsvp {
interface all {
disable;
}
interface ge-1/0/0.1000;
interface lo0.0;
}
mpls {
label-switched-path 1-to-2 {
from 10.0.0.1;
to 10.0.0.2;
fast-reroute;
}
label-switched-path 1-to-3 {
from 10.0.0.1;
to 10.0.0.3;
fast-reroute;
}
interface all {
disable;
}
interface ge-1/0/0.1000;
}
ospf {
traffic-engineering;
area 0.0.0.0 {
interface all {
disable;
}
interface ge-1/0/0.1000 {
hello-interval 3;
dead-interval 12;
bfd-liveness-detection {
minimum-interval 300;
multiplier 3;
}
}
interface lo0.0 {
passive;
}
}
}
ldp {
interface all {
disable;
}
interface lo0.0;
}
}
Brocade:
router mpls
policy
traffic-eng ospf
mpls-interface ve100
path 3-to-1
loose 10.0.0.1
path 3-to-2
loose 10.0.0.2
path S3-to-1
loose 10.0.0.1
path S3-to-2
loose 10.0.0.2
lsp LSP-3-to-1
to 10.0.0.1
primary 3-to-1
secondary S3-to-1
standby
frr
revert-timer 30
enable
lsp LSP-3-to-2
to 10.0.0.2
primary 3-to-2
secondary S3-to-2
standby
frr
revert-timer 30
enable
VPLS
This is where it gets interesting. In my opinion, Brocade does the right thing (packets come out of a VLAN-tagged "pipe", and then go into a VPLS "pipe"), whereas Juniper does it at a lower and less-abstract level (packets with headers that get altered) - it's more flexible, but it's harder to make it do the right thing.
Raw mode, tagged mode, and tags inside raw mode
VPLS has two modes - Raw mode creates a broadcast domain between all peers on the same VPLS, whereas tagged mode allows you to use inner tags within a VPLS circuit. The problem comes where Juniper sends 802.1q VLAN-tagged packets through a raw-mode VPLS. You end up left with a situation where traffic can go one way, but not the other, and it's all quite confusing.
Raw mode interop
Check out these configs:
On Juniper, we make a routing instance, and add interfaces into it. They can be VLAN-tagged or normal access ports, and there's a very important trick to make it all use raw mode properly - the line "vlan-id none". If you don't do this, packets on an untagged port go through fine, but packets from tagged ports come through with 802.1q VLAN tags on them. On Brocade, a VPLS get delivered to a mix of tagged and untagged ports, but all traffic is sent as normal untagged ethernet. The "vlan-id none" line makes the Junipers behave in the same way. The config below delivers VPLS 40 untagged at both ends, and VPLS 140 tagged as VLAN 140.
Don't worry too much about the MTUs - they need to match up, but they don't appear to be enforced. We picked 9100 as it's well under the 9192 byte hardware MTU, but well above the 9000 byte IP MTU - a bit of leeway in each direction.
Juniper:
routing-instances {
vpls-40 {
description vpls-40;
instance-type vpls;
vlan-id none;
interface ge-1/1/9.40;
protocols {
vpls {
no-tunnel-services;
vpls-id 40;
mtu 9100;
neighbor 10.0.0.2;
neighbor 10.0.0.3;
}
}
}
vpls-140 {
description vpls-140;
instance-type vpls;
vlan-id none;
interface ge-1/1/9.140;
protocols {
vpls {
no-tunnel-services;
vpls-id 140;
mtu 9100;
neighbor 10.0.0.2;
neighbor 10.0.0.3;
}
}
}
}
interfaces {
ge-1/1/9 {
flexible-vlan-tagging;
native-vlan-id 40;
mtu 9192;
encapsulation flexible-ethernet-services;
unit 40 {
encapsulation vlan-vpls;
vlan-id 40;
family vpls;
}
unit 140 {
encapsulation vlan-vpls;
vlan-id 140;
family vpls;
}
}
}
Brocade:
router mpls
vpls vlan40 40
vpls-peer 10.0.0.1 10.0.0.2
vpls-mtu 9100
vlan 40
untagged ethe 1/5
vpls vlan40 40
vpls-peer 10.0.0.1 10.0.0.2
vpls-mtu 9100
vlan 140
tagged ethe 1/5
If you're interested in the mechanics behind the Juniper implementation, the "show interface" command gives you a bit of insight - Juniper interprets the "vlan-id none" line in the routing instance and converts that to tag push/pop operations on the interface:
Logical interface ge-1/1/9.40 (Index 332) (SNMP ifIndex 564)
Flags: SNMP-Traps 0x0
VLAN-Tag [ 0x8100.40 ] Native-vlan-id: 40 In(pop) Out(push 0x8100.40)
Encapsulation: VLAN-VPLS
Input packets : 9
Output packets: 7
Protocol vpls, MTU: 9192
Flags: Is-Primary
Logical interface ge-1/1/9.140 (Index 333) (SNMP ifIndex 597)
Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x8100.140 ] In(pop) Out(push 0x8100.140)
Encapsulation: VLAN-VPLS
Input packets : 149
Output packets: 92
Protocol vpls, MTU: 9192
Flags: Is-Primary
VLL/l2circuit
This is the fun one. Tagged-mode VLLs are the easiest to get up and running, but raw-mode should be doable too. There is one problem though - the only way I can make raw-mode work on the Junipers looks like a filthy hack, but it produces the same results that we see above in the "show interface" output for VPLS.
First off, configs for tagged mode
Brocade:
router mpls
vll vlan42 42
vll-mtu 9100
vll-peer 10.0.0.1
vlan 42
tagged e 1/5
Juniper:
interfaces {
ge-1/1/9 {
flexible-vlan-tagging;
mtu 9192;
encapsulation flexible-ethernet-services;
unit 42 {
encapsulation vlan-ccc;
vlan-id 42;
family ccc;
}
}
}
protocols {
l2circuit {
neighbor 10.0.0.3 {
interface ge-1/1/9.42 {
virtual-circuit-id 42;
mtu 9100;
encapsulation-type ethernet-vlan;
}
}
}
}
This is all pretty straightforward, and works out of the box. Here's what the raw mode config looks like
Brocade:
router mpls
vll vlan41 41 raw-mode
vll-mtu 9100
vll-peer 10.0.0.1
vlan 41
tagged e 1/5
Juniper:
interfaces {
ge-1/1/9 {
flexible-vlan-tagging;
mtu 9192;
encapsulation flexible-ethernet-services;
unit 41 {
encapsulation vlan-ccc;
vlan-id 41;
input-vlan-map pop;
output-vlan-map push;
family ccc;
}
}
}
protocols {
l2circuit {
neighbor 10.0.0.3 {
interface ge-1/1/9.41 {
virtual-circuit-id 41;
mtu 9100;
encapsulation-type ethernet;
}
}
}
}
As you can see, the default for Brocade is tagged mode, so we need to explicitly put it in raw mode. On the Juniper end, we set the encapsulation type on the VLL to ethernet instead of ethernet-vlan, but this only works with encapsulation ethernet-ccc on the physical interface. If we have VLAN tagging mode enabled, there doesn't seem to be any way to tell the MX80 about this. The way I've made this work is with the "input-vlan-map" and "output-vlan-map" statements - they seem to round everything out and make it all work. Given the default for both Juniper and Brocade is tagged mode, and we need a bit of mad hax to make raw mod work here, it might make sense to use tagged mode.
Tagging circuits to VRFs (Juniper only)
This was my favourite part. The way to do this seems to be lt- devices, which means you need to set up the tunnel services PIC (if you have one - the MX80's have one built in).
chassis {
fpc 0 {
pic 0 {
tunnel-services {
bandwidth 1g;
}
}
}
}
This next part took a while to figure out, but it totally works - you just need to make sure you match up the encapsulations and it all works fine.
Here's some config:
routing-instances {
vrf {
instance-type vrf;
interface lt-0/0/10.1;
route-distinguisher 1:2;
vrf-target target:1:2;
vrf-table-label;
}
vrf-43 {
instance-type vrf;
interface lt-0/0/10.3;
route-distinguisher 1:42;
vrf-target target:1:42;
vrf-table-label;
}
vpls-40 {
description vpls-40;
instance-type vpls;
vlan-id none;
interface lt-0/0/10.2;
protocols {
vpls {
no-tunnel-services;
vpls-id 40;
mtu 9100;
neighbor 10.0.0.2;
neighbor 10.0.0.3;
}
}
}
}
protcols {
l2circuit {
neighbor 10.0.0.2 {
interface lt-0/0/10.4 {
virtual-circuit-id 43;
mtu 9100;
encapsulation ethernet-vlan;
}
}
}
}
interfaces {
lt-0/0/10 {
mtu 9192;
unit 1 {
encapsulation ethernet;
peer-unit 2;
family inet {
mtu 9000;
address 192.168.0.13/24;
}
}
unit 2 {
encapsulation ethernet-vpls;
peer-unit 1;
}
unit 3 {
encapsulation ethernet;
peer-unit 4;
family inet {
mtu 9000;
address 192.168.43.13/24;
}
}
unit 4 {
encapsulation ethernet-ccc;
peer-unit 3;
}
}
}
This took about a day to get working, but it's totally simple once it matches up. The lt- devices are instead of crossover cables hanging out the back of your router, and the key is to just set the encapsulation types correctly. You can cheat a little with irb interfaces in VPLS routing instances, but this alters the route table directly on the chassis. Doing it this way means it's locked down to a VRF, and everything is a bit nicer.
End
I hope you've enjoyed this - let me know if there's anything I've missed or got wrong. The moral of the story here is - Juniper and Brocade can do VPLS and VLLs fine between each other - just watch out for the little quirks that would trip you up.
