In Intro to VRF lite, we looked at how virtual routing and forwarding (VRF) instances can be employed to logically separate the layer three topologies of unrelated entities sharing a single physical infrastructure. VRFs work at layer three much like VLANs work at layer two, by assigning interfaces to a virtual domain isolated from other virtual domains at the same layer. This is a simple concept to grasp so long as each of the VRFs are maintained in isolation. However, things get a bit more complex when you need to route traffic from one VRF to another.
Consider the following topology, which might be found at a small business park or college campus.
The network pictured serves three customers: we'll call them customers Red, Green, and Blue. Each customer has its own Internet connection, housed in building one. Customer Green has employees in building two, customer Blue has employees in building three, and customer Red has employees in both buildings two and three. Additionally, VOIP service and connection to the PSTN (represented for now by a loopback interface) will be provided in the future by the network operator to customers Red and Green, but not Blue.
Buildings two and three are connected to building one via 802.1Q trunk links. All three customers rely on the same physical network infrastructure for connectivity, but the traffic from each customer must be isolated from the others for security reasons. The IP prefixes in use are as follows:
- Red: 172.16.0.0/16
- Green: 172.17.0.0/16
- Blue: 172.18.0.0/16
- VOIP Services: 192.168.99.0/24
The design question posed by this scenario is as follows: how can we provide local VOIP connectivity to some customers while keeping all of the customers isolated from one another?
Modifying the Multilayer Switches' SDM Template
Before we can implement VRFs on our multilayer switches (Catalyst 3550s, in this case), we have to modify how the internal ternary content-addressable memory (TCAM) is partitioned to store IP routes. Observe what happens if we attempt to enable VRF with the default SDM template:
S1(config)# ip routing S1(config)# ip vrf Red S1(config-vrf)# %L3TCAM-3-SIZE_CONFLICT: VRF requires enabling extended routing
VRF routes require that an additional unique identifier, a route distinguisher, is prepended to each VRF route in the routing table (we'll cover this in more detail shortly). To accommodate for this, the TCAM must be repartitioned. (This step is only necessary when configuring VRFs on certain multilayer switches.)
S1(config)# sdm prefer extended-match Changes to the running SDM preferences have been stored, but cannot take effect until the next reload. Use 'show sdm prefer' to see what SDM preference is currently active. S1(config)# ^Z S1# show sdm prefer The current template is the default template. The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1K VLANs. number of unicast mac addresses: 5K number of igmp groups: 1K number of qos aces: 1K number of security aces: 1K number of unicast routes: 8K number of multicast routes: 1K The template stored for use after the next reload is the default extended-match template. The selected template optimizes the resources in the switch to support this level of features for 8 routed interfaces and 1K VLANs. number of unicast mac addresses: 5K number of igmp groups: 1K number of qos aces: 1K number of security aces: 1K number of unicast routes: 4K number of multicast routes: 1K
Note that the supported number of unicast routes has been halved from 8K to 4K.
We need to apply this change to and reload all three switches before proceeding.
Creating VRFs
We need to create four VRFs on S1: one for each of the three customers, and one for the shared VOIP network. Each VRF must be assigned a unique route distinguisher. A route distinguisher is simply a unique number stored beside each route in the routing table to associate it with its VRF and maintain uniqueness should two or more VRFs have overlapping address space (for example, if all three of our customers were addressed out of the 10.0.0.0/8 network).
There are two formats in which we can define a route distinguisher:
S1
ip routing ! ip vrf Red rd 65000:1 ! ip vrf Green rd 65000:2 ! ip vrf Blue rd 65000:3 ! ip vrf Shared rd 65000:99
Next, we assign the physical and logical interfaces to VRFs and address them appropriately. We're using a loopback interface in place of a physical interface on S1 to emulate the shared VOIP network, however there is no effective difference with regard to VRF configuration. Note that interfaces must be assigned to a VRF before being addressed; assigning an interface to a VRF wipes any IP addresses already configured on that interface.
S1
vlan 16 vlan 17 vlan 18 ! interface Loopback99 description VOIP Services ip vrf forwarding Shared ip address 192.168.99.1 255.255.255.0 ! interface FastEthernet0/1 no switchport ip vrf forwarding Red ip address 172.16.1.2 255.255.255.252 ! interface FastEthernet0/3 no switchport ip vrf forwarding Green ip address 172.17.1.2 255.255.255.252 ! interface FastEthernet0/5 no switchport ip vrf forwarding Blue ip address 172.18.1.2 255.255.255.252 ! interface FastEthernet0/13 switchport trunk encapsulation dot1q switchport mode trunk ! interface FastEthernet0/16 switchport trunk encapsulation dot1q switchport mode trunk ! interface Vlan16 ip vrf forwarding Red ip address 172.16.0.1 255.255.255.0 ! interface Vlan17 ip vrf forwarding Green ip address 172.17.0.1 255.255.255.0 ! interface Vlan18 ip vrf forwarding Blue ip address 172.18.0.1 255.255.255.0
The physical trunk interfaces F0/13 and F0/16 are not assigned to a VRF, because they are not routed interfaces. VRF configuration is only relevant at layer three.
We can use show ip vrf to examine the VRFs and their assigned interfaces:
S1# show ip vrf
Name Default RD Interfaces
Blue 65000:3 Fa0/5
Vl18
Green 65000:2 Fa0/3
Vl17
Red 65000:1 Fa0/1
Vl16
Shared 65000:99 Lo99
S1# show ip vrf interfaces
Interface IP-Address VRF Protocol
Fa0/5 172.18.1.2 Blue up
Vl18 172.18.0.1 Blue up
Fa0/3 172.17.1.2 Green up
Vl17 172.17.0.1 Green up
Fa0/1 172.16.1.2 Red up
Vl16 172.16.0.1 Red up
Lo99 192.168.99.1 Shared up
We only need to create two VRFs on each of the two other switches, as each connects only two customers, and assign the appropriate interfaces.
S2
ip routing ! ip vrf Red rd 65000:1 ! ip vrf Green rd 65000:2 ! vlan 16 vlan 17 vlan 216 vlan 217 ! interface FastEthernet0/13 switchport trunk encapsulation dot1q switchport mode trunk ! interface Vlan16 ip vrf forwarding Red ip address 172.16.0.2 255.255.255.0 ! interface Vlan17 ip vrf forwarding Green ip address 172.17.0.2 255.255.255.0 ! interface Vlan216 ip vrf forwarding Red ip address 172.16.2.1 255.255.255.0 ! interface Vlan217 ip vrf forwarding Green ip address 172.17.2.1 255.255.255.0
S3
ip routing ! ip vrf Red rd 65000:1 ! ip vrf Blue rd 65000:3 ! vlan 16 vlan 18 vlan 316 vlan 318 ! interface FastEthernet0/13 switchport trunk encapsulation dot1q switchport mode trunk ! interface Vlan16 ip vrf forwarding Red ip address 172.16.0.3 255.255.255.0 ! interface Vlan18 ip vrf forwarding Blue ip address 172.18.0.3 255.255.255.0 ! interface Vlan316 ip vrf forwarding Red ip address 172.16.3.1 255.255.255.0 ! interface Vlan318 ip vrf forwarding Blue ip address 172.18.3.1 255.255.255.0
Configuring OSPF
Next we'll configure OSPF to exchange routes among the three buildings. Three independent OSPF processes are to be run, one per customer VRF. OSPF configuration here is pretty straight forward, as we can simply place all interfaces in area 0 within each VRF.
S1
router ospf 1 vrf Red network 0.0.0.0 255.255.255.255 area 0 ! router ospf 2 vrf Green network 0.0.0.0 255.255.255.255 area 0 ! router ospf 3 vrf Blue network 0.0.0.0 255.255.255.255 area 0
S2
router ospf 1 vrf Red network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan216 ! router ospf 2 vrf Green network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan217 !
S3
router ospf 1 vrf Red network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan316 ! router ospf 3 vrf Blue network 0.0.0.0 255.255.255.255 area 0 passive-interface Vlan318
We can verify that OSPF adjacencies have been appropriately established with show ip ospf neighbor. Note that this command is not VRF-specific.
S1# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 172.18.3.1 1 FULL/DR 00:00:31 172.18.0.3 Vlan18 172.17.2.1 1 FULL/DR 00:00:32 172.17.0.2 Vlan17 172.16.2.1 1 FULL/BDR 00:00:32 172.16.0.2 Vlan16 172.16.3.1 1 FULL/DR 00:00:31 172.16.0.3 Vlan16
At this point, all customers in all buildings should have full connectivity within their respective VRFs.
S2# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 O 172.16.1.0/30 [110/2] via 172.16.0.1, 00:00:31, Vlan16 C 172.16.2.0/24 is directly connected, Vlan216 O 172.16.3.0/24 [110/2] via 172.16.0.3, 00:00:31, Vlan16 S2# show ip route vrf Green Routing Table: Green ... 172.17.0.0/16 is variably subnetted, 3 subnets, 2 masks O 172.17.1.0/30 [110/2] via 172.17.0.1, 00:00:37, Vlan17 C 172.17.0.0/24 is directly connected, Vlan17 C 172.17.2.0/24 is directly connected, Vlan217
S3# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 O 172.16.1.0/30 [110/2] via 172.16.0.1, 00:01:04, Vlan16 O 172.16.2.0/24 [110/2] via 172.16.0.2, 00:01:04, Vlan16 C 172.16.3.0/24 is directly connected, Vlan316 S3# show ip route vrf Blue Routing Table: Blue ... 172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks C 172.18.3.0/24 is directly connected, Vlan318 C 172.18.0.0/24 is directly connected, Vlan18 O 172.18.1.0/30 [110/2] via 172.18.0.1, 00:01:05, Vlan18
Configuring Multiprotocol BGP
Multiprotocol BGP (MP-BGP or BGP-MP) will be used only to exchange routes between VRFs, and therefore needs to be configured only on S1. We'll use the private AS number 65000 for the BGP process.
Because all of our routed interfaces on S1 have been assigned to VRFs, we'll create a loopback interface not in a VRF to keep BGP from complaining about not being able to find a router ID. (In the real world, one might consider using the IP of a management interface as the BGP router ID.) A router ID can also be configured per BGP address family.
S1(config)# interface loopback0 S1(config-if)# ip address 192.0.2.1 255.255.255.255
To configure MP-BGP, we configure a separate address family within BGP for each VRF and simply redistribute all connected and OSPF-learned routes within that VRF.
S1
router bgp 65000 no synchronization bgp log-neighbor-changes no auto-summary ! address-family ipv4 vrf Red redistribute connected redistribute ospf 1 vrf Red no synchronization exit-address-family ! address-family ipv4 vrf Green redistribute connected redistribute ospf 2 vrf Green no synchronization exit-address-family ! address-family ipv4 vrf Blue redistribute connected redistribute ospf 3 vrf Blue no synchronization exit-address-family ! address-family ipv4 vrf Shared redistribute connected no synchronization exit-address-family
We can verify that each BGP address family now maintains routes for its respective VRF. (To view the BGP routes for only a specific VRF, use the command show ip bgp ***v4 vrf.)
S1# show ip bgp ***v4 all
BGP table version is 47, local router ID is 192.0.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65000:1 (default for vrf Red)
*> 172.16.0.0/24 0.0.0.0 0 32768 ?
*> 172.16.1.0/30 0.0.0.0 0 32768 ?
*> 172.16.2.0/24 172.16.0.2 2 32768 ?
*> 172.16.3.0/24 172.16.0.3 2 32768 ?
Route Distinguisher: 65000:2 (default for vrf Green)
*> 172.17.0.0/24 0.0.0.0 0 32768 ?
*> 172.17.1.0/30 0.0.0.0 0 32768 ?
*> 172.17.2.0/24 172.17.0.2 2 32768 ?
Route Distinguisher: 65000:3 (default for vrf Blue)
*> 172.18.0.0/24 0.0.0.0 0 32768 ?
*> 172.18.1.0/30 0.0.0.0 0 32768 ?
*> 172.18.3.0/24 172.18.0.3 2 32768 ?
Route Distinguisher: 65000:99 (default for vrf Shared)
*> 192.168.99.0 0.0.0.0 0 32768 ?
Enabling VRF Route Import and Export
Our final task is to configure route import and export among the Red, Green, and Shared VRFs. VRF route import and export are configured as two explicit, independent functions. Under VRFs Red and Green, we need to:
- Export local routes
- Import routes from VRF Shared
Under VRF Shared:
- Export local routes
- Import routes from VRFs Red and Green
To do this, we configure each VRF with import and export route targets. A route target is similar in format to a route distinguisher, but serves a different purpose. Whereas any given VRF route in our scenario has exactly one route distinguisher to uniquely identify it, a route can have zero or more route targets as arbitrary identifiers for reference by import and export policies. This allows for much greater flexibility than if only the route distinguisher was used. Although used only locally in our example, route targets are communicated with MP-BGP neighbors as extended communities.
Route targets can be defined in the same formats as route distinguishers. Here each route target will reuse the same format and value of its VRF's route distinguisher, just to keep things simple. However, remember that there is no direct relation between the two. Additionally, although each VRF is exporting only one route-target in our example, multiple route-targets can be exported simultaneously (resulting in multiple communities attached to an MP-BGP route).
ip vrf Blue rd 65000:3 ! ip vrf Green rd 65000:2 route-target export 65000:2 route-target import 65000:99 ! ip vrf Red rd 65000:1 route-target export 65000:1 route-target import 65000:99 ! ip vrf Shared rd 65000:99 route-target export 65000:99 route-target import 65000:1 route-target import 65000:2
Now VRF Shared has routes from VRFs Red and Green, and VRFs Red and Green each have the route from VRF Shared (it may take a minute for the routes to appear in the table):
S1# show ip route vrf Shared
Routing Table: Shared
...
172.17.0.0/16 is variably subnetted, 2 subnets, 2 masks
B 172.17.1.0/30 is directly connected, 00:00:24, FastEthernet0/3
B 172.17.0.0/24 is directly connected, 00:00:24, Vlan17
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
B 172.16.0.0/24 is directly connected, 00:00:24, Vlan16
B 172.16.1.0/30 is directly connected, 00:00:24, FastEthernet0/1
C 192.168.99.0/24 is directly connected, Loopback99
S1# show ip route vrf Red
Routing Table: Red
...
172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks
C 172.16.0.0/24 is directly connected, Vlan16
C 172.16.1.0/30 is directly connected, FastEthernet0/1
O 172.16.2.0/24 [110/2] via 172.16.0.2, 00:05:38, Vlan16
O 172.16.3.0/24 [110/2] via 172.16.0.3, 00:05:38, Vlan16
B 192.168.99.0/24 is directly connected, 00:00:44, Loopback99
S1# show ip route vrf Green
Routing Table: Green
...
172.17.0.0/16 is variably subnetted, 3 subnets, 2 masks
C 172.17.1.0/30 is directly connected, FastEthernet0/3
C 172.17.0.0/24 is directly connected, Vlan17
O 172.17.2.0/24 [110/2] via 172.17.0.2, 00:05:43, Vlan17
B 192.168.99.0/24 is directly connected, 00:00:49, Loopback99
VRF Blue, however, still contains only its own routes, as we did not configure import or export for this VRF:
S1# show ip route vrf Blue Routing Table: Blue ... 172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks O 172.18.3.0/24 [110/2] via 172.18.0.3, 00:06:02, Vlan18 C 172.18.0.0/24 is directly connected, Vlan18 C 172.18.1.0/30 is directly connected, FastEthernet0/5
Lastly, we need to configure the VRF OSPF processes to redistribute BGP-learned routes to OSPF peers on S1:
S1(config)# router ospf 1 vrf Red S1(config-router)# redistribute bgp 65000 subnets S1(config-router)# router ospf 2 vrf Green S1(config-router)# redistribute bgp 65000 subnets
These routes are now propagated via OSPF to S2 and S3 within their respective VRFs. On S3, we can verify that VLAN 316 (VRF Red) has access to the VOIP network, but VLAN 318 (VRF Blue) does not:
S3# show ip route vrf Red Routing Table: Red ... 172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks C 172.16.0.0/24 is directly connected, Vlan16 O 172.16.1.0/30 [110/2] via 172.16.0.1, 00:10:15, Vlan16 O 172.16.2.0/24 [110/2] via 172.16.0.2, 00:10:15, Vlan16 C 172.16.3.0/24 is directly connected, Vlan316 O E2 192.168.99.0/24 [110/1] via 172.16.0.1, 00:10:15, Vlan16 S3# show ip route vrf Blue Routing Table: Blue ... 172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks C 172.18.3.0/24 is directly connected, Vlan318 C 172.18.0.0/24 is directly connected, Vlan18 O 172.18.1.0/30 [110/2] via 172.18.0.1, 00:25:17, Vlan18 S3# ping vrf Red 192.168.99.1 source vlan316 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.99.1, timeout is 2 seconds: Packet sent with a source address of 172.16.3.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms S3# ping vrf Blue 192.168.99.1 source vlan318 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.99.1, timeout is 2 seconds: Packet sent with a source address of 172.18.3.1 ..... Success rate is 0 percent (0/5)
The final configurations of all three switches are made available here:
- S1.txt
- S2.txt
- S3.txt