< back to notes

Experimenting with Proxmox SDN and EVPN

·
#networking#bgp#proxmox#sdn#evpn

Introduction

I want to experiment with Proxmox SDN, BGP and EVPN within a pre-existing BGP network. I have a basic understanding of BGP, but not so much with EVPN.

As I didn’t want to mess with my production Proxmox instances, I spun up 3 Proxmox clustered nodes on old Dell OptiPlex computers and a single switch.

Equipment/networking available:

  • 1x FortiGate 40F
  • 3x Dell Optiplex desktops
  • 1x Dell N3024ET
  • An existing BGP routed network over a IPsec Tunnel

The physical topology:

Goals

The goal is to be able to peer with my existing BGP network and to be able to route traffic between the Proxmox cluster running Proxmox SDN and the existing network.

The setup

The FortiGate 40F is already peered with a route reflector via iBGP over an IPsec tunnel (ASN 65400). There are several other BGP peers on the route reflector, also over IPsec.

Switch configuration

The switch will be configured with 2 VLANs, one for the Proxmox cluster nodes and one for the switch management. The switch will also have a trunk port to the FortiGate 40F.

ip routing
vlan 100
name "Switches"
exit
vlan 101
name "PVE-Hosts"
exit
interface vlan 1
exit
interface vlan 100
ip address 10.254.100.2 255.255.255.0
exit
interface vlan 101
ip address 10.254.101.1 255.255.255.0
exit
interface Gi1/0/1
switchport access vlan 101
exit
interface Gi1/0/2
switchport access vlan 101
exit
interface Gi1/0/3
switchport access vlan 101
exit
interface Gi1/0/24
switchport mode trunk
switchport trunk allowed vlan 100
exit

The switch will peer with the FortiGate 40F and the Proxmox cluster nodes via BGP. The configuration is as follows:

router bgp 65100
bgp router-id 10.254.100.2
network 10.254.101.0 mask 255.255.255.0
neighbor 10.254.100.1 remote-as 65400
neighbor 10.254.101.2 remote-as 65001
neighbor 10.254.101.3 remote-as 65001
neighbor 10.254.101.4 remote-as 65001
exit

The FortiGate 40F configuration

The FortiGate 40F will be configured to peer with the switch and the Proxmox cluster nodes via BGP. The configuration is as follows:

config router bgp
    set as 65400
    config neighbor
        edit "10.11.0.1"
            set next-hop-self enable
            set remote-as 65400
        next
        edit "10.254.100.2"
            set capability-default-originate enable
            set interface "vlan100"
            set remote-as 65100
        next
    end
exit

The 10.11.0.1 neighbor is the route reflector that the FortiGate 40F is peering with over the IPsec tunnel. The 10.254.100.2 neighbor is the switch that the FortiGate 40F is peering with over the VLAN100 interface.

I had to enable next-hop-self for peers on the upstream side to reach the downstream peers behind the FortiGate 40F (Switch and Proxmox cluster), without it the route reflector would advertise the next-hop as the WAN gateway of the route reflector, not the VTI interface’s IP of the FortiGate 40F.

Another option along with that was the capability-default-originate, it was enabled to allow the FortiGate 40F to advertise the default route out to the internet (0.0.0.0/0) to the switch and downstream peers.

After setting up the BGP peering, I confirmed our peers were in the ESTABLISHED state and that the routes were being advertised and received correctly.

On the FortiGate:

FortiGate40F# get router info bgp neighbors 10.254.100.2 | grep "BGP state"
  BGP state = Established, up for 00:02:32
FortiGate40F# get router info routing-table bgp | grep 10.254.
B       10.254.101.0/24 [20/0] via 10.254.100.2 (recursive is directly connected, vlan100), 00:02:32, [1/0]

The FortiGate is receiving the PVE Hosts network advertised by the switch.

On the switch:

t-switch-01#show ip bgp sum

IPv4 Routing .................................. Enable
BGP Admin Mode ................................ Enable
BGP Router ID ................................. 10.254.100.2
Local AS Number ............................... 65100
...
t-switch-01>show ip route

Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, K - Kernel, S - Static
       B - BGP Derived, E - Externally Derived, IA - OSPF Inter Area
       E1 - OSPF External Type 1, E2 - OSPF External Type 2
       N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2
       S U - Unnumbered Peer, L - Leaked Route, T - Truncated ECMP Route

 * Indicates the best (lowest metric) route for the subnet.

Default Gateway is 10.254.100.1
B      *0.0.0.0/0 [20/0] via 10.254.100.1,   00h:02m:32s,  Vl100
...
C      *10.254.100.0/24 [0/0] directly connected,   Vl100
B       10.254.100.0/24 [20/0] via 10.254.100.1,   00h:02m:32s,  Vl100
C      *10.254.101.0/24 [0/0] directly connected,   Vl101

The switch is properly receiving the default route from the FortiGate 40F, along with the many other routes from the route reflector.

The Proxmox setup and SDN configuration

I installed Proxmox on the 3 Dell Optiplex computers and created a cluster, each host was given a static IP address on the vlan 101 network (10.254.101.0/24). The Proxmox cluster nodes are as follows:

Hostname IP Address
t-pve-01 10.254.101.2
t-pve-02 10.254.101.3
t-pve-03 10.254.101.4

After the cluster was created, I created a EVPN controller:

Datacenter > SDN > Options > Controllers > Add > evpn

Option Value
ID evpn
ASN # 65001
Peers 10.254.101.2, 10.254.101.3, 10.254.101.4

A new zone was created:

Datacenter > SDN > > Zones > Add > EVPN

Option Value
ID evpn1
Primary Controller evpn
VRF-VXLAN Tag 150
Exit Nodes t-pve-01, t-pve-02, t-pve-03
Primary Exit Node t-pve-01
Advertise Subnets true
MTU 1450
IPAM pve

A new VNet was created:

Datacenter > SDN > VNets > Create

Option Value
Name test
Alias Test VNET
Zone evpn1
Tag 1501

And a subnet for the EVPN VNet:

VNets > test > Subnets > Create

Option Value
Subnet 10.254.151.0/24
Gateway 10.254.151.1
SNAT false

The gateway will be installed as an anycast gateway in the EVPN. Each VM that is connected to the EVPN VNet will have the same gateway IP address but have a different route advertised to upstream BGP peers, depending on which Proxmox node the VM is running on. The gateway will be installed on all 3 Proxmox nodes, but only the node that is running the VM will advertise the route to the upstream BGP peers.

Upstream BGP peers were created for each Proxmox node:

Datacenter > SDN > Options > Controllers > Add > bgp

Option Value
Node: t-pve-01
ASN #: 65001
Peers: 10.254.101.1
EBGP: true
Option Value
Node: t-pve-02
ASN #: 65001
Peers: 10.254.101.1
EBGP: true
Option Value
Node: t-pve-03
ASN #: 65001
Peers: 10.254.101.1
EBGP: true

After doing so, I applied the configuration via Datacenter > SDN > Apply

Then confirmed that all 3 hosts were peering with the switch and that the routes were being advertised and received correctly.

t-switch-01>show ip bgp neighbors

...
Remote Address ................................ 10.254.101.2
Remote AS ..................................... 65001
Peer ID ....................................... 10.254.101.2
Peer Admin Status ............................. START
Peer State .................................... ESTABLISHED
...
Remote Address ................................ 10.254.101.3
Remote AS ..................................... 65001
Peer ID ....................................... 10.254.101.3
Peer Admin Status ............................. START
Peer State .................................... ESTABLISHED
...
Remote Address ................................ 10.254.101.4
Remote AS ..................................... 65001
Peer ID ....................................... 10.254.101.4
Peer Admin Status ............................. START
Peer State .................................... ESTABLISHED

Confirmed route to the newly created subnet is being advertised to the switch

t-switch-01>show ip route

Default Gateway is 10.254.100.1
Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, K - Kernel, S - Static
       B - BGP Derived, E - Externally Derived, IA - OSPF Inter Area
       E1 - OSPF External Type 1, E2 - OSPF External Type 2
       N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2
       S U - Unnumbered Peer, L - Leaked Route, T - Truncated ECMP Route

 * Indicates the best (lowest metric) route for the subnet.

...
B      *10.254.151.0/24 [20/0] via 10.254.101.2,   00h:02m:31s,  Vl101

Confirmed that EVPN peering is working amoung the Proxmox nodes:

root@t-pve-01:~# vtysh -c 'show bgp l2vpn evpn summary'
BGP router identifier 10.254.101.2, local AS number 65001 VRF default vrf-id 0
BGP table version 0
RIB entries 11, using 1672 bytes of memory
Peers 2, using 47 KiB of memory
Peer groups 2, using 128 bytes of memory

Neighbor               V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
t-pve-02(10.254.101.3) 4      65001     13648     13577      107    0    0 11:16:42            3        3 FRRouting/10.6.1
t-pve-03(10.254.101.4) 4      65001     13561     13579      107    0    0 11:16:44            1        3 FRRouting/10.6.1

Total number of neighbors 2

Testing/VM Creation

At this point, everything was looking good.

I spun up a Debian instance on the Proxmox cluster, and bridged the network to the test VNet. I originally had the DHCP ranges setup on the VNet’s subnet, but was not able to get a DHCP lease on the VM. Instead, I had assigned it 10.254.151.33 with the default gateway as 10.254.151.1, I was able to complete the installation and drop into a shell where I was able to test pinging out to the internet, which worked, and pinging internal resources to the existing BGP network after creating a firewall rule on the RR.

The /32 route for the VM was advertised to the switch and the FortiGate 40F, and propigate back to the rest of the network fairly quickly.

Observations

After leaving the virtual machine idle for a while, I noticed that the route for the VM was no longer being advertised to the switch and FortiGate 40F and thus the rest of the network. After some research, I found that this is expected behavior with EVPN, as the route is only advertised when there is traffic to/from the VM. If the VM is idle for a certain period of time, the route will be pulled.

Conclusion

The end-toplogy looks like this: