Engineering a solution to a long term home lab problem
Overview
In the last ~10 years of self hosting services, one problem has always stuck with me, and that is not having static IP addresses. As someone who wants to host their own DNS name servers, mail servers, and other services that should be hosted with a static IP address, it is extremely hard to do so behind a residential internet connection, every time the modem reboots or firewall goes down for more than a few minutes, a new IP address is handed out.
While there are solutions to that such as a DDNS service, I don’t really want to hand that responsibility off to another provider, rather I would like to control as much infrastructure as possible.
Recently I was thinking if I could engineer a solution that would provide me a way to DNAT from a VPS to an endpoint locally on my network, or even better, across multiple networks geographically separated. There are solutions out there that are similar to this like some Zero Trust proxy solutions such as Pangolin, but those work on the premise of using either the Proxy protocol or using HTTP headers such as the X-Real-IP header to preserve connection information. I want to host services that might not be HTTP based or might not support the Proxy protocol.
Weighing Risks vs Rewards
Both the networks which I run Home Lab infrastructure at, either in “production” or just to experiment are much less reliable than a large scale VPS provider, I have had issues before with extended power outages, internet outages, or other major inconveniences which have prevented me from accessing my services which I host.
As engineering a solution to increase availability of self hosted services may be complicated, I feel as if the management complexity is worth it to not risk inaccessibility of services (such as DNS) because of a public IP change, though it will be mostly scoped down to the most critical services rather than the “nice to have” services.
As this introduces a single point of failure for ingress traffic, I plan to add a second ingress route for redundancy and load balancing purposes, however it’s not a primary concern right now.
Reasoning to my solution
As mentioned above, some services which I like to host are not compatible with being proxied, or if they are, they would not retain connection information. I’ve evaluated several options including eVPN/Proxmox SDN, but that requires clustering, Kubernetes w/ Calico BGP was also another thought but both of these requires low latency between cluster members, which would not be possible given both sites are Coax connections in two different locations with two different ISPs.
The solution
The first thing I do when thinking about an engineering problem is to draw it out. Below is my thought process to this solution in the form of a diagram:
Let’s jump into the technical details now.
My Goals
What are the goals?
- To have a static IP address, which is DNAT’d in the cloud VPS
- To have the ability to DNAT against servers in two geographically different locations
- To retain connection information across the entire lifetime of session (no SNAT from the VPS or locally)
- To be able to migrate infrastructure from one site to another and have automatic re-convergence via dynamic routing
- To provide a failover route for service IPs if site-to-site communication fails
- To not force regular egress communication out the VPS, as it has a metered connection
- To spend $0 on licensing, other than infrastructure itself
Protocols and Technology stack
As we are dealing with all linux servers (a mix of Debian and Ubuntu), we can use a pretty standardized stack across the servers.
Cloud Router (CR)
Each site will have a IPsec tunnel back to the cloud router, running Ubuntu 24.04 (referred to here as CR), the CR will be running StrongSwan.
The CR will be running FRR (FRRouting) for BGP and BFD, and will have an ASN of 65000
NFTables will be used to DNAT to “Service IPs”.
Site routers
There are currently two sites, each running a FortiGate (Currently on the FortiOS 8.0 version… :)).
They are already in a BGP Hub and Spoke deployment, which provides connectivity to some other FortiGates and routers elsewhere.
The two FortiGates we are focusing on for this solution are in ASN 65400 peered via iBGP to each other. These two FortiGates will be peered to ASN 65000 via eBGP (the CR).
All of the FortiGates running home lab infrastructure have a standardized set of loopback addresses accessible at each site, for this deployment an “anycast” loopback interface was made at 10.254.254.4/32 with the name of loop-ac-bgp, accessible from the “transit”/local subnets which the servers sit on.
Servers
As stated above, these are servers running a mix of Ubuntu and Debian, no Windows.
Each of the servers which will be running BIRD2 as it’s fairly light-weight and supports BFD.
Each server peers back to the FortiGate with bird, and advertises its service IP.
Asymmetric routing
As we are DNATing and not changing the source IP, we run into two issues:
- Reverse Path Forwarding checks fail
- Asymmetric routing
Turning off src-check on the tunnel interface resolves the first problem but the second problem still prevents communication back to the client.
The packet comes in looking like the following:
| Interface | Direction | Source | Destination | Flag |
|---|---|---|---|---|
| to-CR-01 | in | 203.0.113.44:4893 | 10.98.0.14:443 | syn |
and the return:
| Interface | Direction | Source | Destination | Flag |
|---|---|---|---|---|
| wan1 | out | 10.98.0.14:443 | 203.0.113.44:4893 | syn/ack |
Here we’ve run into an asymmetric routing issue, we are not returning communication back out the interface which it came in on because we might run into one of the two following scenarios:
Either there is a static route for dst 0.0.0.0/0 (or equivalent) or the FortiGate picked up a default route from DHCP that matches 0.0.0.0/0
How is this solved?
Policy based routing can assist with this, by creating a Policy Based Route in the FortiGate to catch packets from the source of the service IP range 10.98.0.0/24, we can catch those packets and send them back out the proper interface and gateway. Such policy looks like this:
config router policy
edit 1
set input-device "vlan102"
set src "10.98.0.0/255.255.255.0"
set dst "10.0.0.0/255.0.0.0"
set dst-negate enable
set gateway 10.99.1.0
set output-device "to-CR-01"
set comments "Service IPs return traffic via CR-01"
next
endConfiguration
Enough blabbing on, let’s start configuring everything.
Provisioning the CR (CR-01)
- Networking stack installation:
apt install charon-systemd strongswan-swanctl -yDisable route installations from StrongSwan.
Create a configuration file at /etc/strongswan.d/99-hub.conf
charon-systemd {
install_routes = no
}
charon {
install_routes = no
}- Kernel Parameters:
Because we will be forwarding packets and dealing with asymmetric routing on the CR, we will have to adjust some kernel parameters:
/etc/sysctl.d/90-router.conf:
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2Apply:
sysctl --systemParameter explanation:
net.ipv4.ip_forwardwith a value of1allows IP packet forwarding on the host.net.ipv4.conf.all.rp_filterwith a value of2enables “loose” RPF checks, meaning the address must be reachable via some interface, not specifically the best path back to the sender (has some route in the routing table)net.ipv4.conf.default.rp_filteris the same RPF logic as above, but I believe this parameter takes effect for all new network interfaces
- XFRM Interface setup
The CR will run XFRM interfaces so this will be a route-based tunnel setup.
Each site will have a RFC3021 point-to-point /31 subnet designated, one address for the FortiGate and one address for the CR, as it is point-to-point.
As there are two sites, two networks and XFRM interfaces will be created:
Site 1 (site-a):
/etc/systemd/network/50-xfrm-site-a.netdev:
[NetDev]
Name=xfrm-site-a
Kind=xfrm
[Xfrm]
InterfaceId=1
Independent=yes/etc/systemd/network/50-xfrm-site-a.network:
[Match]
Name=xfrm-site-a
[Link]
MTUBytes=1400
[Network]
Address=10.99.1.0/31Site 2 (site-b):
/etc/systemd/network/50-xfrm-site-b.netdev:
[NetDev]
Name=xfrm-site-b
Kind=xfrm
[Xfrm]
InterfaceId=2
Independent=yes/etc/systemd/network/50-xfrm-site-b.network:
[Match]
Name=xfrm-site-b
[Link]
MTUBytes=1400
[Network]
Address=10.99.1.2/31You may notice the interface’s MTU is set to 1400; this is to account for protocol overhead, such as the headers added on to each packet (ex. IPsec adds ~60 bytes to each packet), and to account for other underlay networks along our communication path.
- Hub Dialup Tunnel Setup:
/etc/swanctl/conf.d/hub.conf:
connections {
site-a-rtr-01-p1 {
version = 2
local_addrs = 203.0.113.89
remote_addrs = %any
proposals = aes256gcm16-prfsha384-ecp384
dpd_delay = 30s
mobike = no
unique = replace
local {
auth = psk
id = @cr-01
}
remote {
auth = psk
id = @site-a-rtr-01
}
children {
site-a-rtr-01-p2 {
local_ts = 0.0.0.0/0
remote_ts = 0.0.0.0/0
if_id_in = 1
if_id_out = 1
esp_proposals = aes256gcm16-ecp384
start_action = none
dpd_action = clear
}
}
}
site-b-rtr-01-p1 {
version = 2
local_addrs = 203.0.113.89
remote_addrs = %any
proposals = aes256gcm16-prfsha384-ecp384
dpd_delay = 30s
mobike = no
unique = replace
local {
auth = psk
id = @cr-01
}
remote {
auth = psk
id = @site-b-rtr-01
}
children {
site-b-rtr-01-p2 {
local_ts = 0.0.0.0/0
remote_ts = 0.0.0.0/0
if_id_in = 2
if_id_out = 2
esp_proposals = aes256gcm16-ecp384
start_action = none
dpd_action = clear
}
}
}
}
secrets {
ike-site-a-rtr-01 {
id = @site-a-rtr-01
secret = "xxx"
}
ike-site-b-rtr-01 {
id = @site-b-rtr-01
secret = "xxx"
}
}PSK secrets can be generated with the following:
openssl rand -base64 32- NFTables setup:
We won’t be using UFW for this, we will use NFTables directly, it gives more control.
#!/usr/sbin/nft -f
flush ruleset
define WAN = "eth0"
define PUBLIC_IP = 203.0.113.89
define MGMT_IP = {
203.0.113.19
}
table inet filter {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
ct state invalid drop
iif "lo" accept
icmp type { echo-request, destination-unreachable, time-exceeded } accept
# Accept tcp/22 from $mgmt_ip
ip saddr $MGMT_IP tcp dport 22 accept
# Accept udp/500 (IKE), udp/4500 (IKE NAT-T) for IPsec underlay
iifname $WAN udp dport { 500, 4500 } accept
# Accept ESP protocol
iifname $WAN meta l4proto esp accept
# Accept (from xfrm-* tunnels) to CR tcp/179 (BGP), udp/3784, udp/3785 (BFD)
iifname "xfrm-*" tcp dport 179 accept
iifname "xfrm-*" udp dport { 3784, 3785 } accept
}
chain forward {
type filter hook forward priority filter; policy drop;
# TCP SYN Maximum segment size (MSS) clamping.
# (Use MTU of outgoing route "rt mtu")
oifname "xfrm-*" tcp flags syn tcp option maxseg size set rt mtu
iifname "xfrm-*" tcp flags syn tcp option maxseg size set rt mtu
ct state established,related accept
ct state invalid drop
# Service IP Dnat rule (See ip nat table below)
iifname $WAN oifname "xfrm-*" ct status dnat accept
# Service IP to service IP between sites (CR fallback path)
iifname "xfrm-*" oifname "xfrm-*" ip saddr 10.98.0.0/24 ip daddr 10.98.0.0/24 accept
}
chain output {
type filter hook output priority filter; policy accept;
}
}
table ip nat {
chain prerouting {
type nat hook prerouting priority dstnat;
# Dnat Rules
# Examples:
#
# DNAT many ports:
# iifname $WAN ip daddr $PUBLIC_IP tcp dport { 80, 443 } dnat to 10.98.0.1
#
# DNAT Dst Port rewrite:
# iifname $WAN ip daddr $PUBLIC_IP tcp dport { 81 } dnat to 10.98.0.2:80
iifname $WAN ip daddr $PUBLIC_IP tcp dport { 80, 443 } dnat to 10.98.0.1
iifname $WAN ip daddr $PUBLIC_IP tcp dport { 81 } dnat to 10.98.0.2:80
}
}Check syntax:
nft -c -f /etc/nftables.conf Enable with systemctl enable --now nftables or apply rules with nft -f /etc/nftables.conf
- FRR setup:
FRR will be responsible for BGP and BFD on the CR.
Install FRR:
apt install frr frr-pythontools -yEnable bgpd and bfdd daemons in /etc/frr/daemons
Restart frr with systemctl restart frr
Check to see if FRR is up:
$ vtysh -c "show ver"
FRRouting 8.4.4 (cr-01) on Linux(6.8.0-31-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
...Write out the FRR config at /etc/frr/frr.conf:
frr defaults traditional
hostname cr-01
log syslog informational
service integrated-vtysh-config
!
interface lo
ip address 10.99.0.0/32
exit
!
! only match /32s
ip prefix-list PL-SERVICES seq 10 permit 10.98.0.0/24 ge 32
!
! CR loopback address
ip prefix-list PL-CR-LOOPBACK seq 10 permit 10.99.0.0/32
!
! communities:
! 1 = site-a
! 2 = site-b
! x = some other site
bgp community-list standard CL-SITE-A seq 10 permit 65400:1
bgp community-list standard CL-SITE-B seq 10 permit 65400:2
!
! allow service IPs inbound
route-map RM-SITE-A-IN permit 10
match ip address prefix-list PL-SERVICES
match community CL-SITE-A
exit
route-map RM-SITE-B-IN permit 10
match ip address prefix-list PL-SERVICES
match community CL-SITE-B
exit
!
! allow CR loopback and service IPs (failover route) outbound by community
route-map RM-TO-SITE-A permit 10
match ip address prefix-list PL-CR-LOOPBACK
exit
route-map RM-TO-SITE-A permit 20
match ip address prefix-list PL-SERVICES
match community CL-SITE-B
exit
route-map RM-TO-SITE-B permit 10
match ip address prefix-list PL-CR-LOOPBACK
exit
route-map RM-TO-SITE-B permit 20
match ip address prefix-list PL-SERVICES
match community CL-SITE-A
exit
!
router bgp 65000
bgp router-id 10.99.0.0
bgp log-neighbor-changes
neighbor 10.99.1.1 remote-as 65400
neighbor 10.99.1.1 description SITE-A-RTR-01
neighbor 10.99.1.1 bfd
neighbor 10.99.1.3 remote-as 65400
neighbor 10.99.1.3 description SITE-B-RTR-01
neighbor 10.99.1.3 bfd
!
address-family ipv4 unicast
network 10.99.0.0/32
! Both sites are AS 65400
! Rewrite the ASN to the CRs 65000 because the downstream FortiGate will not accept its own ASN as a path via eBGP
! Add any site in here with as-override that is in 65400
neighbor 10.99.1.3 as-override
neighbor 10.99.1.1 as-override
neighbor 10.99.1.1 soft-reconfiguration inbound
neighbor 10.99.1.1 route-map RM-SITE-A-IN in
neighbor 10.99.1.1 route-map RM-TO-SITE-A out
neighbor 10.99.1.3 soft-reconfiguration inbound
neighbor 10.99.1.3 route-map RM-SITE-B-IN in
neighbor 10.99.1.3 route-map RM-TO-SITE-B out
! for ECMP/anycast IP addresses, increment if more than 2 sites are using the same service IP
maximum-paths 2
exit-address-family
exitRestart FRR with systemctl restart frr
Standing up FortiGate tunnels and BGP sessions
- Create IPsec tunnels at both sites:
config vpn ipsec phase1-interface
edit "to-CR-01"
set interface "wan1"
set ike-version 2
set peertype any
set net-device disable
set proposal aes256gcm-prfsha384
set dhgrp 20
set remote-gw 203.0.113.89
set localid-type fqdn
set localid "site-a-rtr-01"
set psksecret 1234
set dpd on-idle
set dpd-retryinterval 10
set nattraversal enable
next
end
config vpn ipsec phase2-interface
edit "to-CR-01"
set phase1name "to-CR-01"
set proposal aes256gcm
set dhgrp 20
set auto-negotiate enable
next
endThe tunnel interfaces:
Site 1:
config system interface
edit "to-CR-01"
set ip 10.99.1.1 255.255.255.255
set remote-ip 10.99.1.0 255.255.255.254
set allowaccess ping
set bfd enable
# needed otherwise RPF checks will fail for incoming pkts
set src-check di
next
endSite 2:
config system interface
edit "to-CR-01"
set ip 10.99.1.3 255.255.255.255
set remote-ip 10.99.1.2 255.255.255.254
set allowaccess ping
set bfd enable
# needed otherwise RPF checks will fail for incoming pkts
set src-check di
next
endNotice the src-check set to disable, this is needed because we are doing asymmetric routing, as stated above, we do not have a 0.0.0.0/0 route back through to the CR so RPF checks will fail and drop the packet, which is the default intended behavior.
There are better ways of mitigating this, but for now this will suffice as it is only done on the CR tunnel interface.
The tunnels should be up at this point.
- BGP configuration:
config router prefix-list
# service IPs (/32s)
edit "prefix-Service-IPs"
config rule
edit 1
set prefix 10.98.0.0 255.255.255.0
set ge 32
set le 32
next
end
next
# accept CR loopback and service IPs (failover route) from CR
edit "prefix-From-CR-01"
config rule
edit 1
set prefix 10.99.0.0 255.255.255.255
unset ge
unset le
next
edit 2
set prefix 10.98.0.0 255.255.255.0
set ge 32
set le 32
next
end
next
# send only service IPs to the CR
edit "prefix-To-CR-01"
config rule
edit 1
set prefix 10.98.0.0 255.255.255.0
set ge 32
set le 32
next
end
next
end
config router route-map
edit "rm-Servers-In"
config rule
edit 1
set match-ip-address "prefix-Service-IPs"
# set :1 to community number
set set-community "65400:1"
set set-weight 200
unset set-local-preference
next
end
next
edit "rm-From-CR-01"
config rule
edit 1
set match-ip-address "prefix-Service-IPs"
set set-local-preference 50
next
edit 2
next
end
next
end
config router bgp
config neighbor-group
edit "ng-Servers"
set remote-as 65400
set update-source "loop-ac-bgp"
set route-reflector-client enable
set route-map-in "rm-Servers-In"
set soft-reconfiguration enable
# only servers the password can peer with fg
# set password <secret>
next
end
config neighbor-range
edit 1
# set 10.1.102.0 to local transit vlan for bgp neighbors
set prefix 10.1.102.0 255.255.255.0
set neighbor-group "ng-Servers"
next
end
config neighbor
# CR-01 (eBGP)
# set ip to appropriate tunnel ip for CR
edit "10.99.1.0"
set name "CR-01"
set remote-as 65000
set bfd enable
set send-community standard
set prefix-list-in "prefix-From-CR-01"
set prefix-list-out "prefix-To-CR-01"
set route-map-in "rm-From-CR-01"
next
end
endI also have adjusted RM_REDIST_CONNECTED route map to look like the following:
config router prefix-list
edit "prefix-Local-Svc"
config rule
edit 1
set prefix 10.254.254.0 255.255.255.0
unset ge
set le 32
next
end
next
end
config router route-map
edit "RM_REDIST_CONNECTED"
config rule
edit 1
set action deny
set match-interface "wan"
unset set-ip-prefsrc
next
edit 2
set action deny
set match-ip-address "prefix-Local-Svc"
unset set-ip-prefsrc
next
edit 3
set action deny
set match-interface "to-CR-01"
unset set-ip-prefsrc
next
edit 4
unset set-ip-prefsrc
next
end
next
endAnd IPv4 Redistribute Connect is set to the RM_REDIST_CONNECTED route-map:
config router bgp
...
config redistribute "connected"
set status enable
set route-map "RM_REDIST_CONNECTED"
end
...
endThis differs from my explicit route advertisements I had setup before, such as 10.1.0.0/16 and 10.2.0.0/16 for each site, I figured it would be much better doing it dynamically through route maps.
At this point, we should be peered:
site-a-rtr-01 # get router info bgp neighbors 10.99.1.0 | grep "BGP state"
BGP state = Established, up for 00:10:08cr-01 $ vtysh -c "show bgp sum"
IPv4 Unicast Summary (VRF default):
BGP router identifier 10.99.0.0, local AS number 65000 vrf-id 0
BGP table version 123
RIB entries 5, using 960 bytes of memory
Peers 2, using 1448 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.99.1.1 4 65400 5537 4887 0 0 0 00:10:55 1 2 SITE-A-RTR-01
...We’ll confirm BFD communication as well,
Packet sniffer:
# di snif pack to-CR-01 'host 10.99.1.0 and udp port 3784' 4 0 l
interfaces=[to-CR-01/]
filters=[host 10.99.1.0 and udp port 3784]
2026-09-30 19:50:16.788204 to-CR-01 -- 10.99.1.1.49213 -> 10.99.1.0.3784: udp 24
2026-09-30 19:50:16.931477 to-CR-01 -- 10.99.1.0.49154 -> 10.99.1.1.3784: udp 24
2026-09-30 19:50:17.078205 to-CR-01 -- 10.99.1.1.49213 -> 10.99.1.0.3784: udp 24
2026-09-30 19:50:17.186609 to-CR-01 -- 10.99.1.0.49154 -> 10.99.1.1.3784: udp 24
2026-09-30 19:50:17.398217 to-CR-01 -- 10.99.1.1.49213 -> 10.99.1.0.3784: udp 24
2026-09-30 19:50:17.432697 to-CR-01 -- 10.99.1.0.49154 -> 10.99.1.1.3784: udp 24
^C
12 packets received by filter
0 packets dropped by kernelWe can see the CR sending status checks to the router and the router responding over the tunnel.
vtysh -c "show bfd peers bri"
Session count: 2
SessionId LocalAddress PeerAddress Status
========= ============ =========== ======
3599774638 10.99.1.0 10.99.1.1 up
...The CR shows the router(s) up via BFD.
- Policy based routes:
Because we need traffic to return out to tunnel interface for WAN traffic that originated from the tunnel, we have to create a policy based route in the FortiGates to not run into an asymmetric routing issue (like sending out the actual WAN interface).
We’ll want to match any return traffic that matches our Service IPs and send it out the tunnel interface towards the CR (see next section for service IP setup)
That looks like the following:
config router policy
edit 0
set input-device "vlan102"
set src "10.98.0.0/255.255.255.0"
set dst "0.0.0.0/0.0.0.0"
# change to appropriate tunnel IP for CR-01
set gateway 10.99.1.0
set output-device "to-CR-01"
set comments "Service IPs return traffic via CR-01"
next
endNow this introduces an issue where local devices will also match this, so we will adjust it to look like this:
config router policy
edit 0
set input-device "vlan102"
set src "10.98.0.0/255.255.255.0"
set dst "10.0.0.0/255.0.0.0"
set dst-negate enable
# change to appropriate tunnel IP for CR-01
set gateway 10.99.1.0
set output-device "to-CR-01"
set comments "Service IPs return traffic via CR-01"
next
endHere I negate 10.0.0.0/8 because this matches the IP range we use for local communication, meaning any destination matching 10.0.0.0/8, we will not apply the routing policy and it will take the “best” route that is either statically or dynamically defined, including my existing “Hub and Spoke” architecture at SITE-A-RTR-01. This is especially relevant because of my split horizon DNS setup.
We also have to create firewall policies to allow the CR to talk to each server, and each server to talk to each server via their service IPs (cross-site)
Peering a server with FortiGate
Above we created Neighbor Groups on the FortiGates, which allows us to peer with it if a Neighbor range matches, which in this case the transit/local network for these servers are used.
While this is helpful, it introduces risks; any server within the transit/local network defined in the neighbor group can peer with the FortiGate, which means routes could be potentially hijacked. I’ve commented out the BGP password field for this example, but can easily be defined as a basic protection measure to prevent unauthorized/unintentional route advertisements back into the CR network.
With that said, it might be helpful to restrict communication to a specific port to a user or user group via iptables, I have not yet done this but will be experimenting with it to mitigate this even further.
- Service IP setup:
We will have to setup the service IP interface on the system which we want to peer with the FortiGate:
/etc/systemd/network/50-svc0.netdev:
[NetDev]
Name=svc0
Kind=dummyThe Name field is our interface name, and Kind field just indicates that this is a “virtual” interface, not connected to a physical NIC, almost like the lo loopback adapter.
/etc/systemd/network/50-svc0.network:
[Match]
Name=svc0
[Network]
Address=10.98.0.1/32
LinkLocalAddressing=noThis file is where we define our service IP we want to advertise. This is not managed by a IPAM (yet), later on I’ll write something up that is light weight to control this via an IPAM.
Ideally, all hosts would have a unique service IP unless it is meant to be an Anycast service IP, then we would use the same service IP on the hosts.
We need to apply the changes now:
sudo systemctl enable systemd-networkd
sudo systemctl restart systemd-networkdConfirm the interface was created and has our service IP assigned:
$ ip -br a show svc0
svc0 UNKNOWN 10.98.0.1/32UNKNOWN is normal. It is the link status, which a virtual interface does not have one.
Make sure traffic leaves our primary IP address, not the service IP:
$ ip route get 1.1.1.1
1.1.1.1 via 10.1.102.1 dev enp6s18 src 10.1.102.7 uid 0
cache This service IP setup depends on the system not being managed by NetworkManager!
- BIRD2 setup:
To peer with the FortiGate, we will be using BIRD2, which is a fairly light weight piece of software.
Installing it via APT:
sudo apt install bird2 -yThe bird configuration file is at /etc/bird/bird.conf, and it’s left pretty much default, with the following additions at the top of the file:
router id 10.255.0.1;
protocol direct services {
ipv4;
interface "svc0";
}
protocol bgp fortigate {
local as 65400;
neighbor 10.254.254.4 as 65400;
multihop;
# password "password here if used";
ipv4 {
import none;
export where proto = "services";
};
}The router ID shall be unique among all of the servers, we are just using 10.255.0.0/16 for this, this is not a routable address, just an identifier.
We define services as the svc0 interface and ensure that is the only export back to the FortiGate.
Ensure bird starts on boot, and restart it:
sudo systemctl enable bird
sudo systemctl restart birdAt this point, the server is peered with the FortiGate and advertising service IP back into the network.
Confirm on server using birdc:
$ sudo birdc show proto all fortigate
BIRD 2.14 ready.
Name Proto Table State Since Info
fortigate BGP --- up 19:38:21.431 Established
BGP state: Established
Neighbor address: 10.254.254.4
Neighbor AS: 65400
Local AS: 65400
Neighbor ID: 10.1.0.1
...Confirm on FortiGate:
# get router info bgp neighbors 10.1.102.7
VRF 0 neighbor table:
BGP neighbor is 10.1.102.7, remote AS 65400, local AS 65400, internal link
Member of peer-group "ng-Servers" for session parameters
BGP version 4, remote router ID 10.255.0.1
BGP state = Established, up for 00:00:08
...# get router info bgp neighbors 10.1.102.7 received-routes
VRF 0 BGP table version is 189, local router ID is 10.1.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight RouteTag Path
*>i10.98.0.1/32 10.1.102.7 100 0 0 i <0/0>
Total number of prefixes 1Confirming on the CR:
# show bgp ipv4 neighbor 10.99.1.1 routes
BGP table version is 153, local router ID is 10.99.0.0, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.98.0.1/32 10.99.1.1 0 65400 i
Displayed 1 routes and 3 total pathsThis has been repeated on another host at the other site as well, so two service IPs are currently being used.
Testing Service IP to Service IP connection
For service IP to service IP communication to work, especially for cross-site via the CR, we have to tell the server to use our service IP for service IP communication, not its primary IP.
We add the following to the BIRD configuration:
protocol static svc_route {
ipv4;
route 10.98.0.0/24 recursive 10.254.254.4;
}
protocol kernel {
learn;
ipv4 {
import all;
export filter {
if proto = "svc_route" then {
krt_prefsrc = 10.98.0.1;
accept;
}
reject;
};
};
}I have commented out the existing protocol kernel block, which it appears some versions of Ubuntu include a default configuration, like Ubuntu 24.04, but 26.04 did not.
Reload bird:
systemctl reload birdNow we should have an entry in our server’s FIB for 10.98.0.0/24:
$ ip route
default via 10.1.102.1 dev enp6s18 proto dhcp src 10.1.102.7 metric 100
10.1.102.0/24 dev enp6s18 proto kernel scope link src 10.1.102.7 metric 100
10.1.102.1 dev enp6s18 proto dhcp scope link src 10.1.102.7 metric 100
...
10.98.0.0/24 via 10.1.102.1 dev enp6s18 proto bird src 10.98.0.1 metric 32 # Egress/local traffic out with src as primary ip
$ ip route get 1.1.1.1
1.1.1.1 via 10.1.102.1 dev enp6s18 src 10.1.102.7 uid 0
cache
$ ip route get 10.1.102.1
10.1.102.1 dev enp6s18 src 10.1.102.7 uid 0
cache
# Service IP egress out with src as service IP
$ ip route get 10.98.0.2
10.98.0.2 via 10.1.102.1 dev enp6s18 src 10.98.0.1 uid 0
cache The final BIRD configuration looks a little like this:
router id <unique_router_id>;
protocol device {}
protocol direct services {
ipv4;
interface "svc0";
}
protocol bgp fortigate {
local as 65400;
neighbor 10.254.254.4 as 65400;
multihop;
ipv4 {
import none;
export where proto = "services";
};
}
protocol static svc_route {
ipv4;
route 10.98.0.0/24 recursive 10.254.254.4;
}
protocol kernel {
learn;
ipv4 {
import all;
export filter {
if proto = "svc_route" then {
krt_prefsrc = <service_ip>;
accept;
}
reject;
};
};
}I also created a policy on each of the FortiGates for hosts on certain networks to reach the remote end’s service IPs.
All of this has service IP to service IP communication been done through the tunnel that connects the sites directly together, we have to test what happens when the sites cannot communicate with each other directly anymore. Because there is communication between the sites right now, I cannot just take down the tunnel or shut down the bgp neighbor between them, so we will set a weight on the route map rm-From-CR-01 to something high, so the routes received from the CR are more preferred over the site to site tunnel/hub and spoke setup.
Before the change (from 10.98.0.1 - server-01):
$ traceroute 10.98.0.2
traceroute to 10.98.0.2 (10.98.0.2), 30 hops max, 60 byte packets
1 _gateway (10.1.102.1) 1.418 ms 1.123 ms 1.103 ms
2 10.11.0.2 (10.11.0.2) 61.944 ms 61.923 ms 61.905 ms
3 10.98.0.2 (10.98.0.2) 61.882 ms 69.413 ms 61.842 msOn both the sites:
config router route-map
edit "rm-From-CR-01"
config rule
edit 1
set set-weight 100
next
end
next
endexecute router clear bgp ip 10.99.1.2 soft inexecute router clear bgp ip 10.99.1.0 soft inAfter the change (from 10.98.0.1 - server-01):
$ traceroute 10.98.0.2
traceroute to 10.98.0.2 (10.98.0.2), 30 hops max, 60 byte packets
1 _gateway (10.1.102.1) 0.780 ms 0.766 ms 0.725 ms
2 10.99.1.0 (10.99.1.0) 44.619 ms 44.687 ms 44.650 ms
3 10.99.1.3 (10.99.1.3) 93.997 ms 93.685 ms 93.398 ms
4 10.98.0.2 (10.98.0.2) 93.334 ms 93.286 ms 93.255 msRevert the changes:
config router route-map
edit "rm-From-CR-01"
config rule
edit 1
unset set-weight
next
end
next
endexecute router clear bgp ip 10.99.1.2 soft inexecute router clear bgp ip 10.99.1.0 soft inThat path looks a little like the following (in green):
Conclusion
What we have built here is an Overlay network that dynamically adapts to failures and migrations at each site, as I didn’t want the project to stall out so I am chunking it up, and we will work on the following in the next few posts about this project:
- Asymmetric routing from Local IPs to VPS Public WAN IP
- Automated BIRD2 deployment with Ansible
- Deploying real services behind the CR network
- Using Keepalived to load balance connections across resources behind the CR network
- Hosting authoritative DNS name servers behind the CR network
- Security implications and mitigations