Well after the series regarding integration with a Cisco 9k, I thought that we also needed something regarding Open-Source.
Well VyOS does not get any more open source than that, we will integrate a PVE cluster with a VyOS as a leaf. In addition we will use VyOS as an eBGP peer to access external clients. Read on!
Setup
We will once again use a PVE cluster, the network diagram should be by now familiar to the reader.
We run two VyOS (VyOS 1.5-stream-2025-Q2) appliances vyos-2 and router4. We configure a cluster with a manual fabric . All nodes are accessible via ssh for management (via the MGMT cloud).
VyOS-2 is acting as a leaf, while the concept is the same as with the Cisco device, the semantics of the vxlan and evpn commands are different from what you are used to if you use Cisco.
Setup
PVE
Create a PVE cluster. Refer to the NXOS integration posts for more information.
1) Three nodes: 192,168.1.1, .2 and .3 (refer to the network diagram Fig. 1.)
2) Loopback interfaces 10.10.10.1, ,2 and .3
3) An EVPN controller (DC-1). Add 192.168.1.10 the IP address of the VyOS leaf.
a) Two zones (L3 VRFs) test1 and test2.
b) Each containing a VNet (vnet1 and vnet2).
c) 10.100.1.0/24 and 10.200.1.0/24 for each.
The zones configured are VRFs, you can see this by inspecting the SDN configuration file (“/etc/network/interfaces.d/sdn)” or using vtysh:
pve-3# sh bgp l2vpn evpn vni Advertise Gateway Macip: Disabled Advertise SVI Macip: Disabled Advertise All VNI flag: Enabled BUM flooding: Head-end replication VXLAN flooding: Enabled Number of L2 VNIs: 2 Number of L3 VNIs: 2 Flags: * - Kernel VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF * 51000 L2 192.168.1.3:2 65000:51000 65000:51000 vrf_test1 * 61000 L2 192.168.1.3:4 65000:61000 65000:61000 vrf_test2 * 50000 L3 192.168.1.3:6 65000:50000 65000:50000 vrf_test1 * 60000 L3 192.168.1.3:7 65000:60000 65000:60000 vrf_test2
As you can see, the vni assigned to the Vnets are L2 and the ones assigned to the zones are L3, all under the VRF zone-name.
This is important in particular when configuring “l2vpn evpn” on the VyOS router.
pve-3# sh bgp vrf Type Id routerId #PeersCfg #PeersEstb Name L3-VNI RouterMAC Interface DFLT 0 192.168.1.3 4 4 default 0 00:00:00:00:00:00 unknown VRF 10 192.168.1.3 0 0 vrf_test1 50000 22:11:ed:be:a5:7e vrfbr_test1 VRF 15 192.168.1.3 0 0 vrf_test2 60000 76:fe:bf:5b:ef:86 vrfbr_test2
VyOS
Below are the relevant parts of the configuration:
set interfaces dummy dum0 address '10.10.10.10/32' set interfaces ethernet eth1 address '192.168.1.10/24' set interfaces vxlan vxlan510 mtu '1500' set interfaces vxlan vxlan510 parameters nolearning set interfaces vxlan vxlan510 port '4789' set interfaces vxlan vxlan510 source-address '192.168.1.10' set interfaces vxlan vxlan510 vni '51000' set interfaces vxlan vxlan610 mtu '1500' set interfaces vxlan vxlan610 parameters nolearning set interfaces vxlan vxlan610 port '4789' set interfaces vxlan vxlan610 source-address '192.168.1.10'
Above sets a dummy interface such that we can add it to the ospf process in the cluster. In addition it creates two vxlan interfaces using the VyOS IP address as source.
Take not that here we use the vni defined for the vnets created previously.
We setup OSPF:
set protocols ospf area 0 set protocols ospf interface dum0 area '0' set protocols ospf interface eth1 area '0' set protocols ospf log-adjacency-changes detail set protocols ospf parameters abr-type 'cisco'
Now we setup BGP:
set protocols bgp address-family l2vpn-evpn advertise ipv4 unicast set protocols bgp address-family l2vpn-evpn advertise-all-vni set protocols bgp neighbor 192.168.1.1 peer-group 'ibgp' set protocols bgp neighbor 192.168.1.2 peer-group 'ibgp' set protocols bgp neighbor 192.168.1.3 peer-group 'ibgp' set protocols bgp parameters log-neighbor-changes set protocols bgp parameters router-id '10.10.10.10' set protocols bgp peer-group ibgp address-family l2vpn-evpn set protocols bgp peer-group ibgp remote-as '65000' set protocols bgp system-as '65000'
At this point you will see routes, since the vxlan interfaces will be up however, you will not be able to ping anything. You should see peering in the cluster.
Now we setup VRFs to isolate each tenant.
First create the bridges and add the interfaces.
set interfaces bridge br500 address '10.100.1.254/24' set interfaces bridge br500 description 'customer vfr_test1' set interfaces bridge br500 member interface eth2 set interfaces bridge br500 member interface vxlan510 set interfaces bridge br500 vrf 'vrf_test1' set interfaces bridge br600 address '10.200.1.254/24' set interfaces bridge br600 description 'customer vfr_test2' set interfaces bridge br600 member interface eth3 set interfaces bridge br600 member interface vxlan610 set interfaces bridge br600 vrf 'vrf_test2'
Notice we use vrtf_test1 and vrf_test2 for the VRFs.
set vrf name vrf_test1 protocols bgp address-family ipv4-unicast redistribute connected set vrf name vrf_test1 protocols bgp address-family l2vpn-evpn advertise ipv4 unicast set vrf name vrf_test1 protocols bgp peer-group ibgp address-family l2vpn-evpn set vrf name vrf_test1 protocols bgp peer-group ibgp remote-as '65000' set vrf name vrf_test1 protocols bgp system-as '65000' set vrf name vrf_test1 table '500' set vrf name vrf_test1 vni '50000' set vrf name vrf_test2 protocols bgp address-family ipv4-unicast redistribute connected set vrf name vrf_test2 protocols bgp address-family l2vpn-evpn advertise ipv4 unicast set vrf name vrf_test2 protocols bgp peer-group ibgp address-family l2vpn-evpn set vrf name vrf_test2 protocols bgp peer-group ibgp remote-as '65000' set vrf name vrf_test2 protocols bgp system-as '65000' set vrf name vrf_test2 table '600' set vrf name vrf_test2 vni '60000'
Notice that we map the VNIs of the zones we pointed out earlier in the cluster rather than the ones used in the VXLAN configuration.
eBGP
This is the whole configuration. Refer to the network diagram. Create an eBGP controller (use 192.168.1.20 for the peer).
set interfaces ethernet eth0 address 'dhcp' set interfaces ethernet eth0 hw-id '50:01:00:04:00:00' set interfaces ethernet eth1 address '192.168.1.20/24' set interfaces ethernet eth1 hw-id '50:01:00:04:00:01' set interfaces ethernet eth2 address '192.168.0.9/30' set interfaces ethernet eth2 hw-id '50:01:00:04:00:02' set interfaces ethernet eth3 address '192.168.0.13/30' set interfaces ethernet eth3 hw-id '50:01:00:04:00:03' set interfaces ethernet eth4 hw-id '50:01:00:04:00:04' set interfaces ethernet eth5 hw-id '50:01:00:04:00:05' set interfaces ethernet eth6 hw-id '50:01:00:04:00:07' set interfaces ethernet eth7 hw-id '50:01:00:04:00:06' set interfaces loopback lo address '10.10.10.4/32' set policy prefix-list to-tna rule 10 action 'permit' set policy prefix-list to-tna rule 10 prefix '10.100.1.0/24' set policy prefix-list to-tnb rule 10 action 'permit' set policy prefix-list to-tnb rule 10 prefix '10.200.1.0/24' set protocols bgp address-family ipv4-unicast set protocols bgp neighbor 192.168.0.10 address-family ipv4-unicast prefix-list export 'to-tna' set protocols bgp neighbor 192.168.0.10 address-family ipv4-unicast soft-reconfiguration inbound set protocols bgp neighbor 192.168.0.10 remote-as '64000' set protocols bgp neighbor 192.168.0.14 address-family ipv4-unicast prefix-list export 'to-tnb' set protocols bgp neighbor 192.168.0.14 address-family ipv4-unicast soft-reconfiguration inbound set protocols bgp neighbor 192.168.0.14 remote-as '63000' set protocols bgp neighbor 192.168.1.3 address-family ipv4-unicast soft-reconfiguration inbound set protocols bgp neighbor 192.168.1.3 remote-as '65000' set protocols bgp parameters router-id '10.10.10.4' set protocols bgp system-as '65100' set protocols ospf area 0 set protocols ospf interface eth1 area '0' set protocols ospf interface lo area '0' set protocols ospf log-adjacency-changes set protocols ospf parameters router-id '10.10.10.4' set service ntp allow-client address '127.0.0.0/8' set service ntp allow-client address '169.254.0.0/16' set service ntp allow-client address '10.0.0.0/8' set service ntp allow-client address '172.16.0.0/12' set service ntp allow-client address '192.168.0.0/16' set service ntp allow-client address '::1/128' set service ntp allow-client address 'fe80::/10' set service ntp allow-client address 'fc00::/7' set service ntp server time1.vyos.net set service ntp server time2.vyos.net set service ntp server time3.vyos.net set service ssh set system config-management commit-revisions '100' set system console device ttyS0 speed '115200' set system host-name 'vyos-1' set system login user vyos authentication encrypted-password '$6$rounds=656000$EWLnOKfePeZbAhel$mps4vfL4PmhLVegxMImNQYIqWMOWG8Ad5j2buAy9EzhaRVTnj1g79pOwT41EzPEBs/exVGm9spc9TDTfsECih1' set system login user vyos authentication plaintext-password '' set system option reboot-on-upgrade-failure '5' set system syslog local facility all level 'info' set system syslog local facility local7 level 'debug'
Straight forward, we do not use VRFs to separate the tenants. I was lazy after configuring the leaf. However, we achieved the same by creating a few IP prefix lists and route maps and only announce to the tenants the corresponding networks. A good exercise in creating such objects.
It should work off the bat thus I will not go into details.
Testing EVPN
On PX3.
pve-3# sh bgp l2vpn evpn summary BGP router identifier 192.168.1.3, local AS number 65000 VRF default vrf-id 0 BGP table version 0 RIB entries 39, using 4992 bytes of memory Peers 3, using 70 KiB of memory Peer groups 2, using 128 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc pve-1(192.168.1.1) 4 65000 81534 81214 338 0 0 2d19h40m 2 6 FRRouting/10.3.1 pve-2(192.168.1.2) 4 65000 81502 81216 338 0 0 2d19h39m 2 6 FRRouting/10.3.1 vyos(192.168.1.10) 4 65000 23841 23841 338 0 0 19:51:50 4 6 FRRouting/9.1.3
We can see routes from the leaf.
pve-3# sh bgp l2vpn evpn neighbors 192.168.1.10 routes BGP table version is 4, local router ID is 192.168.1.3 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id] EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP] Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 10.10.10.10:3 *>i [2]:[0]:[48]:[c2:07:b8:ee:00:00] 192.168.1.10(vyos) 100 0 i RT:65000:51000 ET:8 *>i [3]:[0]:[32]:[192.168.1.10] 192.168.1.10(vyos) 100 0 i RT:65000:51000 ET:8 Route Distinguisher: 10.10.10.10:5 *>i [2]:[0]:[48]:[c2:08:b9:dc:00:00] 192.168.1.10(vyos) 100 0 i RT:65000:61000 ET:8 *>i [3]:[0]:[32]:[192.168.1.10] 192.168.1.10(vyos) 100 0 i RT:65000:61000 ET:8 Displayed 4 out of 14 total prefixes
On the VyOS leaf.
vyos-2:~$ sh ip bgp vrf vrf_test1 BGP table version is 1, local router ID is 10.100.1.254, vrf id 16 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 10.100.1.0/24 0.0.0.0 0 32768 ?
And the BGP summary.
vyos-2:~$ sh bgp l2vpn evpn summary BGP router identifier 10.10.10.10, local AS number 65000 vrf-id 0 BGP table version 0 RIB entries 27, using 2592 bytes of memory Peers 3, using 60 KiB of memory Peer groups 1, using 64 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 192.168.1.1 4 65000 23917 23771 154 0 0 19:48:19 6 4 FRRouting/10.3.1 192.168.1.2 4 65000 23884 23771 154 0 0 19:48:19 6 4 FRRouting/10.3.1 192.168.1.3 4 65000 24034 24034 154 0 0 20:01:28 6 4 FRRouting/10.3.1
Of course you can do more probing, go crazy.
Now on PC7 you will be able to ping one of the containers you created before, in this case 10.100.1.10.
pc7#sh ip int bri Interface IP-Address OK? Method Status Protocol FastEthernet0/0 10.100.1.100 YES NVRAM up up FastEthernet0/1 unassigned YES NVRAM administratively down down pc7#ping 10.100.1.10 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.100.1.10, timeout is 2 seconds: .!!!! Success rate is 80 percent (4/5), round-trip min/avg/max = 8/23/36 ms
There you have it. VyOS acting as a leaf. You could then create a bridge to another device, a Cisco switch for example and trunk networks to it. Now you have a ToR switch (not VXLAN capable) able to connect devices between the cluster and your VLANS. Perhaps the theme for another post.
Next, I could do a FRR device (a Ubuntu server for example) as a leaf to a PVE cluster. However since PVE uses Debian you can look at the configuration it creates and have an idea how to do it for another Linux server.
Happy networking,
Ciao.