{"id":682,"date":"2025-11-26T16:26:28","date_gmt":"2025-11-26T16:26:28","guid":{"rendered":"http:\/\/blog.miguelsarmiento.com\/?p=682"},"modified":"2025-11-26T16:31:46","modified_gmt":"2025-11-26T16:31:46","slug":"pve-sdn-vyos-integration-oh-my","status":"publish","type":"post","link":"https:\/\/blog.miguelsarmiento.com\/?p=682","title":{"rendered":"PVE SDN VyOS Integration Oh My!"},"content":{"rendered":"

Well after the series regarding integration with a Cisco 9k, I thought that we also needed something regarding Open-Source.<\/p>\n

Well VyOS does not get any more open source than that, we will integrate a PVE cluster with a VyOS as a leaf. In addition we will use VyOS as an eBGP peer to access external clients. Read on!<\/p>\n

Setup<\/h1>\n

We will once again use a PVE cluster, the network diagram should be by now familiar to the reader.<\/p>\n

\"\"<\/a>
Fig, 1. Network Diagram.<\/figcaption><\/figure>\n

We run two VyOS (VyOS 1.5-stream-2025-Q2) appliances vyos-2 and router4.\u00a0 We configure a cluster with a manual fabric . All nodes are accessible via ssh for management (via the MGMT cloud).<\/p>\n

VyOS-2 is acting as a leaf, while the concept is the same as with the Cisco device, the semantics of the vxlan and evpn commands are different from what you are used to if you use Cisco.<\/p>\n

Setup<\/h1>\n

PVE<\/h2>\n

Create a PVE cluster. Refer to the NXOS integration posts for more information.<\/p>\n

1) Three nodes: 192,168.1.1, .2 and .3 (refer to the network diagram Fig. 1.)<\/p>\n

2) Loopback interfaces 10.10.10.1, ,2 and .3<\/p>\n

3) An EVPN controller (DC-1). Add 192.168.1.10 the IP address of the VyOS leaf.<\/p>\n

a) Two zones (L3 VRFs) test1 and test2.<\/p>\n

b) Each containing a VNet (vnet1 and vnet2).<\/p>\n

c) 10.100.1.0\/24 and 10.200.1.0\/24 for each.<\/p>\n

The zones configured are VRFs, you can see this by inspecting the SDN configuration file (“\/etc\/network\/interfaces.d\/sdn)” or using vtysh:<\/p>\n

pve-3# sh bgp l2vpn evpn vni\r\nAdvertise Gateway Macip: Disabled\r\nAdvertise SVI Macip: Disabled\r\nAdvertise All VNI flag: Enabled\r\nBUM flooding: Head-end replication\r\nVXLAN flooding: Enabled\r\nNumber of L2 VNIs: 2\r\nNumber of L3 VNIs: 2\r\nFlags: * - Kernel\r\n VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF\r\n* 51000 L2 192.168.1.3:2 65000:51000 65000:51000 vrf_test1\r\n* 61000 L2 192.168.1.3:4 65000:61000 65000:61000 vrf_test2\r\n* 50000 L3 192.168.1.3:6 65000:50000 65000:50000 vrf_test1\r\n* 60000 L3 192.168.1.3:7 65000:60000 65000:60000 vrf_test2<\/pre>\n
As you can see, the vni assigned to the Vnets are L2 and the ones assigned to the zones are L3, all under the VRF zone-name.<\/p>\n
This is important in particular when configuring “l2vpn evpn” on the VyOS router.<\/p>\n
pve-3# sh bgp vrf\r\nType Id routerId #PeersCfg #PeersEstb Name\r\n L3-VNI RouterMAC Interface\r\nDFLT 0 192.168.1.3 4 4 default\r\n 0 00:00:00:00:00:00 unknown\r\n VRF 10 192.168.1.3 0 0 vrf_test1\r\n 50000 22:11:ed:be:a5:7e vrfbr_test1\r\n VRF 15 192.168.1.3 0 0 vrf_test2\r\n 60000 76:fe:bf:5b:ef:86 vrfbr_test2<\/pre>\n
VyOS<\/h2>\n
Below are the relevant parts of the configuration:<\/p>\n
set interfaces dummy dum0 address '10.10.10.10\/32'\r\nset interfaces ethernet eth1 address '192.168.1.10\/24'\r\nset interfaces vxlan vxlan510 mtu '1500'\r\nset interfaces vxlan vxlan510 parameters nolearning\r\nset interfaces vxlan vxlan510 port '4789'\r\nset interfaces vxlan vxlan510 source-address '192.168.1.10'\r\nset interfaces vxlan vxlan510 vni '51000'\r\nset interfaces vxlan vxlan610 mtu '1500'\r\nset interfaces vxlan vxlan610 parameters nolearning\r\nset interfaces vxlan vxlan610 port '4789'\r\nset interfaces vxlan vxlan610 source-address '192.168.1.10'<\/pre>\n
Above sets a dummy interface such that we can add it to the ospf process in the cluster. In addition it creates two vxlan interfaces using the VyOS IP address as source.<\/p>\n
Take not that here we use the vni defined for the vnets created previously.<\/p>\n
We setup OSPF:<\/p>\n
set protocols ospf area 0\r\nset protocols ospf interface dum0 area '0'\r\nset protocols ospf interface eth1 area '0'\r\nset protocols ospf log-adjacency-changes detail\r\nset protocols ospf parameters abr-type 'cisco'<\/pre>\n
Now we setup BGP:<\/p>\n
set protocols bgp address-family l2vpn-evpn advertise ipv4 unicast\r\nset protocols bgp address-family l2vpn-evpn advertise-all-vni\r\nset protocols bgp neighbor 192.168.1.1 peer-group 'ibgp'\r\nset protocols bgp neighbor 192.168.1.2 peer-group 'ibgp'\r\nset protocols bgp neighbor 192.168.1.3 peer-group 'ibgp'\r\nset protocols bgp parameters log-neighbor-changes\r\nset protocols bgp parameters router-id '10.10.10.10'\r\nset protocols bgp peer-group ibgp address-family l2vpn-evpn\r\nset protocols bgp peer-group ibgp remote-as '65000'\r\nset protocols bgp system-as '65000'<\/pre>\n
At this point you will see routes, since the vxlan interfaces will be up however, you will not be able to ping anything. You should see peering in the cluster.<\/p>\n
Now we setup VRFs to isolate each tenant.<\/p>\n
First create the bridges and add the interfaces.<\/p>\n
set interfaces bridge br500 address '10.100.1.254\/24'\r\nset interfaces bridge br500 description 'customer vfr_test1'\r\nset interfaces bridge br500 member interface eth2\r\nset interfaces bridge br500 member interface vxlan510\r\nset interfaces bridge br500 vrf 'vrf_test1'\r\nset interfaces bridge br600 address '10.200.1.254\/24'\r\nset interfaces bridge br600 description 'customer vfr_test2'\r\nset interfaces bridge br600 member interface eth3\r\nset interfaces bridge br600 member interface vxlan610\r\nset interfaces bridge br600 vrf 'vrf_test2'<\/pre>\n
Notice we use vrtf_test1 and vrf_test2 for the VRFs.<\/p>\n
set vrf name vrf_test1 protocols bgp address-family ipv4-unicast redistribute connected\r\nset vrf name vrf_test1 protocols bgp address-family l2vpn-evpn advertise ipv4 unicast\r\nset vrf name vrf_test1 protocols bgp peer-group ibgp address-family l2vpn-evpn\r\nset vrf name vrf_test1 protocols bgp peer-group ibgp remote-as '65000'\r\nset vrf name vrf_test1 protocols bgp system-as '65000'\r\nset vrf name vrf_test1 table '500'\r\nset vrf name vrf_test1 vni '50000'\r\nset vrf name vrf_test2 protocols bgp address-family ipv4-unicast redistribute connected\r\nset vrf name vrf_test2 protocols bgp address-family l2vpn-evpn advertise ipv4 unicast\r\nset vrf name vrf_test2 protocols bgp peer-group ibgp address-family l2vpn-evpn\r\nset vrf name vrf_test2 protocols bgp peer-group ibgp remote-as '65000'\r\nset vrf name vrf_test2 protocols bgp system-as '65000'\r\nset vrf name vrf_test2 table '600'\r\nset vrf name vrf_test2 vni '60000'<\/pre>\n
Notice that we map the VNIs of the zones we pointed out earlier in the cluster rather than the ones used in the VXLAN configuration.<\/p>\n
eBGP<\/h2>\n
This is the whole configuration. Refer to the network diagram. Create an eBGP controller (use 192.168.1.20 for the peer).<\/p>\n
set interfaces ethernet eth0 address 'dhcp'\r\nset interfaces ethernet eth0 hw-id '50:01:00:04:00:00'\r\nset interfaces ethernet eth1 address '192.168.1.20\/24'\r\nset interfaces ethernet eth1 hw-id '50:01:00:04:00:01'\r\nset interfaces ethernet eth2 address '192.168.0.9\/30'\r\nset interfaces ethernet eth2 hw-id '50:01:00:04:00:02'\r\nset interfaces ethernet eth3 address '192.168.0.13\/30'\r\nset interfaces ethernet eth3 hw-id '50:01:00:04:00:03'\r\nset interfaces ethernet eth4 hw-id '50:01:00:04:00:04'\r\nset interfaces ethernet eth5 hw-id '50:01:00:04:00:05'\r\nset interfaces ethernet eth6 hw-id '50:01:00:04:00:07'\r\nset interfaces ethernet eth7 hw-id '50:01:00:04:00:06'\r\nset interfaces loopback lo address '10.10.10.4\/32'\r\nset policy prefix-list to-tna rule 10 action 'permit'\r\nset policy prefix-list to-tna rule 10 prefix '10.100.1.0\/24'\r\nset policy prefix-list to-tnb rule 10 action 'permit'\r\nset policy prefix-list to-tnb rule 10 prefix '10.200.1.0\/24'\r\nset protocols bgp address-family ipv4-unicast\r\nset protocols bgp neighbor 192.168.0.10 address-family ipv4-unicast prefix-list export 'to-tna'\r\nset protocols bgp neighbor 192.168.0.10 address-family ipv4-unicast soft-reconfiguration inbound\r\nset protocols bgp neighbor 192.168.0.10 remote-as '64000'\r\nset protocols bgp neighbor 192.168.0.14 address-family ipv4-unicast prefix-list export 'to-tnb'\r\nset protocols bgp neighbor 192.168.0.14 address-family ipv4-unicast soft-reconfiguration inbound\r\nset protocols bgp neighbor 192.168.0.14 remote-as '63000'\r\nset protocols bgp neighbor 192.168.1.3 address-family ipv4-unicast soft-reconfiguration inbound\r\nset protocols bgp neighbor 192.168.1.3 remote-as '65000'\r\nset protocols bgp parameters router-id '10.10.10.4'\r\nset protocols bgp system-as '65100'\r\nset protocols ospf area 0\r\nset protocols ospf interface eth1 area '0'\r\nset protocols ospf interface lo area '0'\r\nset protocols ospf log-adjacency-changes\r\nset protocols ospf parameters router-id '10.10.10.4'\r\nset service ntp allow-client address '127.0.0.0\/8'\r\nset service ntp allow-client address '169.254.0.0\/16'\r\nset service ntp allow-client address '10.0.0.0\/8'\r\nset service ntp allow-client address '172.16.0.0\/12'\r\nset service ntp allow-client address '192.168.0.0\/16'\r\nset service ntp allow-client address '::1\/128'\r\nset service ntp allow-client address 'fe80::\/10'\r\nset service ntp allow-client address 'fc00::\/7'\r\nset service ntp server time1.vyos.net\r\nset service ntp server time2.vyos.net\r\nset service ntp server time3.vyos.net\r\nset service ssh\r\nset system config-management commit-revisions '100'\r\nset system console device ttyS0 speed '115200'\r\nset system host-name 'vyos-1'\r\nset system login user vyos authentication encrypted-password '$6$rounds=656000$EWLnOKfePeZbAhel$mps4vfL4PmhLVegxMImNQYIqWMOWG8Ad5j2buAy9EzhaRVTnj1g79pOwT41EzPEBs\/exVGm9spc9TDTfsECih1'\r\nset system login user vyos authentication plaintext-password ''\r\nset system option reboot-on-upgrade-failure '5'\r\nset system syslog local facility all level 'info'\r\nset system syslog local facility local7 level 'debug'<\/pre>\n
Straight forward, we do not use VRFs to separate the tenants. I was lazy after configuring the leaf. However, we achieved the same by creating a few IP prefix lists and route maps and only announce to the tenants the corresponding networks. A good exercise in creating such objects.<\/p>\n
It should work off the bat thus I will not go into details.<\/p>\n
Testing EVPN<\/h1>\n
On PX3.<\/p>\n
pve-3# sh bgp l2vpn evpn summary\r\nBGP router identifier 192.168.1.3, local AS number 65000 VRF default vrf-id 0\r\nBGP table version 0\r\nRIB entries 39, using 4992 bytes of memory\r\nPeers 3, using 70 KiB of memory\r\nPeer groups 2, using 128 bytes of memory\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd PfxSnt Desc\r\npve-1(192.168.1.1) 4 65000 81534 81214 338 0 0 2d19h40m 2 6 FRRouting\/10.3.1\r\npve-2(192.168.1.2) 4 65000 81502 81216 338 0 0 2d19h39m 2 6 FRRouting\/10.3.1\r\nvyos(192.168.1.10) 4 65000 23841 23841 338 0 0 19:51:50 4 6 FRRouting\/9.1.3<\/pre>\n
We can see routes from the leaf.<\/p>\n
pve-3# sh bgp l2vpn evpn neighbors 192.168.1.10 routes\r\nBGP table version is 4, local router ID is 192.168.1.3\r\nStatus codes: s suppressed, d damped, h history, * valid, > best, i - internal\r\nOrigin codes: i - IGP, e - EGP, ? - incomplete\r\nEVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]\r\nEVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]\r\nEVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]\r\nEVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]\r\nEVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]\r\n\r\nNetwork Next Hop Metric LocPrf Weight Path\r\nRoute Distinguisher: 10.10.10.10:3\r\n *>i [2]:[0]:[48]:[c2:07:b8:ee:00:00]\r\n 192.168.1.10(vyos)\r\n 100 0 i\r\n RT:65000:51000 ET:8\r\n *>i [3]:[0]:[32]:[192.168.1.10]\r\n 192.168.1.10(vyos)\r\n 100 0 i\r\n RT:65000:51000 ET:8\r\nRoute Distinguisher: 10.10.10.10:5\r\n *>i [2]:[0]:[48]:[c2:08:b9:dc:00:00]\r\n 192.168.1.10(vyos)\r\n 100 0 i\r\n RT:65000:61000 ET:8\r\n *>i [3]:[0]:[32]:[192.168.1.10]\r\n 192.168.1.10(vyos)\r\n 100 0 i\r\n RT:65000:61000 ET:8\r\n\r\nDisplayed 4 out of 14 total prefixes<\/pre>\n
On the VyOS leaf.<\/p>\n
vyos-2:~$ sh ip bgp vrf vrf_test1\r\nBGP table version is 1, local router ID is 10.100.1.254, vrf id 16\r\nDefault local pref 100, local AS 65000\r\nStatus codes: s suppressed, d damped, h history, * valid, > best, = multipath,\r\n i internal, r RIB-failure, S Stale, R Removed\r\nNexthop codes: @NNN nexthop's vrf id, < announce-nh-self\r\nOrigin codes: i - IGP, e - EGP, ? - incomplete\r\nRPKI validation codes: V valid, I invalid, N Not found\r\n\r\nNetwork Next Hop Metric LocPrf Weight Path\r\n *> 10.100.1.0\/24 0.0.0.0 0 32768 ?<\/pre>\n
And the BGP summary.<\/p>\n
vyos-2:~$ sh bgp l2vpn evpn summary\r\nBGP router identifier 10.10.10.10, local AS number 65000 vrf-id 0\r\nBGP table version 0\r\nRIB entries 27, using 2592 bytes of memory\r\nPeers 3, using 60 KiB of memory\r\nPeer groups 1, using 64 bytes of memory\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd PfxSnt Desc\r\n192.168.1.1 4 65000 23917 23771 154 0 0 19:48:19 6 4 FRRouting\/10.3.1\r\n192.168.1.2 4 65000 23884 23771 154 0 0 19:48:19 6 4 FRRouting\/10.3.1\r\n192.168.1.3 4 65000 24034 24034 154 0 0 20:01:28 6 4 FRRouting\/10.3.1<\/pre>\n
Of course you can do more probing, go crazy.<\/p>\n
Now on PC7 you will be able to ping one of the containers you created before, in this case 10.100.1.10.<\/p>\n
pc7#sh ip int bri\r\nInterface IP-Address OK? Method Status Protocol\r\nFastEthernet0\/0 10.100.1.100 YES NVRAM up up \r\nFastEthernet0\/1 unassigned YES NVRAM administratively down down \r\npc7#ping 10.100.1.10\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.100.1.10, timeout is 2 seconds:\r\n.!!!!\r\nSuccess rate is 80 percent (4\/5), round-trip min\/avg\/max = 8\/23\/36 ms<\/pre>\n
There you have it. VyOS acting as a leaf. You could then create a bridge to another device, a Cisco switch for example and trunk networks to it. Now you have a ToR switch (not VXLAN capable) able to connect devices between the cluster and your VLANS. Perhaps the theme for another post.<\/p>\n
Next, I could do a FRR device (a Ubuntu server for example) as a leaf to a PVE cluster. However since PVE uses Debian you can look at the configuration it creates and have an idea how to do it for another Linux server.<\/p>\n
Happy networking,<\/p>\n
Ciao.<\/p>\n","protected":false},"excerpt":{"rendered":"
Well after the series regarding integration with a Cisco 9k, I thought that we also needed something regarding Open-Source. Well VyOS does not get any more open source than that, we will integrate a PVE cluster with a VyOS as a leaf. In addition we will use VyOS as an eBGP peer to access external … <\/p>\n
Continue reading “PVE SDN VyOS Integration Oh My!”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-682","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/682","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=682"}],"version-history":[{"count":24,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/682\/revisions"}],"predecessor-version":[{"id":707,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/682\/revisions\/707"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=682"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=682"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=682"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}