{"id":603,"date":"2025-11-21T20:24:20","date_gmt":"2025-11-21T20:24:20","guid":{"rendered":"http:\/\/blog.miguelsarmiento.com\/?p=603"},"modified":"2025-11-21T20:33:53","modified_gmt":"2025-11-21T20:33:53","slug":"pve-vxlan-nxos-integration-continued-oh-my","status":"publish","type":"post","link":"https:\/\/blog.miguelsarmiento.com\/?p=603","title":{"rendered":"PVE VXLAN NXOS Integration Continued Oh My!"},"content":{"rendered":"

Introduction<\/h1>\n

In part 2 we will continue with integration of a PVE cluster with a Cisco 9kv running NXOS.<\/p>\n

Figure 1. shows the network diagram.<\/p>\n

\"\"<\/a>
Fig. 1. Network Diagram<\/figcaption><\/figure>\n

\u00a0We are using the 9000v as an eBGP peer for the PVE cluster. Any device that supports BGP can replace the Cisco 9k.<\/p>\n

SETUP<\/h1>\n

We have a PVE cluster. However, we are now connecting to two tenants via eBGP and we are accessing them via eBGP peering, This neatly separates the functions of the cluster (internal machines) and access to external clients.<\/p>\n

A few notes:<\/span><\/p>\n

1) We have two zones configured in the cluster.<\/p>\n

a) Zone 1 for tenant A (TNA), 10.10.100.0\/24<\/p>\n

b) Zone 2 for tenant B (TNB), 10.10.200.0\/24<\/p>\n

c) Make sure any container in each zone configuration can ping each other.<\/p>\n

2) Once again we use a manual fabric. This time though we configure OSPF so the cluster and the Cisco are reachable sans static routes.<\/p>\n

Configuration<\/h1>\n

PVE<\/h2>\n

Configure a standard PVE cluster.<\/p>\n

1) Configure the DATA segment. Add a new vmbr1 bridge on each node and give it the corresponding IP address.<\/span><\/p>\n

2) Create an evpn controller (DC1) and add each IP configured in (1) to it: 192.168.1.41,42,43. Use ASN 65000 the default.<\/span><\/p>\n

3) Create an eBGP controller and add 192.168.1.10 (IP of the Cisco). Use PX3 as the source of the peering.<\/span><\/p>\n

4) Create two zones:<\/p>\n

a) v100 and v200. Put them under the controller that you just created.<\/span><\/p>\n

b) Assign vxlan:10000 to v100, vxlan:20000 to v200.<\/p>\n

c) Create two VNETS:<\/p>\n

1. TEST1: vni 11000 on v100<\/p>\n

2. TEST2: vni 21000 on v200<\/p>\n

d) Assign 10.10.100.0\/24 (GW 10.10.100.1) to TEST1, 10.10.200.0\/24 (GW 10.10.200.1) to TEST2.<\/p>\n

e) Configure SNAT if you wish. IMPORTANT: make PX3 the exit node for the zones. This will guarantee that BGP routes are advertised correctly.<\/p>\n

Create file “snd-local” and put it under \u201c\/etc\/network\/interface.d\u201d and add the following to it:<\/p>\n

auto dummy_DC1<\/b>\r\niface dummy_DC1 inet static<\/b>\r\n        address 10.10.10.3\/32<\/b>\r\n        link-type dummy<\/b>\r\n        ip-forward 1<\/b>\r\n\r\nauto vmbr1<\/b>\r\niface vmbr1 inet static<\/b>\r\n        address 192.168.1.43\/24<\/b>\r\n        ip-forward 1<\/b><\/pre>\n
Replace above on all the nodes making sure to enter the correct IP addresses for each node. Here DC1 refers to the controller name created earlier.<\/p>\n
Create “frr.conf.local” in the \u201c\/etc\/frr\/\u201d directory:<\/p>\n
router ospf<\/b><\/span><\/span>\r\n ospf router-id 10.10.10.3<\/b><\/span><\/span>\r\nexit<\/b><\/span><\/span>\r\n!<\/b><\/span><\/span>\r\ninterface dummy_DC1<\/b><\/span><\/span>\r\n ip ospf area 0<\/b><\/span><\/span>\r\n ip ospf passive<\/b><\/span><\/span>\r\nexit<\/b><\/span><\/span>\r\n!<\/b><\/span><\/span>\r\ninterface vmbr1<\/b><\/span><\/span>\r\n ip ospf area 0<\/b><\/span><\/span>\r\nexit<\/b><\/span><\/span><\/pre>\n
Do the same on each node and enter the correct router-id.<\/p>\n
Create \u201c\/etc\/default\/frr\u201d and add \u201cospfd=yes\u201d. This will prevent PVE from disabling “ospfd” under the daemons setting every time you apply new SDN settings.<\/p>\n
Apply SDN settings and make sure there are no issues, correct them if so.<\/p>\n
Finally create containers in each zone and make sure they can ping each other. Check that OSPF is running and peering is up. Use “vtysh” to do so, the syntax is similar to that of Cisco\u2019s CLI.<\/p>\n
Instead of creating dummy interfaces you could also have used the standard loopback interface \u201clo\u201d that Linux uses. I did it this way to show how to manipulate the creation on interfaces outside the GUI.<\/p>\n
Cisco 9000v<\/h1>\n
Configuration<\/h2>\n
Below are the pertinent parts of the 9k configuration.<\/p>\n
nv overlay evpn<\/b>\r\nfeature ospf<\/b>\r\nfeature bgp<\/b>\r\nfeature pim<\/b>\r\nfeature fabric forwarding<\/b>\r\nfeature interface-vlan<\/b>\r\nfeature vn-segment-vlan-based<\/b>\r\nfeature nv overlay<\/b><\/pre>\n
Next the\u00a0 vrf part of the setup.<\/p>\n
ip prefix-list tna seq 10 permit 10.10.100.0\/24 <\/b>\r\nip prefix-list tna-export seq 10 permit 172.16.10.0\/24 <\/b>\r\nip prefix-list tna-export seq 20 permit 192.168.0.0\/30 <\/b>\r\nip prefix-list tnb seq 10 permit 10.10.200.0\/24 <\/b>\r\nip prefix-list tnb-export seq 10 permit 172.16.20.0\/24 <\/b>\r\nip prefix-list tnb-export seq 20 permit 192.168.4.0\/30 <\/b>\r\nroute-map tna-export permit 10<\/b>\r\n  match ip address prefix-list tna-export <\/b>\r\nroute-map tna-import permit 10<\/b>\r\n  match ip address prefix-list tna <\/b>\r\nroute-map tnb-export permit 10<\/b>\r\n  match ip address prefix-list tnb-export <\/b>\r\nroute-map tnb-import permit 10<\/b>\r\n  match ip address prefix-list tnb <\/b>\r\nvrf context management<\/b>\r\nvrf context tna<\/b>\r\n  rd 10.10.10.10:100<\/b>\r\n  address-family ipv4 unicast<\/b>\r\n    route-target import 10.10.10.10:100<\/b>\r\n    route-target export 10.10.10.10:100<\/b>\r\n    import vrf default map tna-import<\/b>\r\n    export vrf default map tna-export<\/b>\r\nvrf context tnb<\/b>\r\n  rd 10.10.10.10:200<\/b>\r\n  address-family ipv4 unicast<\/b>\r\n    route-target import 10.10.10.10:200<\/b>\r\n    route-target export 10.10.10.10:200<\/b>\r\n    import vrf default map tnb-import<\/b>\r\n    export vrf default map tnb-export<\/b>\r\n<\/pre>\n
We need to leak routes between the global table route (GRT) and the vrfs TNA and TNB. The reason for this should be apparent. The eBGP controller from the cluster is sending routes from the VNETS, they need to be injected into the corresponding tenants.<\/p>\n
We need to make sure we do not leak routes from VRF TNA onto VRF TNB or vice versa, when the routes from the tenants are send back back to the cluster, PVE will put them in the correct VNET.<\/p>\n
Next we set the correct IP addresses.<\/p>\n
interface Ethernet1\/1<\/b>\r\n  no switchport<\/b>\r\n  mac-address 0000.1111.1111<\/b>\r\n  ip address 192.168.1.10\/24<\/b>\r\n  no shutdown<\/b>\r\n\r\ninterface Ethernet1\/2<\/b>\r\n\r\ninterface Ethernet1\/3<\/b>\r\n\r\ninterface Ethernet1\/4<\/b>\r\n\r\ninterface Ethernet1\/5<\/b>\r\n\r\ninterface Ethernet1\/6<\/b>\r\n  no switchport<\/b>\r\n  mac-address 0000.2222.1234<\/b>\r\n  vrf member tna<\/b>\r\n  ip address 192.168.0.1\/30<\/b>\r\n  no shutdown<\/b>\r\n\r\ninterface Ethernet1\/7<\/b>\r\n  no switchport<\/b>\r\n  mac-address 0000.3333.1234<\/b>\r\n  vrf member tnb<\/b>\r\n  ip address 192.168.0.5\/30<\/b>\r\n  no shutdown<\/b>\r\n\r\n<\/pre>\n
Finally we set the mgmt vrf address (if we want to access via ssh) and the loopback address.<\/b><\/p>\n
Then we configure eBGP back to the cluster and the vrf BGP to the tenants<\/p>\n
interface mgmt0<\/b>\r\n  vrf member management<\/b>\r\n  ip address 192.168.100.10\/24<\/b>\r\n\r\ninterface loopback0<\/b>\r\n  ip address 10.10.10.10\/32<\/b>\r\nicam monitor scale<\/b>\r\n\r\nline console<\/b>\r\n  exec-timeout 0<\/b>\r\nline vty<\/b>\r\n  exec-timeout 0<\/b>\r\nboot nxos bootflash:\/nxos.9.3.5.bin sup-1<\/b>\r\nrouter bgp 65100<\/b>\r\n  router-id 10.10.10.10<\/b>\r\n  address-family ipv4 unicast<\/b>\r\n    network 10.10.200.0\/24<\/b>\r\n  neighbor 192.168.1.43<\/b>\r\n    remote-as 65000<\/b>\r\n    address-family ipv4 unicast<\/b>\r\n      soft-reconfiguration inbound<\/b>\r\n    address-family l2vpn evpn<\/b>\r\n      send-community<\/b>\r\n      send-community extended<\/b>\r\n  vrf tna<\/b>\r\n    address-family ipv4 unicast<\/b>\r\n      network 10.10.100.0\/24<\/b>\r\n    neighbor 192.168.0.2<\/b>\r\n      remote-as 64000<\/b>\r\n      address-family ipv4 unicast<\/b>\r\n        send-community<\/b>\r\n        send-community extended<\/b>\r\n  vrf tnb<\/b>\r\n    address-family ipv4 unicast<\/b>\r\n      network 10.10.200.0\/24<\/b>\r\n    neighbor 192.168.0.6<\/b>\r\n      remote-as 63000<\/b>\r\n      address-family ipv4 unicast<\/b>\r\n        send-community<\/b>\r\n        send-community extended<\/b>\r\n!<\/b>\r\n<\/pre>\n
\u00a0Remarks:<\/p>\n
1) Two tenants are created tenant A (TNA) and tenant B (TNB).<\/p>\n
2) In Fig 1. TNA and TNB are routers capable of peering via BGP.<\/p>\n
3) As stated two vrfs are created tna and tnb.<\/p>\n
4) By default the global routing table (GRT) and each for the tenants (TNA), (TNB) will only hold routes for the cluster and each tenant. We need to leak routes so each other is aware of the other. From the security perspective this is not an issue, if your are the provider you need access to all routes.<\/p>\n
a) However, you need to have prefix lists and route maps to prevent accidentally leaking routes between tenants.<\/p>\n
b) Pay attention under the VRF definitions and how we apply route maps to inject routes to the GRT, TNA and TNB.<\/p>\n
5) The peering to PX3 brings the routes from and to the cluster.<\/p>\n
Testing<\/h1>\n
Cisco 9000v<\/h2>\n
We should see now that both the cluster and and 9k should have exchange routes.<\/p>\n
On the 9k the standard IOS commands should show you the status of BGP peering.<\/p>\n
nxos-9k# sh ip bgp summary \r\nBGP summary information for VRF default, address family IPv4 Unicast\r\nBGP router identifier 10.10.10.10, local AS number 65100\r\nBGP table version is 9, IPv4 Unicast config peers 1, capable peers 1\r\n5 network entries and 6 paths using 980 bytes of memory\r\nBGP attribute entries [5\/860], BGP AS path entries [3\/18]\r\nBGP community entries [0\/0], BGP clusterlist entries [0\/0]\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd\r\n192.168.1.43 4 65000 41 38 9 0 0 00:01:41 2<\/pre>\n
This of course shows peering to PX3 because we are doing IPV4 unicast to it.<\/p>\n
Next we can see routes that the 9k is getting from the cluster and what we will be sending to it.<\/p>\n
nxos-9k# sh ip bgp \r\nBGP routing table information for VRF default, address family IPv4 Unicast\r\nBGP table version is 9, Local Router ID is 10.10.10.10\r\nStatus: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best\r\nPath type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i\r\nnjected\r\nOrigin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b\r\nest2\r\n\r\nNetwork Next Hop Metric LocPrf Weight Path\r\n*>e10.10.100.0\/24 192.168.1.43 0 0 65000 ?\r\n l10.10.200.0\/24 0.0.0.0 100 32768 i\r\n*>e 192.168.1.43 0 0 65000 ?\r\n*>e172.16.10.0\/24 192.168.0.2 0 0 64000 i\r\n*>e172.16.20.0\/24 192.168.0.6 0 0 63000 i\r\n*>e192.168.0.0\/30 192.168.0.2 0 0 64000 ?<\/pre>\n
As you can see 10.10.x.0\/24 are the containers, we also see the 172.16.x.0 routes we injected from the tenants, They should appear within the cluster.<\/p>\n
Finally if we do:<\/p>\n
nxos-9k# sh bgp l2vpn evpn \r\nnxos-9k# sh bgp l2vpn evpn summary \r\nBGP summary information for VRF default, address family L2VPN EVPN\r\nBGP router identifier 10.10.10.10, local AS number 65100\r\nBGP table version is 2, L2VPN EVPN config peers 1, capable peers 0\r\n0 network entries and 0 paths using 0 bytes of memory\r\nBGP attribute entries [0\/0], BGP AS path entries [0\/0]\r\nBGP community entries [0\/0], BGP clusterlist entries [0\/0]\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd\r\n192.168.1.43 4 65000 220 216 0 0 0 00:10:37 0 (No Cap)<\/pre>\n
You may think this an error that needs to be resolved. It is not, we are trying to do EVPN to a device that is not configured for it (the cluster). If you notice we left the commands for the address family L2VPN EVPN in the configuration, they are not needed. They should be taken out.<\/p>\n
PVE Cluster<\/h2>\n
At a command prompt issue “vtysh”, this is a command that you should know by heart if you use FRR (which PVE uses).<\/p>\n
We should see routes.<\/p>\n
pve-3# sh bgp pv4 unicast\r\nBGP table version is 5, local router ID is 192.168.1.43, vrf id 0\r\nDefault local pref 100, local AS 65000\r\nStatus codes: s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,\r\n i internal, r RIB-failure, S Stale, R Removed\r\nNexthop codes: @NNN nexthop's vrf id, < announce-nh-self\r\nOrigin codes: i - IGP, e - EGP, ? - incomplete\r\nRPKI validation codes: V valid, I invalid, N Not found\r\n\r\nNetwork Next Hop Metric LocPrf Weight Path\r\n *> 10.10.100.0\/24 0.0.0.0(pve-3)@10<\r\n 0 32768 ?\r\n *> 10.10.200.0\/24 0.0.0.0(pve-3)@15<\r\n 0 32768 ?\r\n *> 172.16.10.0\/24 192.168.1.10 0 65100 64000 i\r\n *> 172.16.20.0\/24 192.168.1.10 0 65100 63000 i\r\n *> 192.168.0.0\/30 192.168.1.10 0 65100 64000 ?\r\n\r\nDisplayed 5 routes and 5 total paths<\/pre>\n
As you can see we have the 10.x.x.x from our containers and in addition the 176.16.x.x routes from the external tenants.<\/p>\n
Tenants<\/h2>\n
You should be able to ping from the tenants (tenant b) here:<\/p>\n
tnb#sh ip int bri\r\nInterface IP-Address OK? Method Status Protocol\r\nFastEthernet0\/0 192.168.0.6 YES NVRAM up up \r\nFastEthernet0\/1 172.16.20.1 YES NVRAM up up \r\nLoopback0 172.16.1.2 YES NVRAM up up \r\ntnb#ping 10.10.200.20 so\r\ntnb#ping 10.10.200.20 source 172.16.20.1\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.10.200.20, timeout is 2 seconds:\r\nPacket sent with a source address of 172.16.20.1 \r\n!!!!!\r\nSuccess rate is 100 percent (5\/5), round-trip min\/avg\/max = 8\/12\/20 ms\r\ntnb#ping 10.10.200.10 source 172.16.20.1\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.10.200.10, timeout is 2 seconds:\r\nPacket sent with a source address of 172.16.20.1 \r\n!!!!!\r\nSuccess rate is 100 percent (5\/5), round-trip min\/avg\/max = 8\/10\/12 ms\r\ntnb#<\/pre>\n
Notice we need to source the ping from the right interface. And for those from Missouri the show me state, to show were not pulling a fast one.<\/p>\n
From the container of tenant a:<\/p>\n
\u00a0test100:~# ifconfig\r\neth0 Link encap:Ethernet HWaddr BC:24:11:79:34:C1 \r\n inet addr:10.10.100.10 Bcast:0.0.0.0 Mask:255.255.255.0\r\n inet6 addr: fe80::be24:11ff:fe79:34c1\/64 Scope:Link\r\n UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1\r\n RX packets:8 errors:0 dropped:0 overruns:0 frame:0\r\n TX packets:9 errors:0 dropped:0 overruns:0 carrier:0\r\n collisions:0 txqueuelen:1000 \r\n RX bytes:656 (656.0 B) TX bytes:726 (726.0 B)\r\n\r\nlo Link encap:Local Loopback \r\n inet addr:127.0.0.1 Mask:255.0.0.0\r\n inet6 addr: ::1\/128 Scope:Host\r\n UP LOOPBACK RUNNING MTU:65536 Metric:1\r\n RX packets:0 errors:0 dropped:0 overruns:0 frame:0\r\n TX packets:0 errors:0 dropped:0 overruns:0 carrier:0\r\n collisions:0 txqueuelen:1000 \r\n RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)\r\n\r\ntest100:~# ping 172.16.10.1\r\nPING 172.16.10.1 (172.16.10.1): 56 data bytes\r\n64 bytes from 172.16.10.1: seq=0 ttl=252 time=8.330 ms\r\n64 bytes from 172.16.10.1: seq=1 ttl=252 time=9.146 ms\r\n^C\r\n--- 172.16.10.1 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min\/avg\/max = 8.330\/8.738\/9.146 ms<\/pre>\n
We can ping router TNA!<\/p>\n
Conclusion<\/h1>\n
This lab shows how to add an eBGP peer so you can connect external clients to the PVE cluster. Hope you enjoyed it.<\/p>\n
Take care,<\/p>\n
Ciao.<\/p>\n
 <\/p>\n
 <\/p>\n
\n","protected":false},"excerpt":{"rendered":"
Introduction In part 2 we will continue with integration of a PVE cluster with a Cisco 9kv running NXOS. Figure 1. shows the network diagram. \u00a0We are using the 9000v as an eBGP peer for the PVE cluster. Any device that supports BGP can replace the Cisco 9k.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-603","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/603","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=603"}],"version-history":[{"count":35,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/603\/revisions"}],"predecessor-version":[{"id":638,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/603\/revisions\/638"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=603"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=603"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=603"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}