Juniper Networks<\/a> that makes this clear in general not just for PVE clusters.<\/p>\n3) In a real production system, the PVE nodes would access storage over a dedicated network.<\/p>\n
4) Connectivity to the 9000v (leaf-a) would also be on a dedicated network.<\/p>\n
5) Finally, the PVE nodes would also form a cluster on a dedicated network.<\/p>\n
ToR stands for a top-of-the rack switch which will not have VXLAN capabilities however, it will allow you to use the VLANS mapped in the Cisco 9000v.<\/p>\n
Configuration<\/h1>\nPVE<\/h2>\n
Configure a normal PVE cluster. Configure SDN,\u00a0 do not create a fabric. The reason will be apparent next and create VXLAN zones as follows:<\/p>\n
1)\u00a0 Create a VXLAN zone for each vxlan you want to use.<\/p>\n
2) Add the three PVE nodes (192.168.1.x) and the loopback IP of the Cisco 9000v. This is important since the VTEPs will establish connections against this IP. In our case we will create two such zones.<\/p>\n
3) VNETS:<\/p>\n
VX1-test: 10.10.1.0\/24 (GW 10.10.1.1) vni:10000.<\/p>\n
VX2-test: 10.10.2.0\/24 (GW 10.10.2.1) vni:15000.<\/p>\n
4) Set the MTU to 9216 for the VNETS to match Cisco\u2019s default.<\/p>\n
The reason for above is that if you were to create a normal fabric using the GUI, when you want to add the Cisco 9000v to the zones you get an error. Try many things to fix it, I finally found in the PVE forums that a patch was issued that allows you to do this. Did not install the patch so I do not know if it actually fixes the issue. Perhaps someone can comment on this.<\/p>\n
Create a file named “lo” in \u201c\/etc\/networks\/interfaces.d\/\u201d on all nodes of the cluster.<\/p>\n
Add the following to it:<\/p>\n
iface lo:0 inet static<\/span>\r\n address 10.10.10.1<\/span>\r\n netmask 255.255.255.255<\/span>\r\n # Add a route to 10.10.10.4 via 192.168.1.10<\/span>\r\n post-up ip route add 10.10.10.4 via 192.168.1.10<\/span>\r\n # Remove the route when the interface goes down<\/span>\r\n pre-down ip route del 10.10.10.4 via 192.168.1.10<\/span><\/pre>\nReplace each IP accordingly, 10.10.10.1,2, and 3 for each node in the cluster. Reboot, better yet restart the network service only.<\/p>\n
After having a working cluster, test that you can create containers in the VXLANs with the IPs in the range describe in (3) above.<\/p>\n
You should be able to ping containers on the same VNIs but not across. You will also notice that even though you can configure a GW and do SNAT, you will not be able to connect outside the cluster.<\/p>\n
This is of course because VXLAN is a layer 2 protocol and thus has no knowledge of layer 3 (i.e. routing).<\/p>\n
Cisco 9000v<\/h2>\n
Configure the Cisco 9000v as follows:<\/p>\n
Features<\/span><\/span><\/p>\nfeature ospf<\/span>\r\nfeature bgp<\/span>\r\nfeature pim<\/span>\r\nfeature fabric forwarding<\/span>\r\nfeature interface-vlan<\/span>\r\nfeature vn-segment-vlan-based<\/span>\r\nfeature nv overlay <\/span>\r\nnv overlay evpn<\/span>\r\n<\/pre>\nInterfaces<\/span><\/span><\/p>\nYou need a loopback interface (NVE uses it). You also need to assign a MAC address to the interface otherwise you will not be able to communicate with the cluster. Add static routes for the loopback interfaces of the cluster.<\/p>\n
interface loopback0<\/span>\r\n ip address 10.10.10.4\/32<\/span>\r\n\r\nip route 10.10.10.1\/32 192.168.1.1<\/span>\r\nip route 10.10.10.2\/32 192.168.1.2<\/span>\r\nip route 10.10.10.3\/32 192.168.1.3<\/span>\r\n\r\ninterface Ethernet1\/1<\/span>\r\n no switchport<\/span>\r\n mac-address 0000.1111.1111<\/span>\r\n ip address 192.168.1.10\/24<\/span>\r\n no shutdown<\/span>\r\n \r\ninterface Ethernet1\/2<\/span>\r\n switchport mode trunk <\/span>\r\n\r\ninterface Ethernet1\/3<\/span>\r\n switchport access vlan 11<\/span>\r\n<\/pre>\nNVE<\/span><\/span><\/p>\nWe will do static ingress replication and use the loopback interface as the source for peering.<\/p>\n
interface nve1<\/span><\/span>\r\n no shutdown<\/span><\/span>\r\n source-interface loopback0<\/span><\/span>\r\n member vni 15000<\/span><\/span>\r\n ingress-replication protocol static<\/span><\/span>\r\n peer-ip 10.10.10.1<\/span><\/span>\r\n peer-ip 10.10.10.2<\/span><\/span>\r\n peer-ip 10.10.10.3<\/span><\/span>\r\n member vni 10000<\/span><\/span>\r\n ingress-replication protocol static<\/span><\/span>\r\n peer-ip 10.10.10.1<\/span><\/span>\r\n peer-ip 10.10.10.2<\/span><\/span>\r\n <\/strong>peer-ip 10.10.10.3 <\/strong> <\/span><\/span><\/pre>\nNotice that you can have several vnis defined however only one NVE interface (you can only create one!)<\/p>\n
Normally you would also use BGP ingress replication which dynamically would learn the VTEPs in use. Here though we do not have an evpn setup.<\/p>\n
Also notice that we need connections to all members of the cluster. This would not be necessary if we were using evpn since we will be carrying labels and mac addresses in the payload. That setup would need a route reflector.<\/p>\n
Testing<\/h2>\n
Now that you have everything configured you should be able to ping across. I do not show the configuration of the ToR switch. This should be easy to do. I created on the ToR switch a couple of VLAN interfaces with IP 10.10.x.1, they act as gateways for all devices including the containers, I also use NAT to connect to the Internet. Leaf-a and ToR communicate via a trunk port.<\/p>\n
Above is left to the reader to configure. If you want inter-container communications, you may have to play with the security settings in the cluster if you cannot ping between containers in different vxlans. This is beyond the scope of this lab.<\/p>\n
If you cannot ping the containers make sure that the cluster loopback interfaces can ping the loopback interface of the Cisco. This is because the VTEP tunnels are being established to the Cisco loopback interface. Just re-apply SDN settings this will reload the network in all members.<\/p>\n
From PC1:<\/p>\n
pc1#sh ip int bri\r\nInterface IP-Address OK? Method Status Protocol\r\nFastEthernet0\/0 10.10.1.20 YES NVRAM up up \r\nFastEthernet0\/1 unassigned YES NVRAM administratively down down\r\n\r\npc1#sh ip route\r\nCodes: C - connected, S - static, R - RIP, M - mobile, B - BGP\r\n D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area \r\n N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2\r\n E1 - OSPF external type 1, E2 - OSPF external type 2\r\n i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2\r\n ia - IS-IS inter area, * - candidate default, U - per-user static route\r\n o - ODR, P - periodic downloaded static route\r\n\r\nGateway of last resort is 10.10.1.1 to network 0.0.0.0\r\n\r\n10.0.0.0\/24 is subnetted, 1 subnets\r\nC 10.10.1.0 is directly connected, FastEthernet0\/0\r\nS* 0.0.0.0\/0 [1\/0] via 10.10.1.1<\/pre>\nNow ping the container at 10.10.1.10:<\/p>\n
pc1#ping 10.10.1.10\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.10.1.10, timeout is 2 seconds:\r\n!!!!!\r\nSuccess rate is 100 percent (5\/5), round-trip min\/avg\/max = 4\/8\/12 ms<\/pre>\nFrom PC2:<\/p>\n
pc2#ping 10.10.1.11\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.10.1.11, timeout is 2 seconds:\r\n.!!!!\r\nSuccess rate is 80 percent (4\/5), round-trip min\/avg\/max = 8\/14\/20 ms<\/pre>\nIP 10.10.1.11 is a second container configured in the cluster. Finally in the 9k, you can see the NVE peers. They all should be up.<\/p>\n
nxos-9k# show nve peers\r\nInterface Peer-IP State LearnType Uptime Route\r\nr-Mac \r\n--------- -------------------------------------- ----- --------- -------- -----\r\n------------\r\nnve1 192.168.1.1 Up DP 00:11:22 n\/a \r\n \r\nnve1 10.10.10.1 Up DP 02:37:56 n\/a \r\n \r\nnve1 192.168.1.2 Up DP 00:11:34 n\/a \r\n \r\nnve1 10.10.10.2 Up DP 02:37:56 n\/a \r\n \r\nnve1 192.168.1.3 Up DP 00:11:26 n\/a \r\n \r\nnve1 10.10.10.3 Up DP 02:37:56 n\/a \u00a0\r\n\r\n<\/pre>\nYou should also be able to ping the Internet since the ToR Switch is doing NAT. Of course in a production system you would not want your tenants to use you as their Internet access.<\/p>\n
There you have it. This shows how to extend vxlans, first to a Cisco 9000v and\u00a0 to a top-of-the-rack switch that may not have vxlan capabilities.<\/p>\n
This can be done for other devices that support vxlan. It was not as complicated as I though it would be.<\/p>\n
Take care,<\/p>\n
Ciao.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction The purpose of the following proof of concept is to demonstrate integration between Proxmox PVE and a Cisco 9000v. PVE\u00a0is a fantastic open-source virtualization environment. Recently it has gained a lot of traction in particular with the issues with VSphere\u00a0licensing. However, while I see several blogs and videos regarding it and in particular its … <\/p>\n
Continue reading “PVE VXLAN NXOS Integration Oh My!”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-554","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/554","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=554"}],"version-history":[{"count":44,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/554\/revisions"}],"predecessor-version":[{"id":601,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/554\/revisions\/601"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=554"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=554"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=554"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-vxlan-nxos-1.png\")
<\/a>