Introduction
The purpose of the following proof of concept is to demonstrate integration between Proxmox PVE and a Cisco 9000v.
PVE is a fantastic open-source virtualization environment. Recently it has gained a lot of traction in particular with the issues with VSphere licensing.
However, while I see several blogs and videos regarding it and in particular its SDN capabilities, I have not seen detail setups involving integration with other SDN devices.
This is unfortunate since the Cisco 9000 for example is very prevalent and at one point if someone decides to move to PVE, they will need to integrate Cisco or another device to it.
In any case, from the perspective of learning I think the following will be very instructional.
Road Map
The intention of this lab is to do provide an example of extending a VXLAN setup from PVE onto a Cisco 9000v.
The environment and appliances used are the following:
1) EVE-NG. Another fantastic open-source application that allows you to create labs and tests configurations. For this proof of concept lab, I am using the latest EVE-NG community edition with 32 GB.
2) Several Cisco appliances. They include a 3252 image, a 7206 image and a Cisco 9000v running NXOS 9.3.5.
3) PVE running the latest (as of my writing it was 9.1). It runs the community software. The environment is setup to run containers as devices for testing. Configuring of a cluster is beyond the scope of this lab. There are several blogs and Youtube videos that will point you in the right direction.
The observant reader will notice that I said containers instead of VMs. The reason is that the PVE instances I am running are inside another hypervisor (ESXI in this case). Because of this PVE will not run VMs in this so called nested-hypervisor environment. However, it can run containers without an issue. I will let the astute reader to realize why this is the case.
Network Diagram
Figure 1. shows the network diagram for the lab.
A few notes:
1) MGMT: Management vlan for access via ssh. All nodes are accessible in this fashion only the PVE nodes are shown to make the diagram easier to read.
2) PVE is a collapsed spine architecture. Some may dispute this however, read the following white paper by Juniper Networks that makes this clear in general not just for PVE clusters.
3) In a real production system, the PVE nodes would access storage over a dedicated network.
4) Connectivity to the 9000v (leaf-a) would also be on a dedicated network.
5) Finally, the PVE nodes would also form a cluster on a dedicated network.
ToR stands for a top-of-the rack switch which will not have VXLAN capabilities however, it will allow you to use the VLANS mapped in the Cisco 9000v.
Configuration
PVE
Configure a normal PVE cluster. Configure SDN, do not create a fabric. The reason will be apparent next and create VXLAN zones as follows:
1) Create a VXLAN zone for each vxlan you want to use.
2) Add the three PVE nodes (192.168.1.x) and the loopback IP of the Cisco 9000v. This is important since the VTEPs will establish connections against this IP. In our case we will create two such zones.
3) VNETS:
VX1-test: 10.10.1.0/24 (GW 10.10.1.1) vni:10000.
VX2-test: 10.10.2.0/24 (GW 10.10.2.1) vni:15000.
4) Set the MTU to 9216 for the VNETS to match Cisco’s default.
The reason for above is that if you were to create a normal fabric using the GUI, when you want to add the Cisco 9000v to the zones you get an error. Try many things to fix it, I finally found in the PVE forums that a patch was issued that allows you to do this. Did not install the patch so I do not know if it actually fixes the issue. Perhaps someone can comment on this.
Create a file named “lo” in “/etc/networks/interfaces.d/” on all nodes of the cluster.
Add the following to it:
iface lo:0 inet static address 10.10.10.1 netmask 255.255.255.255 # Add a route to 10.10.10.4 via 192.168.1.10 post-up ip route add 10.10.10.4 via 192.168.1.10 # Remove the route when the interface goes down pre-down ip route del 10.10.10.4 via 192.168.1.10
Replace each IP accordingly, 10.10.10.1,2, and 3 for each node in the cluster. Reboot, better yet restart the network service only.
After having a working cluster, test that you can create containers in the VXLANs with the IPs in the range describe in (3) above.
You should be able to ping containers on the same VNIs but not across. You will also notice that even though you can configure a GW and do SNAT, you will not be able to connect outside the cluster.
This is of course because VXLAN is a layer 2 protocol and thus has no knowledge of layer 3 (i.e. routing).
Cisco 9000v
Configure the Cisco 9000v as follows:
Features
feature ospf feature bgp feature pim feature fabric forwarding feature interface-vlan feature vn-segment-vlan-based feature nv overlay nv overlay evpn
Interfaces
You need a loopback interface (NVE uses it). You also need to assign a MAC address to the interface otherwise you will not be able to communicate with the cluster. Add static routes for the loopback interfaces of the cluster.
interface loopback0 ip address 10.10.10.4/32 ip route 10.10.10.1/32 192.168.1.1 ip route 10.10.10.2/32 192.168.1.2 ip route 10.10.10.3/32 192.168.1.3 interface Ethernet1/1 no switchport mac-address 0000.1111.1111 ip address 192.168.1.10/24 no shutdown interface Ethernet1/2 switchport mode trunk interface Ethernet1/3 switchport access vlan 11
NVE
We will do static ingress replication and use the loopback interface as the source for peering.
interface nve1 no shutdown source-interface loopback0 member vni 15000 ingress-replication protocol static peer-ip 10.10.10.1 peer-ip 10.10.10.2 peer-ip 10.10.10.3 member vni 10000 ingress-replication protocol static peer-ip 10.10.10.1 peer-ip 10.10.10.2 peer-ip 10.10.10.3
Notice that you can have several vnis defined however only one NVE interface (you can only create one!)
Normally you would also use BGP ingress replication which dynamically would learn the VTEPs in use. Here though we do not have an evpn setup.
Also notice that we need connections to all members of the cluster. This would not be necessary if we were using evpn since we will be carrying labels and mac addresses in the payload. That setup would need a route reflector.
Testing
Now that you have everything configured you should be able to ping across. I do not show the configuration of the ToR switch. This should be easy to do. I created on the ToR switch a couple of VLAN interfaces with IP 10.10.x.1, they act as gateways for all devices including the containers, I also use NAT to connect to the Internet. Leaf-a and ToR communicate via a trunk port.
Above is left to the reader to configure. If you want inter-container communications, you may have to play with the security settings in the cluster if you cannot ping between containers in different vxlans. This is beyond the scope of this lab.
If you cannot ping the containers make sure that the cluster loopback interfaces can ping the loopback interface of the Cisco. This is because the VTEP tunnels are being established to the Cisco loopback interface. Just re-apply SDN settings this will reload the network in all members.
From PC1:
pc1#sh ip int bri Interface IP-Address OK? Method Status Protocol FastEthernet0/0 10.10.1.20 YES NVRAM up up FastEthernet0/1 unassigned YES NVRAM administratively down down pc1#sh ip route Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route Gateway of last resort is 10.10.1.1 to network 0.0.0.0 10.0.0.0/24 is subnetted, 1 subnets C 10.10.1.0 is directly connected, FastEthernet0/0 S* 0.0.0.0/0 [1/0] via 10.10.1.1
Now ping the container at 10.10.1.10:
pc1#ping 10.10.1.10 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.10.1.10, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/12 ms
From PC2:
pc2#ping 10.10.1.11 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.10.1.11, timeout is 2 seconds: .!!!! Success rate is 80 percent (4/5), round-trip min/avg/max = 8/14/20 ms
IP 10.10.1.11 is a second container configured in the cluster. Finally in the 9k, you can see the NVE peers. They all should be up.
nxos-9k# show nve peers Interface Peer-IP State LearnType Uptime Route r-Mac --------- -------------------------------------- ----- --------- -------- ----- ------------ nve1 192.168.1.1 Up DP 00:11:22 n/a nve1 10.10.10.1 Up DP 02:37:56 n/a nve1 192.168.1.2 Up DP 00:11:34 n/a nve1 10.10.10.2 Up DP 02:37:56 n/a nve1 192.168.1.3 Up DP 00:11:26 n/a nve1 10.10.10.3 Up DP 02:37:56 n/a
You should also be able to ping the Internet since the ToR Switch is doing NAT. Of course in a production system you would not want your tenants to use you as their Internet access.
There you have it. This shows how to extend vxlans, first to a Cisco 9000v and to a top-of-the-rack switch that may not have vxlan capabilities.
This can be done for other devices that support vxlan. It was not as complicated as I though it would be.
Take care,
Ciao.