Introduction
In the final part of these series, we will integrate a Cisco 9k with a PVE cluster using the previous configurations we did setup.
The network diagram is shown in Fig, 1.
The plan is to:
1) Create a SDN data center. It will consists of:
a) A PVE cluster
b) An OSPF manual fabric,
c) An EVPN controller with a Cisco 9000v as a leaf.
d) An eBGP controller to announce routes to external BGP peers.
e) Two tenants that are part of the SDN environment and also reside externally.
Configurations
PVE
Configure a cluster. We will create a manual fabric. See the previous posts for why we do this.
Create a bridge on each node (vmbr1) and a assign IP addresses per the network diagram. This is the network that the cluster nodes will use to communicate.
Create a file and named it “frr” in “/etc/default/” it should look like this:
root@pve-3:~# cat /etc/default/frr ospfd=yes
Do this on all nodes and make sure to use the correct IP addresses.
Under “/etc/frr/” create “frr.conf/local”, add the following to it doing it on all nodes (use the respective “lo” IP create above):
interface lo ip ospf area 0 exit ! interface vmbr1 ip ospf area 0 exit ! router ospf ospf router-id 10.10.10.3 exit
Create an EVPN controller:
1) Use the IP address configured for the bridge you created. Do not use the loopback interfaces. The loopback interfaces are in used for the fabric and for connectivity to the Cisco devices via OSPF. Add the IP of the Cisco leaf to the controller.
2) Apply your SDN settings. The new configuration for “frr” should be merged and you should have OSPF peering between the nodes. If there are issues fix them. Each node should have an additional IP for the loopback interface.
3) Create zones and VNETS:
a) Zones v100 (tag 30000) and v200 (tag 40000). Make all nodes part of the exit nodes (for SNAT), make PX3 the preferred exit node for BGP.
b) Create VNETs for each zone: test1 10.100.1.0/24:tag 31000 (GW 10.100.1.1), test2: 10.200.1.0/24:tag 41000 (GW 10.200.1.1).
c) Use AS 65000 (default). Remember the tags giving these are important and are needed by the Cisco leaf.
e) Create a BGP controller (eBGP) and use 192.168.1.20 (per the network diagram) as the peer. Use AS 65000, this is confusing since it is the AS of the cluster not the AS of the client (which will be 65100). When we configure the 9k the cluster will accept 65100 and peer with it.
Apply the settings for SDN. If you do not have any errors, create your containers in each zone making sure they can ping each other.
Cisco
Leaf A
The main parts of the configuration for the leaf are shown below.
allow feature-set mpls limit-resource vlan minimum 16 maximum 4094 limit-resource vrf minimum 2 maximum 4096 limit-resource port-channel minimum 0 maximum 511 limit-resource u4route-mem minimum 248 maximum 248 limit-resource u6route-mem minimum 96 maximum 96 limit-resource m4route-mem minimum 58 maximum 58 limit-resource m6route-mem minimum 8 maximum 8 limit-resource vni_bd minimum 4096 maximum 4096 nv overlay evpn feature ospf feature bgp feature pim feature fabric forwarding feature interface-vlan feature vn-segment-vlan-based feature nv overlay
fabric forwarding anycast-gateway-mac 0000.dc01.dc01
vlan 1,100,200,1000,2000
vlan 100
vn-segment 31000
vlan 200
vn-segment 41000
vlan 1000
vn-segment 30000
vlan 2000
vn-segment 40000
vrf context management
vrf context vrfvx_v100
vni 30000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context vrfvx_v200
vni 40000
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
hardware access-list tcam region racl 512
hardware access-list tcam region arp-ether 256
After the last two commands are issued you may need to reboot the 9k, do so.
Now we configure the interfaces.
interface Vlan1000
no shutdown
mtu 9216
mac-address 0000.9999.9999
vrf member vrfvx_v100
ip forward
fabric forwarding mode anycast-gateway
interface Vlan2000
no shutdown
mtu 9216
mac-address 0000.8888.8888
vrf member vrfvx_v200
ip forward
fabric forwarding mode anycast-gateway
interface nve1
no shutdown
description to-pve
host-reachability protocol bgp
source-interface loopback0
member vni 30000 associate-vrf
member vni 31000
suppress-arp
ingress-replication protocol bgp
member vni 40000 associate-vrf
member vni 41000
suppress-arp
ingress-replication protocol bgp
Note that while we map VLANs 31000 and 32000, the magic is done by creating VLANs 1000 and 2000, These interfaces even though do not have an IP are forwarding “l2vpn evpn” packets so L2 and L3 information can flow between the devices.
The NVE interface that will carry VXLAN info is configured to use BGP ingress replication and have as it source the loopback interface created.
The interface that talks to the cluster.
interface Ethernet1/1 no switchport mac-address 0000.1111.1111 ip address 192.168.1.10/24 ip router ospf MSG area 0.0.0.0 no shutdown
Finally the interfaces towards tenant A and tenant B,
interface Ethernet1/4 switchport access vlan 100 interface Ethernet1/5 switchport access vlan 200
Do not forget the loopback address.
interface loopback0 ip address 10.10.10.10/32 ip router ospf MSG area 0.0.0.0
router ospf MSG router-id 10.10.10.10
At this point the Cisco leaf should be peering via OSPF to the PVE cluster.
Now for the L2VPN EVPN configuration.
router bgp 65000
router-id 10.10.10.10
log-neighbor-changes
address-family l2vpn evpn
neighbor 192.168.1.41
remote-as 65000
log-neighbor-changes
address-family ipv4 unicast
send-community
send-community extended
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.1.42
remote-as 65000
log-neighbor-changes
address-family ipv4 unicast
send-community
send-community extended
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.1.43
remote-as 65000
log-neighbor-changes
address-family ipv4 unicast
send-community
send-community extended
address-family l2vpn evpn
send-community
send-community extended
evpn
vni 31000 l2
rd auto
route-target import auto
route-target export auto
vni 41000 l2
rd auto
route-target import auto
route-target export auto
!
The NVE interface should have peered with the cluster. Double check that the VNI definitions match between the cluster and the Cisco.
eBGP
Below are the tidbits concerning the eBGP peering to the external tenants. While we can use any device that can do BGP, we are using a Cisco 9000v for this.
First the VRFs so we can leak routes from the global routing table (GRT) and each tenant.
ip prefix-list tna seq 10 permit 10.100.1.0/24
ip prefix-list tna-export seq 10 permit 172.16.10.0/24
ip prefix-list tna-export seq 20 permit 192.168.8.0/30
ip prefix-list tnb seq 20 permit 10.200.1.0/24
ip prefix-list tnb-export seq 10 permit 172.16.20.0/24
ip prefix-list tnb-export seq 20 permit 192.168.12.0/30
route-map tna-export permit 10
match ip address prefix-list tna-export
route-map tna-import permit 10
match ip address prefix-list tna
route-map tnb-export permit 10
match ip address prefix-list tnb-export
route-map tnb-import permit 10
match ip address prefix-list tnb
vrf context management
vrf context tna
rd 10.10.10.20:100
address-family ipv4 unicast
route-target import 10.10.10.20:100
route-target export 10.10.10.20:100
import vrf default map tna-import
export vrf default map tna-export
vrf context tnb
rd 10.10.10.10:200
address-family ipv4 unicast
route-target import 10.10.10.20:200
route-target export 10.10.10.20:200
import vrf default map tnb-import
export vrf default map tnb-export
Now for the interfaces.
interface Ethernet1/1 no switchport mac-address 0000.1111.4444 ip address 192.168.1.20/24 no shutdown
Above is the IP address you configured for the BGP controller in the cluster.
interface Ethernet1/6 no switchport mac-address 0000.9999.1234 vrf member tna ip address 192.168.0.9/30 no shutdown interface Ethernet1/7 no switchport mac-address 0000.3333.1234 vrf member tnb ip address 192.168.0.13/30 no shutdown
Above are the links towards each tenant.
interface loopback0 ip address 10.10.10.20/32
Finally eBGP with PX3 in the cluster.
router bgp 65100
router-id 10.10.10.20
neighbor 192.168.1.43
remote-as 65000
address-family ipv4 unicast
soft-reconfiguration inbound
vrf tna
address-family ipv4 unicast
network 10.100.2.0/24
neighbor 192.168.0.10
remote-as 64000
address-family ipv4 unicast
send-community
send-community extended
vrf tnb
address-family ipv4 unicast
network 10.200.1.0/24
neighbor 192.168.0.14
remote-as 63000
address-family ipv4 unicast
send-community
send-community extended
!
Some remarks.
1) We have tenants tna and tnb defined.
2) We need to leak routes between the GRT and each tenant. We also need to make sure we do not leak routes between tenant thus the prefix lists and route maps created.
3) As mentioned before the cluster will accept connections and peer itself, this is not an issue if you control all legs of the network otherwise BGP authentication is a must.
4) We are using VRFs for connecting to external clients however, in most production environments you would use instead BGP MPLS (which creates a VPN network). In this case the configuration would need to be changed to use the VPNV4 address family command set.
Verification
We are now ready to verify that we can connect from all tenants into the cluster.
PVE
Using “vtysh” at a command prompt, in this case at PX3.
pve-3# sh bgp l2vpn evpn summary BGP router identifier 192.168.1.43, local AS number 65000 VRF default vrf-id 0 BGP table version 0 RIB entries 31, using 3968 bytes of memory Peers 3, using 70 KiB of memory Peer groups 2, using 128 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 192.168.1.10 4 65000 23385 23416 40 0 0 19:30:35 4 6 N/A pve-1(192.168.1.41) 4 65000 23547 23550 40 0 0 19:36:52 2 6 FRRouting/10.3.1 pve-2(192.168.1.42) 4 65000 23547 23550 40 0 0 19:36:52 2 6 FRRouting/10.3.1 Total number of neighbors 3
As you can see we have peering with the other members of the cluster but also to the Cisco (k (192.168.1.10).
pve-3# sh bgp l2vpn evpn 192.168.1.10 BGP routing table entry for 192.168.1.43:2:[5]:[0]:[0]:[::] Paths: (1 available, best #1) Advertised to non peer-group peers: 192.168.1.10 pve-1(192.168.1.41) pve-2(192.168.1.42) Route [5]:[0]:[0]:[::] VNI 30000 Local 192.168.1.43(pve-3) from 0.0.0.0 (192.168.1.43) Origin IGP, weight 32768, valid, sourced, local, bestpath-from-AS Local, best (First path received) Extended Community: ET:8 RT:65000:30000 Rmac:f6:9c:57:01:a9:f8 Last update: Fri Nov 21 15:35:19 2025 BGP routing table entry for 192.168.1.43:3:[5]:[0]:[0]:[::] Paths: (1 available, best #1) Advertised to non peer-group peers: 192.168.1.10 pve-1(192.168.1.41) pve-2(192.168.1.42) Route [5]:[0]:[0]:[::] VNI 40000 Local 192.168.1.43(pve-3) from 0.0.0.0 (192.168.1.43) Origin IGP, weight 32768, valid, sourced, local, bestpath-from-AS Local, best (First path received) Extended Community: ET:8 RT:65000:40000 Rmac:22:54:28:7a:6d:32 Last update: Fri Nov 21 15:35:19 2025
Above shows the type 5 BGP routes exchanged with the 9k. Below we can see that we are receiving routes via BGP and in particular from tenants tna and tnb. These routes will also show in the routing table if you do “show ip route”.
pve-3# sh ip route bgp B>* 10.100.1.0/24 [20/0] is directly connected, vrf_v100 (vrf vrf_v100), weight 1, 19:44:17 B>* 10.200.1.0/24 [20/0] is directly connected, vrf_v200 (vrf vrf_v200), weight 1, 19:44:17 B>* 172.16.10.0/24 [20/0] via 192.168.1.20, vmbr1, weight 1, 19:38:45 B>* 172.16.20.0/24 [20/0] via 192.168.1.20, vmbr1, weight 1, 19:38:45
Cisco 9k
We show the neighbors we have.
nxos-9k# sh bgp l2vpn evpn summary BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier 10.10.10.10, local AS number 65000 BGP table version is 68, L2VPN EVPN config peers 3, capable peers 3 28 network entries and 32 paths using 5632 bytes of memory BGP attribute entries [18/3096], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.1.41 4 65000 23712 23658 68 0 0 19:45:22 4 192.168.1.42 4 65000 23715 23661 68 0 0 19:45:29 4 192.168.1.43 4 65000 23713 23659 68 0 0 19:45:25 6
As you can see we are peering with all members of the cluster. Other BGP commands will show information similar to what you saw for the cluster.
eBGP
This standard BGP and you should see something like:
bgp-1# sh ip bgp BGP routing table information for VRF default, address family IPv4 Unicast BGP table version is 6, Local Router ID is 10.10.10.20 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b est2 Network Next Hop Metric LocPrf Weight Path *>e10.100.1.0/24 192.168.1.43 0 0 65000 ? *>e10.200.1.0/24 192.168.1.43 0 0 65000 ? *>e172.16.10.0/24 192.168.0.10 0 0 64000 i *>e172.16.20.0/24 192.168.0.14 0 0 63000 i bgp-1# sh ip bgp summary BGP summary information for VRF default, address family IPv4 Unicast BGP router identifier 10.10.10.20, local AS number 65100 BGP table version is 6, IPv4 Unicast config peers 1, capable peers 1 4 network entries and 4 paths using 736 bytes of memory BGP attribute entries [3/516], BGP AS path entries [3/18] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.1.43 4 65000 23813 23762 6 0 0 19:50:22 2
We are exchanging routes.
Now from PC7 we should be able to ping the container in the cluster in the same network.
Recall that you have created containers in the cluster.
ct-101:~# ifconfig eth0 Link encap:Ethernet HWaddr BC:24:11:7C:D9:C7 inet addr:10.100.1.10 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::be24:11ff:fe7c:d9c7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9216 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:9 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:656 (656.0 B) TX bytes:726 (726.0 B)
On PC7:
pc7#sh ip int bri Interface IP-Address OK? Method Status Protocol FastEthernet0/0 10.100.1.100 YES NVRAM up up FastEthernet0/1 unassigned YES NVRAM administratively down down pc7#sh ip route Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route Gateway of last resort is 10.100.1.1 to network 0.0.0.0 10.0.0.0/24 is subnetted, 1 subnets C 10.100.1.0 is directly connected, FastEthernet0/0 S* 0.0.0.0/0 [1/0] via 10.100.1.1 pc7# pc7#ping 10.100.1.10 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.100.1.10, timeout is 2 seconds: .!!!!
As you can see we can ping all the way the container in the cluster.
Now to test access from the tenants. From tna:
tna#sh ip int bri Interface IP-Address OK? Method Status Protocol FastEthernet0/0 192.168.0.10 YES NVRAM up up FastEthernet0/1 172.16.10.1 YES NVRAM up up Loopback0 176.16.1.1 YES NVRAM up up tna#ping 10.100.1.10 source 172.16.10.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.100.1.10, timeout is 2 seconds: Packet sent with a source address of 172.16.10.1 !!!!!
Notice we nee to source the packet from the correct interface.
From the container I could not ping the tenant, it was driving me crazy.
However, I decided to test some other way, so I created a user “cisco priv 15 password cisco” in tna then allow “login local” and “transport input all” just for testing.
From container:
ct-101:~# telnet 172.16.10.1 Connected to 172.16.10.1 Entering character mode Escape character is '^]'. User Access Verification Username: cisco Password: tna#
I can connect! I need to investigate why I cannot ping from the container. It could be security settings in the cluster, if someone has a clue let me know.
Conclusions
There you have it. We can access VLANs in the cluster and attached to a Cisco acting as a leaf. Also externals tenants are able to access devices in the cluster and vice versa.
Hope you enjoyed it.
Ciao.