{"id":639,"date":"2025-11-22T21:07:22","date_gmt":"2025-11-22T21:07:22","guid":{"rendered":"http:\/\/blog.miguelsarmiento.com\/?p=639"},"modified":"2025-11-25T17:07:09","modified_gmt":"2025-11-25T17:07:09","slug":"pve-sdn-nxos-integration-oh-my","status":"publish","type":"post","link":"https:\/\/blog.miguelsarmiento.com\/?p=639","title":{"rendered":"PVE SDN NXOS Integration Oh My!"},"content":{"rendered":"<h1>Introduction<\/h1>\n<p>In the final part of these series, we will integrate a Cisco 9k with a PVE cluster using the previous configurations we did setup.<\/p>\n<p><!--more--><\/p>\n<p>The network diagram is shown in Fig, 1.<\/p>\n<figure id=\"attachment_642\" aria-describedby=\"caption-attachment-642\" style=\"width: 1231px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-sdn-leaf-eBGP.png\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-642 size-full\" src=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-sdn-leaf-eBGP.png\" alt=\"\" width=\"1231\" height=\"640\" srcset=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-sdn-leaf-eBGP.png 1231w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-sdn-leaf-eBGP-300x156.png 300w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-sdn-leaf-eBGP-768x399.png 768w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2025\/11\/pve-sdn-leaf-eBGP-1024x532.png 1024w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/a><figcaption id=\"caption-attachment-642\" class=\"wp-caption-text\">Fig. 1. Network Diagram.<\/figcaption><\/figure>\n<p>The plan is to:<\/p>\n<p style=\"padding-left: 30px;\">1) Create a SDN data center. It will consists of:<\/p>\n<p style=\"padding-left: 60px;\">a) A PVE cluster<\/p>\n<p style=\"padding-left: 60px;\">b) An OSPF manual fabric,<\/p>\n<p style=\"padding-left: 60px;\">c) An EVPN controller with a Cisco 9000v as a leaf.<\/p>\n<p style=\"padding-left: 60px;\">d) An eBGP controller to announce routes to external BGP peers.<\/p>\n<p style=\"padding-left: 60px;\">e) Two tenants that are part of the SDN environment and also reside externally.<\/p>\n<h1>Configurations<\/h1>\n<h2>PVE<\/h2>\n<p>Configure a cluster. We will create a manual fabric. See the previous posts for why we do this.<\/p>\n<p>Create a bridge on each node (vmbr1) and a assign IP addresses per the network diagram. This is the network that the cluster nodes will use to communicate.<\/p>\n<p class=\"western\" align=\"left\">Create a file and named it &#8220;frr&#8221; in \u201c\/etc\/default\/\u201d it should look like this:<\/p>\n<pre class=\"western\"><b>root@pve-3:~# cat \/etc\/default\/frr<\/b>\r\n<b>ospfd=yes<\/b><\/pre>\n<p class=\"western\" align=\"left\">Do this on all nodes and make sure to use the correct IP addresses.<\/p>\n<p class=\"western\" align=\"left\">Under \u201c\/etc\/frr\/\u201d create \u201cfrr.conf\/local\u201d, add the following to it doing it on all nodes (use the respective &#8220;lo&#8221; IP create above):<\/p>\n<pre class=\"western\"><b>interface lo<\/b>\r\n <b>ip ospf area 0<\/b>\r\n<b>exit<\/b>\r\n<b>!<\/b>\r\n<b>interface vmbr1<\/b>\r\n <b>ip ospf area 0<\/b>\r\n<b>exit<\/b>\r\n<b>!<\/b>\r\n<b>router ospf<\/b>\r\n <b>ospf router-id 10.10.10.3<\/b>\r\n<b>exit<\/b><\/pre>\n<p>Create an EVPN controller:<\/p>\n<p style=\"padding-left: 30px;\">1) Use the IP address configured for the bridge you created. Do not use the loopback interfaces. The loopback interfaces are in used for the fabric and for connectivity to the Cisco devices via OSPF. Add the IP of the Cisco leaf to the controller.<\/p>\n<p style=\"padding-left: 30px;\">2) Apply your SDN settings. The new configuration for &#8220;frr&#8221; should be merged and you should have OSPF peering between the nodes. If there are issues fix them. Each node should have an additional IP for the loopback interface.<\/p>\n<p style=\"padding-left: 30px;\">3) Create zones and VNETS:<\/p>\n<p style=\"padding-left: 60px;\">a)\u00a0 \u00a0Zones v100 (tag 30000) and v200 (tag 40000). Make all nodes part of the exit nodes (for SNAT), make PX3 the preferred exit node for BGP.<\/p>\n<p style=\"padding-left: 60px;\">b) Create VNETs for each zone: test1 10.100.1.0\/24:tag 31000 (GW 10.100.1.1), test2: 10.200.1.0\/24:tag 41000\u00a0 (GW 10.200.1.1).<\/p>\n<p style=\"padding-left: 60px;\">c) Use AS 65000 (default). Remember the tags giving these are important and are needed by the Cisco leaf.<\/p>\n<p style=\"padding-left: 60px;\">e) Create a BGP controller (eBGP) and use 192.168.1.20 (per the network diagram) as the peer. Use AS 65000, this is confusing since it is the AS of the cluster not the AS of the client (which will be 65100). When we configure the 9k the cluster will accept 65100 and peer with it.<\/p>\n<p>Apply the settings for SDN. If you do not have any errors, create your containers in each zone making sure they can ping each other.<\/p>\n<h1>Cisco<\/h1>\n<h2>Leaf A<\/h2>\n<p>The main parts of the configuration for the leaf are shown below.<\/p>\n<pre class=\"western\">  <b>allow feature-set mpls<\/b>\r\n  <b>limit-resource vlan minimum 16 maximum 4094<\/b>\r\n  <b>limit-resource vrf minimum 2 maximum 4096<\/b>\r\n  <b>limit-resource port-channel minimum 0 maximum 511<\/b>\r\n  <b>limit-resource u4route-mem minimum 248 maximum 248<\/b>\r\n  <b>limit-resource u6route-mem minimum 96 maximum 96<\/b>\r\n  <b>limit-resource m4route-mem minimum 58 maximum 58<\/b>\r\n  <b>limit-resource m6route-mem minimum 8 maximum 8<\/b>\r\n  <b>limit-resource vni_bd minimum 4096 maximum 4096<\/b>\r\n\r\n<b>nv overlay evpn<\/b>\r\n<b>feature ospf<\/b>\r\n<b>feature bgp<\/b>\r\n<b>feature pim<\/b>\r\n<b>feature fabric forwarding<\/b>\r\n<b>feature interface-vlan<\/b>\r\n<b>feature vn-segment-vlan-based<\/b>\r\n<b>feature nv overlay<\/b>\r\n<\/pre>\n<pre class=\"western\"><b>fabric forwarding anycast-gateway-mac 0000.dc01.dc01<\/b>\r\n<b>vlan 1,100,200,1000,2000<\/b>\r\n<b>vlan 100<\/b>\r\n  <b>vn-segment 31000<\/b>\r\n<b>vlan 200<\/b>\r\n  <b>vn-segment 41000<\/b>\r\n<b>vlan 1000<\/b>\r\n  <b>vn-segment 30000<\/b>\r\n<b>vlan 2000<\/b>\r\n  <b>vn-segment 40000<\/b>\r\n\r\n<b>vrf context management<\/b>\r\n<b>vrf context vrfvx_v100<\/b>\r\n  <b>vni 30000<\/b>\r\n  <b>rd auto<\/b>\r\n  <b>address-family ipv4 unicast<\/b>\r\n    <b>route-target both auto<\/b>\r\n    <b>route-target both auto evpn<\/b>\r\n<b>vrf context vrfvx_v200<\/b>\r\n  <b>vni 40000<\/b>\r\n  <b>address-family ipv4 unicast<\/b>\r\n    <b>route-target both auto<\/b>\r\n    <b>route-target both auto evpn<\/b>\r\n<b>hardware access-list tcam region racl 512<\/b>\r\n<b>hardware access-list tcam region arp-ether 256<\/b><\/pre>\n<p>After the last two commands are issued you may need to reboot the 9k, do so.<\/p>\n<p>Now we configure the interfaces.<\/p>\n<pre class=\"western\"><b>interface Vlan1000<\/b>\r\n  <b>no shutdown<\/b>\r\n  <b>mtu 9216<\/b>\r\n  <b>mac-address 0000.9999.9999 <\/b>\r\n  <b>vrf member vrfvx_v100<\/b>\r\n  <b>ip forward<\/b>\r\n  <b>fabric forwarding mode anycast-gateway<\/b>\r\n\r\n<b>interface Vlan2000<\/b>\r\n  <b>no shutdown<\/b>\r\n  <b>mtu 9216<\/b>\r\n  <b>mac-address 0000.8888.8888 <\/b>\r\n  <b>vrf member vrfvx_v200<\/b>\r\n  <b>ip forward<\/b>\r\n  <b>fabric forwarding mode anycast-gateway<\/b>\r\n\r\n<b>interface nve1<\/b>\r\n  <b>no shutdown<\/b>\r\n  <b>description to-pve<\/b>\r\n  <b>host-reachability protocol bgp<\/b>\r\n  <b>source-interface loopback0<\/b>\r\n  <b>member vni 30000 associate-vrf<\/b>\r\n  <b>member vni 31000<\/b>\r\n    <b>suppress-arp<\/b>\r\n    <b>ingress-replication protocol bgp<\/b>\r\n  <b>member vni 40000 associate-vrf<\/b>\r\n  <b>member vni 41000<\/b>\r\n    <b>suppress-arp<\/b>\r\n    <b>ingress-replication protocol bgp<\/b>\r\n<\/pre>\n<p>Note that while we map VLANs 31000 and 32000, the magic is done by creating VLANs 1000 and 2000, These interfaces even though do not have an IP are forwarding &#8220;l2vpn evpn&#8221; packets so L2 and L3 information can flow between the devices.<\/p>\n<p>The NVE interface that will carry VXLAN info is configured to use BGP ingress replication and have as it source the loopback interface created.<\/p>\n<p>The interface that talks to the cluster.<\/p>\n<pre class=\"western\"><b>interface Ethernet1\/1<\/b>\r\n  <b>no switchport<\/b>\r\n  <b>mac-address 0000.1111.1111<\/b>\r\n  <b>ip address 192.168.1.10\/24<\/b>\r\n  <b>ip router ospf MSG area 0.0.0.0<\/b>\r\n  <b>no shutdown<\/b><\/pre>\n<p>Finally the interfaces towards tenant A and tenant B,<\/p>\n<pre class=\"western\"><b>interface Ethernet1\/4<\/b>\r\n  <b>switchport access vlan 100<\/b>\r\n\r\n<b>interface Ethernet1\/5<\/b>\r\n  <b>switchport access vlan 200<\/b><\/pre>\n<p>Do not forget the loopback address.<\/p>\n<pre class=\"western\"><b>interface loopback0<\/b>\r\n  <b>ip address 10.10.10.10\/32<\/b>\r\n  <b>ip router ospf MSG area 0.0.0.0<\/b><\/pre>\n<pre class=\"western\"><b>router ospf MSG<\/b>\r\n  <b>router-id 10.10.10.10<\/b><\/pre>\n<p>At this point the Cisco leaf should be peering via OSPF to the PVE cluster.<\/p>\n<p>Now for the L2VPN EVPN configuration.<\/p>\n<pre class=\"western\"><b>router bgp 65000<\/b>\r\n  <b>router-id 10.10.10.10<\/b>\r\n  <b>log-neighbor-changes<\/b>\r\n  <b>address-family l2vpn evpn<\/b>\r\n  <b>neighbor 192.168.1.41<\/b>\r\n    <b>remote-as 65000<\/b>\r\n    <b>log-neighbor-changes<\/b>\r\n    <b>address-family ipv4 unicast<\/b>\r\n      <b>send-community<\/b>\r\n      <b>send-community extended<\/b>\r\n    <b>address-family l2vpn evpn<\/b>\r\n      <b>send-community<\/b>\r\n      <b>send-community extended<\/b>\r\n  <b>neighbor 192.168.1.42<\/b>\r\n    <b>remote-as 65000<\/b>\r\n    <b>log-neighbor-changes<\/b>\r\n    <b>address-family ipv4 unicast<\/b>\r\n      <b>send-community<\/b>\r\n      <b>send-community extended<\/b>\r\n    <b>address-family l2vpn evpn<\/b>\r\n      <b>send-community<\/b>\r\n      <b>send-community extended<\/b>\r\n  <b>neighbor 192.168.1.43<\/b>\r\n    <b>remote-as 65000<\/b>\r\n    <b>log-neighbor-changes<\/b>\r\n    <b>address-family ipv4 unicast<\/b>\r\n      <b>send-community<\/b>\r\n      <b>send-community extended<\/b>\r\n    <b>address-family l2vpn evpn<\/b>\r\n      <b>send-community<\/b>\r\n      <b>send-community extended<\/b>\r\n<b>evpn<\/b>\r\n  <b>vni 31000 l2<\/b>\r\n    <b>rd auto<\/b>\r\n    <b>route-target import auto<\/b>\r\n    <b>route-target export auto<\/b>\r\n  <b>vni 41000 l2<\/b>\r\n    <b>rd auto<\/b>\r\n    <b>route-target import auto<\/b>\r\n    <b>route-target export auto<\/b>\r\n<b>!<\/b>\r\n<\/pre>\n<p>The NVE interface should have peered with the cluster. Double check that the VNI definitions match between the cluster and the Cisco.<\/p>\n<h2>eBGP<\/h2>\n<p>Below are the tidbits concerning the eBGP peering to the external tenants. While we can use any device that can do BGP, we are using a Cisco 9000v for this.<\/p>\n<p>First the VRFs so we can leak routes from the global routing table (GRT) and each tenant.<\/p>\n<pre class=\"western\"><b>ip prefix-list tna seq 10 permit 10.100.1.0\/24 <\/b>\r\n<b>ip prefix-list tna-export seq 10 permit 172.16.10.0\/24 <\/b>\r\n<b>ip prefix-list tna-export seq 20 permit 192.168.8.0\/30 <\/b>\r\n<b>ip prefix-list tnb seq 20 permit 10.200.1.0\/24 <\/b>\r\n<b>ip prefix-list tnb-export seq 10 permit 172.16.20.0\/24 <\/b>\r\n<b>ip prefix-list tnb-export seq 20 permit 192.168.12.0\/30 <\/b>\r\n<b>route-map tna-export permit 10<\/b>\r\n  <b>match ip address prefix-list tna-export <\/b>\r\n<b>route-map tna-import permit 10<\/b>\r\n  <b>match ip address prefix-list tna <\/b>\r\n<b>route-map tnb-export permit 10<\/b>\r\n  <b>match ip address prefix-list tnb-export <\/b>\r\n<b>route-map tnb-import permit 10<\/b>\r\n  <b>match ip address prefix-list tnb <\/b>\r\n<b>vrf context management<\/b>\r\n<b>vrf context tna<\/b>\r\n  <b>rd 10.10.10.20:100<\/b>\r\n  <b>address-family ipv4 unicast<\/b>\r\n    <b>route-target import 10.10.10.20:100<\/b>\r\n    <b>route-target export 10.10.10.20:100<\/b>\r\n    <b>import vrf default map tna-import<\/b>\r\n    <b>export vrf default map tna-export<\/b>\r\n<b>vrf context tnb<\/b>\r\n  <b>rd 10.10.10.10:200<\/b>\r\n  <b>address-family ipv4 unicast<\/b>\r\n    <b>route-target import 10.10.10.20:200<\/b>\r\n    <b>route-target export 10.10.10.20:200<\/b>\r\n    <b>import vrf default map tnb-import<\/b>\r\n    <b>export vrf default map tnb-export<\/b><\/pre>\n<p>Now for the interfaces.<\/p>\n<pre class=\"western\"><b>interface Ethernet1\/1<\/b>\r\n  <b>no switchport<\/b>\r\n  <b>mac-address 0000.1111.4444<\/b>\r\n  <b>ip address 192.168.1.20\/24<\/b>\r\n  <b>no shutdown<\/b><\/pre>\n<p>Above is the IP address you configured for the BGP controller in the cluster.<\/p>\n<pre class=\"western\"><b>interface Ethernet1\/6<\/b>\r\n  <b>no switchport<\/b>\r\n  <b>mac-address 0000.9999.1234<\/b>\r\n  <b>vrf member tna<\/b>\r\n  <b>ip address 192.168.0.9\/30<\/b>\r\n  <b>no shutdown<\/b>\r\n\r\n<b>interface Ethernet1\/7<\/b>\r\n  <b>no switchport<\/b>\r\n  <b>mac-address 0000.3333.1234<\/b>\r\n  <b>vrf member tnb<\/b>\r\n  <b>ip address 192.168.0.13\/30<\/b>\r\n  <b>no shutdown<\/b><\/pre>\n<p>Above are the links towards each tenant.<\/p>\n<pre class=\"western\"><b>interface loopback0<\/b>\r\n  <b>ip address 10.10.10.20\/32<\/b><\/pre>\n<p>Finally eBGP with PX3 in the cluster.<\/p>\n<pre class=\"western\"><b>router bgp 65100<\/b>\r\n  <b>router-id 10.10.10.20<\/b>\r\n  <b>neighbor 192.168.1.43<\/b>\r\n    <b>remote-as 65000<\/b>\r\n    <b>address-family ipv4 unicast<\/b>\r\n      <b>soft-reconfiguration inbound<\/b>\r\n  <b>vrf tna<\/b>\r\n    <b>address-family ipv4 unicast<\/b>\r\n      <b>network 10.100.2.0\/24<\/b>\r\n    <b>neighbor 192.168.0.10<\/b>\r\n      <b>remote-as 64000<\/b>\r\n      <b>address-family ipv4 unicast<\/b>\r\n        <b>send-community<\/b>\r\n        <b>send-community extended<\/b>\r\n  <b>vrf tnb<\/b>\r\n    <b>address-family ipv4 unicast<\/b>\r\n      <b>network 10.200.1.0\/24<\/b>\r\n    <b>neighbor 192.168.0.14<\/b>\r\n      <b>remote-as 63000<\/b>\r\n      <b>address-family ipv4 unicast<\/b>\r\n        <b>send-community<\/b>\r\n        <b>send-community extended<\/b>\r\n<b>!<\/b><\/pre>\n<p>Some remarks.<\/p>\n<p style=\"padding-left: 30px;\">1)\u00a0 We have tenants tna and tnb defined.<\/p>\n<p style=\"padding-left: 30px;\">2) We need to leak routes between the GRT and each tenant. We also need to make sure we do not leak routes between tenant thus the prefix lists and route maps created.<\/p>\n<p style=\"padding-left: 30px;\">3) As mentioned before the cluster will accept connections and peer itself, this is not an issue if you control all legs of the network otherwise BGP authentication is a must.<\/p>\n<p style=\"padding-left: 30px;\">4) We are using VRFs for connecting to external clients however, in most production environments you would use instead BGP MPLS (which creates a VPN network).\u00a0 In this case the configuration would need to be changed to use the VPNV4 address family command set.<\/p>\n<h1>Verification<\/h1>\n<p>We are now ready to verify that we can connect from all tenants into the cluster.<\/p>\n<h2>PVE<\/h2>\n<p>Using &#8220;vtysh&#8221; at a command prompt, in this case at PX3.<\/p>\n<pre>pve-3# sh bgp l2vpn evpn summary\r\nBGP router identifier 192.168.1.43, local AS number 65000 VRF default vrf-id 0\r\nBGP table version 0\r\nRIB entries 31, using 3968 bytes of memory\r\nPeers 3, using 70 KiB of memory\r\nPeer groups 2, using 128 bytes of memory\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd PfxSnt Desc\r\n192.168.1.10 4 65000 23385 23416 40 0 0 19:30:35 4 6 N\/A\r\npve-1(192.168.1.41) 4 65000 23547 23550 40 0 0 19:36:52 2 6 FRRouting\/10.3.1\r\npve-2(192.168.1.42) 4 65000 23547 23550 40 0 0 19:36:52 2 6 FRRouting\/10.3.1\r\n\r\nTotal number of neighbors 3<\/pre>\n<p>As you can see we have peering with the other members of the cluster but also to the Cisco (k (192.168.1.10).<\/p>\n<pre>\u00a0pve-3# sh bgp l2vpn evpn 192.168.1.10\r\nBGP routing table entry for 192.168.1.43:2:[5]:[0]:[0]:[::]\r\nPaths: (1 available, best #1)\r\n Advertised to non peer-group peers:\r\n 192.168.1.10 pve-1(192.168.1.41) pve-2(192.168.1.42)\r\n Route [5]:[0]:[0]:[::] VNI 30000\r\n Local\r\n 192.168.1.43(pve-3) from 0.0.0.0 (192.168.1.43)\r\n Origin IGP, weight 32768, valid, sourced, local, bestpath-from-AS Local, best (First path received)\r\n Extended Community: ET:8 RT:65000:30000 Rmac:f6:9c:57:01:a9:f8\r\n Last update: Fri Nov 21 15:35:19 2025\r\nBGP routing table entry for 192.168.1.43:3:[5]:[0]:[0]:[::]\r\nPaths: (1 available, best #1)\r\n Advertised to non peer-group peers:\r\n 192.168.1.10 pve-1(192.168.1.41) pve-2(192.168.1.42)\r\n Route [5]:[0]:[0]:[::] VNI 40000\r\n Local\r\n 192.168.1.43(pve-3) from 0.0.0.0 (192.168.1.43)\r\n Origin IGP, weight 32768, valid, sourced, local, bestpath-from-AS Local, best (First path received)\r\n Extended Community: ET:8 RT:65000:40000 Rmac:22:54:28:7a:6d:32\r\n Last update: Fri Nov 21 15:35:19 2025<\/pre>\n<p>Above shows the type 5 BGP routes exchanged with the 9k. Below we can see that we are receiving routes via BGP and in particular from tenants tna and tnb. These routes will also show in the routing table if you do &#8220;show ip route&#8221;.<\/p>\n<pre>pve-3# sh ip route bgp\r\n\r\nB&gt;* 10.100.1.0\/24 [20\/0] is directly connected, vrf_v100 (vrf vrf_v100), weight 1, 19:44:17\r\nB&gt;* 10.200.1.0\/24 [20\/0] is directly connected, vrf_v200 (vrf vrf_v200), weight 1, 19:44:17\r\nB&gt;* 172.16.10.0\/24 [20\/0] via 192.168.1.20, vmbr1, weight 1, 19:38:45\r\nB&gt;* 172.16.20.0\/24 [20\/0] via 192.168.1.20, vmbr1, weight 1, 19:38:45<\/pre>\n<h2>Cisco 9k<\/h2>\n<p>We show the neighbors we have.<\/p>\n<pre>nxos-9k# sh bgp l2vpn evpn summary \r\nBGP summary information for VRF default, address family L2VPN EVPN\r\nBGP router identifier 10.10.10.10, local AS number 65000\r\nBGP table version is 68, L2VPN EVPN config peers 3, capable peers 3\r\n28 network entries and 32 paths using 5632 bytes of memory\r\nBGP attribute entries [18\/3096], BGP AS path entries [0\/0]\r\nBGP community entries [0\/0], BGP clusterlist entries [0\/0]\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd\r\n192.168.1.41 4 65000 23712 23658 68 0 0 19:45:22 4 \r\n192.168.1.42 4 65000 23715 23661 68 0 0 19:45:29 4 \r\n192.168.1.43 4 65000 23713 23659 68 0 0 19:45:25 6<\/pre>\n<p>As you can see we are peering with all members of the cluster. Other BGP commands will show information similar to what you saw for the cluster.<\/p>\n<h2>eBGP<\/h2>\n<p>This standard BGP and you should see something like:<\/p>\n<pre>bgp-1# sh ip bgp\r\nBGP routing table information for VRF default, address family IPv4 Unicast\r\nBGP table version is 6, Local Router ID is 10.10.10.20\r\nStatus: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, &gt;-best\r\nPath type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i\r\nnjected\r\nOrigin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, &amp; - backup, 2 - b\r\nest2\r\n\r\nNetwork Next Hop Metric LocPrf Weight Path\r\n*&gt;e10.100.1.0\/24 192.168.1.43 0 0 65000 ?\r\n*&gt;e10.200.1.0\/24 192.168.1.43 0 0 65000 ?\r\n*&gt;e172.16.10.0\/24 192.168.0.10 0 0 64000 i\r\n*&gt;e172.16.20.0\/24 192.168.0.14 0 0 63000 i\r\n\r\nbgp-1# sh ip bgp summary \r\nBGP summary information for VRF default, address family IPv4 Unicast\r\nBGP router identifier 10.10.10.20, local AS number 65100\r\nBGP table version is 6, IPv4 Unicast config peers 1, capable peers 1\r\n4 network entries and 4 paths using 736 bytes of memory\r\nBGP attribute entries [3\/516], BGP AS path entries [3\/18]\r\nBGP community entries [0\/0], BGP clusterlist entries [0\/0]\r\n\r\nNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd\r\n192.168.1.43 4 65000 23813 23762 6 0 0 19:50:22 2<\/pre>\n<p>We are exchanging routes.<\/p>\n<p>Now from PC7 we should be able to ping the container in the cluster in the same network.<\/p>\n<p>Recall that you have created containers in the cluster.<\/p>\n<pre>ct-101:~# ifconfig\r\neth0 Link encap:Ethernet HWaddr BC:24:11:7C:D9:C7 \r\n inet addr:10.100.1.10 Bcast:0.0.0.0 Mask:255.255.255.0\r\n inet6 addr: fe80::be24:11ff:fe7c:d9c7\/64 Scope:Link\r\n UP BROADCAST RUNNING MULTICAST MTU:9216 Metric:1\r\n RX packets:8 errors:0 dropped:0 overruns:0 frame:0\r\n TX packets:9 errors:0 dropped:0 overruns:0 carrier:0\r\n collisions:0 txqueuelen:1000 \r\n RX bytes:656 (656.0 B) TX bytes:726 (726.0 B)<\/pre>\n<p>On PC7:<\/p>\n<pre>pc7#sh ip int bri\r\nInterface IP-Address OK? Method Status Protocol\r\nFastEthernet0\/0 10.100.1.100 YES NVRAM up up \r\nFastEthernet0\/1 unassigned YES NVRAM administratively down down \r\npc7#sh ip route \r\nCodes: C - connected, S - static, R - RIP, M - mobile, B - BGP\r\n D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area \r\n N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2\r\n E1 - OSPF external type 1, E2 - OSPF external type 2\r\n i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2\r\n ia - IS-IS inter area, * - candidate default, U - per-user static route\r\n o - ODR, P - periodic downloaded static route\r\n\r\nGateway of last resort is 10.100.1.1 to network 0.0.0.0\r\n\r\n10.0.0.0\/24 is subnetted, 1 subnets\r\nC 10.100.1.0 is directly connected, FastEthernet0\/0\r\nS* 0.0.0.0\/0 [1\/0] via 10.100.1.1\r\npc7#\r\npc7#ping 10.100.1.10\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.100.1.10, timeout is 2 seconds:\r\n.!!!!<\/pre>\n<p>As you can see we can ping all the way the container in the cluster.<\/p>\n<p>Now to test access from the tenants. From tna:<\/p>\n<pre>tna#sh ip int bri \r\nInterface IP-Address OK? Method Status Protocol\r\nFastEthernet0\/0 192.168.0.10 YES NVRAM up up \r\nFastEthernet0\/1 172.16.10.1 YES NVRAM up up \r\nLoopback0 176.16.1.1 YES NVRAM up up \r\n\r\ntna#ping 10.100.1.10 source 172.16.10.1\r\n\r\nType escape sequence to abort.\r\nSending 5, 100-byte ICMP Echos to 10.100.1.10, timeout is 2 seconds:\r\nPacket sent with a source address of 172.16.10.1 \r\n!!!!!<\/pre>\n<p>Notice we nee to source the packet from the correct interface.<\/p>\n<p>From the container I could not ping the tenant, it was driving me crazy.<\/p>\n<p>However, I decided to test some other way, so I created a user &#8220;cisco priv 15 password cisco&#8221; in tna then allow &#8220;login local&#8221; and &#8220;transport input all&#8221; just for testing.<\/p>\n<p>From container:<\/p>\n<pre>ct-101:~# telnet 172.16.10.1\r\nConnected to 172.16.10.1\r\n\r\nEntering character mode\r\nEscape character is '^]'.\r\n\r\nUser Access Verification\r\n\r\nUsername: cisco\r\nPassword: \r\ntna#<\/pre>\n<p>I can connect! I need to investigate why I cannot ping from the container. It could be security settings in the cluster, if someone has a clue let me know.<\/p>\n<h1>Conclusions<\/h1>\n<p>There you have it. We can access VLANs in the cluster and attached to a Cisco acting as a leaf. Also externals tenants are able to access devices in the cluster and vice versa.<\/p>\n<p>Hope you enjoyed it.<\/p>\n<p>Ciao.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction In the final part of these series, we will integrate a Cisco 9k with a PVE cluster using the previous configurations we did setup.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-639","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/639","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=639"}],"version-history":[{"count":41,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/639\/revisions"}],"predecessor-version":[{"id":681,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/639\/revisions\/681"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=639"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=639"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=639"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}