{"id":1042,"date":"2026-08-14T17:03:56","date_gmt":"2026-08-14T17:03:56","guid":{"rendered":"https:\/\/blog.miguelsarmiento.com\/?p=1042"},"modified":"2026-08-14T17:05:54","modified_gmt":"2026-08-14T17:05:54","slug":"open-nebula-vxlan-evpn-not-oh-my","status":"publish","type":"post","link":"https:\/\/blog.miguelsarmiento.com\/?p=1042","title":{"rendered":"Open Nebula, VXLAN, EVPN Not, Oh My!"},"content":{"rendered":"\n<p>This is a follow up to my previous <a href=\"https:\/\/blog.miguelsarmiento.com\/?p=934\" data-type=\"post\" data-id=\"934\" target=\"_blank\" rel=\"noreferrer noopener\">post<\/a>.<\/p>\n\n\n\n<p>On that post I did install an almost production Open Nebula cluster. The reason it is almost a production cluster, is because it has some pitfalls. <\/p>\n\n\n\n<!--more-->\n\n\n\n\n\n<h2 id=\"introduction\" class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<p>The network diagram on my previous post, shows for the purpose of the discussion the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A management network (10.x.x.x), the network that EVE-NG uses. It allows me to install and upgrade the servers using the Internet.<\/li>\n\n\n\n<li>A cluster network (172.a.a.a.a), this is the actual network Open Nebula uses when creating images, moving VMs, etc.<\/li>\n\n\n\n<li>Several other networks to allow access to external networks, VXLAN (since VXLAN needs and IP for the purposes of creating a VTEP).<\/li>\n<\/ul>\n\n\n\n<p>The main issue is a follows:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>We create a VXLAN, lets say VM1 needs to talk to VM2 on host2.<\/li>\n\n\n\n<li>VM1 requires the MAC address of VM2, it queries EVPN.<\/li>\n\n\n\n<li>EVPN tells VM1 that the MAC address in question is at 10.x.x.3 (because it is advertising that VNI using a RD). Thus, host1 at 10.x.x.2 opens a VTEP tunnel.<\/li>\n\n\n\n<li>So far so good. Host1 opens a VTEP using 172.a.a.2 (which is one of the interfaces on 10.x.x.2) as source with destination 10.x.x.3 (on host2).<\/li>\n\n\n\n<li>Nothing out of the ordinary here. However you should see an issue here.<\/li>\n\n\n\n<li>After the tunnels are up and both VMs interchange traffic, that traffic is not going via the Open Nebula cluster network, it is going via the management network!<\/li>\n\n\n\n<li>You may think this is not a big deal. Well it is, in a production environment with hundreds of VXLAN networks and probably thousand of VMs, you will overwhelm the network.<\/li>\n\n\n\n<li>While it may have 1 Gigabit throughput, that may not be enough. It should be at least 10G or more.<\/li>\n\n\n\n<li>For a small network not an issue, if you are a cloud provider for example now you have a problem.<\/li>\n<\/ul>\n\n\n\n<p>What do you do.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Move BGP peering from the management network to a dedicated network (the OP network perhaps), and your peering is now on that segment.<\/li>\n\n\n\n<li>In this fashion you are ensuring BGP traffic and VTEP traffic traverse a &#8220;faster&#8221; network.<\/li>\n\n\n\n<li>Create loopback interfaces for BGP peering.<\/li>\n\n\n\n<li>However if you do loopback interfaces, you need to create an underlay and an overlay network.<\/li>\n\n\n\n<li>The underlay network runs either OSPF or IS-IS an ensures that loopback interfaces are always available regardless.<\/li>\n\n\n\n<li>The overlay network then carries BGP traffic. However since you will be using loopback interfaces, you need to suppress the Management network advertisements.<\/li>\n\n\n\n<li>This is done by also peering via IPV4 unicast and only  advertising the OP network, the VXLAN network(s), and any other networks that need to be accessed by VMs, external networks (internet access), etc.<\/li>\n\n\n\n<li>This ensures that management traffic is only for management, updating the servers, accessing Sunstone, etc.<\/li>\n<\/ul>\n\n\n\n<p>Above is reason enough for a follow up. However, did run into an issue with kernel space VXLAN versus user space VXLAN, i.e. using Open-vswitch (OVS).<\/p>\n\n\n\n<p>There are reasons why we would like to use OVS, it is OS agnostic (although you will need to at one point connect to network resources on the local machine), that thought is not much of problem.<\/p>\n\n\n\n<p>It is also fairly scalable and you do not have to deal with Network Manager, netplan, etc.<\/p>\n\n\n\n<p>You can use the iproute2 commands and appropriate rc.local commands ( yeah I know rc.local keeps popping out now and then).<\/p>\n\n\n\n<p>Well while VXLAN and EVPN using kernel space works fine, as shown in my previous post, It does not do so well at least EVPN, VXLAN does work up to a point (not VNI advertising and that is a problem, multicasting is not scalable).<\/p>\n\n\n\n<p>Let me show what I mean and it is reproducible.<\/p>\n\n\n\n<h2 id=\"the-problem\" class=\"wp-block-heading\">The Problem<\/h2>\n\n\n\n<p>The following diagram shows the basic setup.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/OpenNebula-OVS.png\" target=\"_blank\" rel=\" noreferrer noopener\"><img loading=\"lazy\" decoding=\"async\" width=\"1347\" height=\"660\" src=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/OpenNebula-OVS.png\" alt=\"\" class=\"wp-image-1059\" srcset=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/OpenNebula-OVS.png 1347w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/OpenNebula-OVS-300x147.png 300w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/OpenNebula-OVS-1024x502.png 1024w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/OpenNebula-OVS-768x376.png 768w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/a><figcaption class=\"wp-element-caption\">Network Diagram<\/figcaption><\/figure>\n\n\n\n<p>Above is simple version of the original post setup.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>One controller, two hypervisors.<\/li>\n\n\n\n<li>No shared storage for simplicity.<\/li>\n\n\n\n<li>ovsbr0 is an OVS bridge with physical interface ens7.<\/li>\n\n\n\n<li>ovsbr0 has an IP so VTEPs can be made for VXLAN. <\/li>\n\n\n\n<li>OP is the cluster network. We use it for BGP peering as discussed earlier.<\/li>\n\n\n\n<li>No loopback interfaces to keep things simple.<\/li>\n\n\n\n<li>VX is used for kernel space VXLANs to compare it with OVS. <\/li>\n<\/ul>\n\n\n\n<p>We installed everything and make sure Open Nebula is up and Sunstone is accessible.<\/p>\n\n\n\n<p>We reconfigure BGP peering. <\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">hostname C1<br>hostname 172.16.10.1<br>!<br>router bgp 64512<br> bgp router-id 172.16.10.1<br> no bgp default ipv4-unicast<br> bgp cluster-id 172.16.10.1<br> neighbor fabric peer-group<br> neighbor fabric remote-as 64512<br> neighbor fabric update-source 172.16.10.1<br> neighbor fabric capability extended-nexthop<br> bgp listen range 172.16.10.0\/24 peer-group fabric<br> !<br> address-family l2vpn evpn<br>  neighbor fabric activate<br>  neighbor fabric route-reflector-client<br>  advertise-all-vni<br>  advertise-svi-ip<br> exit-address-family<br>exit<\/pre>\n\n\n\n<p>We do the same on the hypervisors.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">C1# sh bgp sum<br><br>L2VPN EVPN Summary:<br>BGP router identifier 172.16.10.1, local AS number 64512 VRF default vrf-id 0<br>BGP table version 0<br>RIB entries 3, using 384 bytes of memory<br>Peers 2, using 47 KiB of memory<br>Peer groups 1, using 64 bytes of memory<br><br>Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up\/Down State\/PfxRcd   PfxSnt Desc<br>*172.16.10.2    4      64512       238       265       33    0    0 03:25:39            3        4 FRRouting\/10.5.3<br>*172.16.10.3    4      64512       235       265       33    0    0 03:25:39            1        4 FRRouting\/10.5.3<br><br>Total number of neighbors 2<br>* - dynamic neighbor<br>2 dynamic neighbor(s), limit 100<\/pre>\n\n\n\n<p>As we can see, we BGP peering is up and we are exchanging routes.<\/p>\n\n\n\n<h3 id=\"vxlans-using-kernel-space\" class=\"wp-block-heading\"><em>VXLANs using Kernel Space<\/em><\/h3>\n\n\n\n<p>We then create a VXLAN Network.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Normal VXLAN.<\/li>\n\n\n\n<li>We force the virtual network to use EVPN when configuring it (foreshadowing!)<\/li>\n\n\n\n<li>Finish the network.<\/li>\n<\/ul>\n\n\n\n<p>We then create a couple of VMs on the network we just created (VLAN 500 or VNI 500).<\/p>\n\n\n\n<p>We look into BGP:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">C1# sh bgp l2vpn evpn route<br>BGP table version is 34, local router ID is 172.16.10.1<br>Status codes: s suppressed, d damped, h history, * valid, &gt; best, i - internal<br>Origin codes: i - IGP, e - EGP, ? - incomplete<br>EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]<br>EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]<br>EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]<br>EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]<br>EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]<br><br>   Network          Next Hop            Metric LocPrf Weight Path<br>                    Extended Community<br>Route Distinguisher: 172.16.10.2:2<br> *&gt;i [2]:[0]:[48]:[0a:3e:d7:18:94:9a]:[128]:[fe80::346d:90ff:fe72:49cd]<br>                    172.16.10.2                   100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [2]:[0]:[48]:[1a:3f:74:30:e3:c9]:[128]:[fe80::346d:90ff:fe72:49cd]<br>                    172.16.10.2                   100      0 i<br>                    RT:64512:500 ET:8<br>Route Distinguisher: 172.16.10.3:2<br> *&gt;i [2]:[0]:[48]:[56:b1:de:25:69:7a]:[128]:[fe80::c060:95ff:fe89:b05b]<br>                    172.16.10.3                   100      0 i<br>                    RT:64512:500 ET:8<br><br>Displayed 3 prefixes (3 paths)<\/pre>\n\n\n\n<p>We can see that we are getting VNIs and the RDs are there. They tell either VM where to connect when trying to access each other.<\/p>\n\n\n\n<p>This is the expected outcome.<\/p>\n\n\n\n<h3 id=\"vxlans-using-ovs-user-space\" class=\"wp-block-heading\"><em>VXLANs using OVS user space <\/em><\/h3>\n\n\n\n<p>Once again we create a VXLAN Network.<\/p>\n\n\n\n<p>We choose OVS VXLAN for the network type.<\/p>\n\n\n\n<p>However, we do not have a way of telling we want EVPN, we can choose the bridge we configure for OVS and the VNI (1000 in this case).<\/p>\n\n\n\n<p>We then create a couple of VMS in this network.<\/p>\n\n\n\n<p>They can talk to each other however as you can see from above not VNIs are being advertised!<\/p>\n\n\n\n<p>If you are puzzle so am I.<\/p>\n\n\n\n<p>To the Open Nebula docs we go.<\/p>\n\n\n\n<p>The VXLAN configuration is done according to their steps.<\/p>\n\n\n\n<p>According to their documentation, you need to update:<\/p>\n\n\n\n<p>&#8220;\/var\/lib\/one\/remotes\/etc\/vnm\/OpenNebulaNetwork.conf&#8221;<\/p>\n\n\n\n<p>with the following:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"># Multicast protocol for multi destination BUM traffic. Options:<br>#   - multicast, for IP multicast<br>#   - evpn, for BGP EVPN control plane<br>:vxlan_mode: evpn<br><br># Tunnel endpoint communication type. Only for evpn vxlan_mode.<br>#   - dev, tunnel endpoint communication is sent to PHYDEV<br>#   - local_ip, first ip addr of PHYDEV is used as address for the communiation<br>:vxlan_tep: local_ip<br><br># Additional ip link options, uncomment the following to disable learning for<br># EVPN mode<br>:ip_link_conf:<br>    :nolearning:<\/pre>\n\n\n\n<p>Guess what, as soon as I synched the configurations I was not only not getting advertisements for the OVS VXLAN (the kernel wise still worked), OVS VXLAN stopped working. I could not ping between them!<\/p>\n\n\n\n<p>So after reverting the configuration, everything starting working.<\/p>\n\n\n\n<p>After a careful view of the documentation, I did realize they talk about EVPN with kernel space VXLANs, the talk about OVS and VXLAN but no mention of OVS VXLAN with EVPN.<\/p>\n\n\n\n<p>Unless someone can set me straight, it seems that MP-BGP EVPN is only supported for kernel space VXLANS (in addition to multicasting).<\/p>\n\n\n\n<p>For OVS VXLAN only multicasting seems to work. Is this is the case that is a pity.<\/p>\n\n\n\n<h2 id=\"conclusions\" class=\"wp-block-heading\">Conclusions<\/h2>\n\n\n\n<p>Unless someone corrects me, drop me a line, it is a pity as I said.<\/p>\n\n\n\n<p>Not a deal breaker in my opinion however it means that you have less flexibility.<\/p>\n\n\n\n<p>This because MP-BGP EVPN could allow to you to integrate with other SDN implementations of VXLAN, like Cisco, Proxmox SDN, VyOS, etc.<\/p>\n\n\n\n<p>This would allow you to extend your fabric which is one of the reasons to use Open Nebula, Open Stack or Proxmox. Without this, it can be done but you will have to do a much more complex configuration.<\/p>\n\n\n\n<p>There you have it.<\/p>\n\n\n\n<p>Ciao.  <\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a follow up to my previous post. On that post I did install an almost production Open Nebula cluster. The reason it is almost a production cluster, is because it has some pitfalls.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1042","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/1042","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1042"}],"version-history":[{"count":50,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/1042\/revisions"}],"predecessor-version":[{"id":1094,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/1042\/revisions\/1094"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1042"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1042"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1042"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}