{"id":934,"date":"2026-08-12T17:17:05","date_gmt":"2026-08-12T17:17:05","guid":{"rendered":"https:\/\/blog.miguelsarmiento.com\/?p=934"},"modified":"2026-08-12T19:41:49","modified_gmt":"2026-08-12T19:41:49","slug":"opennebula-cluster-iso-install-oh-my","status":"publish","type":"post","link":"https:\/\/blog.miguelsarmiento.com\/?p=934","title":{"rendered":"OpenNebula Cluster, ISO install, oh My!"},"content":{"rendered":"\n<p>In this blog I will show how to install an OpenNebula cluster from scratch, well almost, using an ISO that allows you to create a minimal configuration. The initial setup can be extended to have an almost functional production setup.<\/p>\n\n\n\n<!--more-->\n\n\n\n\n\n<h2 id=\"motivation\" class=\"wp-block-heading\">Motivation<\/h2>\n\n\n\n<p>In the past I have used the so called all-in-one setups for both OpenNebula and OpenStack.<\/p>\n\n\n\n<p>While they do work fine, you need to setup a production like system. It actually teaches you the ins and outs of the setup. I will eventually have a blog on how to do this for OpenStack.<\/p>\n\n\n\n<p>But first OpenNebula, I was in the process of starting a manual installation, i.e. get the services, install them, create whatever databases were needed, configure networking, etc.<\/p>\n\n\n\n<p>However, came across an ISO install that OpenNebula recommends to use for lab purposes.<\/p>\n\n\n\n<p>Fairly quickly I realized it could be use for a more production like setup.<\/p>\n\n\n\n<h2 id=\"requirements\" class=\"wp-block-heading\">Requirements<\/h2>\n\n\n\n<p>Not many people have several servers lying around so I am using EVE-NG.<\/p>\n\n\n\n<p>You need:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Three servers for the cluster controllers. They have 2 GB of memory. In a real production system 16 GB and at least 4 cores should be enough. Remember the controllers do not run any virtual machines or do networking, the hypervisors do.<\/li>\n\n\n\n<li>Two hypervisors nodes, In my case with 8 GB of memory, once again in production you should have at least three hypervisors with 512 GB of memory (1 TB in my opinion since in that case if you do maintenance on one you will still have enough processing power to have the cluster running on two nodes). And at least 16 cores, the more the better.<\/li>\n\n\n\n<li>A NFS server, cluster configuration will not work with local storage, you need shared storage. You can use NFS or an Equalogic or CEPH. My preference is CEPH. Here I alredy have a VM running NFS on EXSI, it was just a matter of connecting EVE-NG to it. If you are curious how to do it, it is fairly simple. EVE-NG has a recipe on their web site.  <\/li>\n\n\n\n<li>The ISO file. Go to <a href=\"https:\/\/docs.opennebula.io\/7.2\/getting_started\/try_opennebula\/opennebula_sandbox_deployment\/deploy_opennebula_onprem_with_poc_iso\/\" target=\"_blank\" rel=\"noreferrer noopener\">&#8220;Deploy OpenNebula On-prem with an ISO&#8221;<\/a>, download the ISO.<\/li>\n\n\n\n<li>Create the virtual machines, of course you can use whatever app you want. For EVE-NG you need to create the ISO on a folder then create a qemu disk of least 40 GB. How to do this in EVE-NG is beyond the scope of this blog, once again EVE-NG has recipes.<\/li>\n\n\n\n<li>Every time you start a new machine it will install Alma Linux and configure the OS.  <\/li>\n<\/ul>\n\n\n\n<p>You need to pay attention on how you would segment your setup and how it wil access the Internet.<\/p>\n\n\n\n<p>On a production system of course, you will have a management network, an external  network, a cluster network and so on.<\/p>\n\n\n\n<p>On EVE-NG you have to be a  bit creative.<\/p>\n\n\n\n<p>Your mangement network is also your Internet or external network.<\/p>\n\n\n\n<p>So we will have the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>ens3 (e0 for EVE-NG) management, this allows us to upgrade the servers. It will eventually server as the network to have access to the Internet.<\/li>\n\n\n\n<li>ens4 (e1) The network to use for VXLAN tunnels. It needs an actual IP address otherwise VXLAN wolud not function.<\/li>\n\n\n\n<li> ens5 (e2) VX network for virtual machines not using VXLAN, this can eventually allow you to talk external switches via 802.1Q trunks for example.<\/li>\n\n\n\n<li>en6 (e3) NFS network to have images, instances etc.<\/li>\n\n\n\n<li>ens7 (e4) The actual cluster network that arbitrates, moves VMs around, creates networks, etc. <\/li>\n<\/ul>\n\n\n\n<p>With above in mind the following is a logical (almost) diagram of the network.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/open-nebula-cluster.png\" target=\"_blank\" rel=\" noreferrer noopener\"><img loading=\"lazy\" decoding=\"async\" width=\"1382\" height=\"757\" src=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/open-nebula-cluster.png\" alt=\"\" class=\"wp-image-933\" srcset=\"https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/open-nebula-cluster.png 1382w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/open-nebula-cluster-300x164.png 300w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/open-nebula-cluster-1024x561.png 1024w, https:\/\/blog.miguelsarmiento.com\/wp-content\/uploads\/2026\/08\/open-nebula-cluster-768x421.png 768w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/a><figcaption class=\"wp-element-caption\">Network Diagram.<\/figcaption><\/figure>\n\n\n\n<p>A few things to consider, the VM, VX and OP are dumb bridges the EVE-NG sets up for communication. In a real setup those will actually be connections to L2\/L3 switches (preferably via trunks for higher throughput).<\/p>\n\n\n\n<h2 id=\"installation\" class=\"wp-block-heading\">Installation<\/h2>\n\n\n\n<p>Now that we have everything in place, we can start.<\/p>\n\n\n\n<h3 id=\"controller\" class=\"wp-block-heading\"><em>Controller<\/em><\/h3>\n\n\n\n<p>We will install a single controller first.<\/p>\n\n\n\n<p>Fire c1, you will be presented with a menu, choose &#8220;POC&#8221; and hit return. <\/p>\n\n\n\n<p>After complaining that you should have more memory, disk space, blah, blah&#8230;., it will install the software needed to run as a controller.<\/p>\n\n\n\n<p> Use root\/0p3nN3bul4 as credentials. Log onto the console and create an additional user with sudo privileges, believe me I got bit by this and had to re-install. Change the root password.<\/p>\n\n\n\n<p>Run &#8220;onefemenu&#8221;, this is a curses like window.<\/p>\n\n\n\n<p>Follow the steps indicated on the OpenNebulas&#8217; ISO web site.<\/p>\n\n\n\n<p>For the first controller you want:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>ens3: 10.229.128.231\/24 gw: 10.229.128.10 (this are my settings please use you own!)<\/li>\n\n\n\n<li>ens4: 172.16.2.1\/24 (no GW)<\/li>\n\n\n\n<li>ens5: No IP address<\/li>\n\n\n\n<li>ens6: 172.168.1.231\/24 (once again this is the network where my NFS is, use your own!)<\/li>\n\n\n\n<li>ens7: 172.16.1.1\/24 (this is the network use for management access, I know it is confusing but the is OpenNebula management access not EVE-NG management access).<\/li>\n<\/ul>\n\n\n\n<p>The app will run several ansible playbooks.<\/p>\n\n\n\n<p> The network should look like below:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# ip add<br>1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<br>    link\/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>    inet 127.0.0.1\/8 scope host lo<br>       valid_lft forever preferred_lft forever<br>    inet6 ::1\/128 scope host<br>       valid_lft forever preferred_lft forever<br>2: ens3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000<br>    link\/ether 00:50:01:00:01:00 brd ff:ff:ff:ff:ff:ff<br>    altname enp0s3<br>    inet 10.229.128.231\/24 brd 10.229.128.255 scope global noprefixroute ens3<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::250:1ff:fe00:100\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<br>3: ens4: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000<br>    link\/ether 00:50:01:00:01:01 brd ff:ff:ff:ff:ff:ff<br>    altname enp0s4<br>    inet 172.16.2.1\/24 brd 172.16.2.255 scope global noprefixroute ens4<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::250:1ff:fe00:101\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<br>4: ens5: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000<br>    link\/ether 00:50:01:00:01:02 brd ff:ff:ff:ff:ff:ff<br>    altname enp0s5<br>5: ens6: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000<br>    link\/ether 00:50:01:00:01:03 brd ff:ff:ff:ff:ff:ff<br>    altname enp0s6<br>    inet 172.168.1.231\/24 brd 172.168.1.255 scope global noprefixroute ens6<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::250:1ff:fe00:103\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<br>6: ens7: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000<br>    link\/ether 00:50:01:00:01:04 brd ff:ff:ff:ff:ff:ff<br>    altname enp0s7<br>    inet 172.16.1.1\/24 brd 172.16.1.255 scope global noprefixroute ens7<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::250:1ff:fe00:104\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<\/pre>\n\n\n\n<p>Next you will need to add hypervisors. It is worth noting that we will diverge from the standard installation. Normally you are using this for an all-in-one setup, in this fashion you add the controller you just configured as a host so it can run VMs.<\/p>\n\n\n\n<p>We will not add the controller as a hypervisor, we will instead add the hypervisors as nodes that will be managed from the controller(s). <\/p>\n\n\n\n<h3 id=\"hypervisors\" class=\"wp-block-heading\"><em>Hypervisors<\/em><\/h3>\n\n\n\n<p>Once again start one of the hosts. Choose add a node instead at the boot prompt.<\/p>\n\n\n\n<p>Let it finished. Do access the console, the added node does not have the &#8220;onefemenu&#8221; app, instead run &#8220;onehostmenu&#8221;.<\/p>\n\n\n\n<p>Configure the network, use the network diagram for guidance.<\/p>\n\n\n\n<p>After it is done, go to the controller run &#8220;onefemenu&#8221;.<\/p>\n\n\n\n<p>Choose add node, follow the prompts. You are adding the node you just created on the OP network( meaning use the OPenNebula management IP! ). Use root as the user, it will ask you for credentials. Let it finished.<\/p>\n\n\n\n<p>Add the second hypervisor.<\/p>\n\n\n\n<p>If everything run correctly you should be able to access horizon on c1 port 2626.<\/p>\n\n\n\n<p>The credentials can be shown by running &#8220;onefemenu&#8221; and displaying them.<\/p>\n\n\n\n<p>Before we continue though we need to do a couple of things on the hosts.<\/p>\n\n\n\n<p>We would like to use the VM network for virtual machines but also a gateway so said machines can access external networks (i.e. the Internet).<\/p>\n\n\n\n<p>Thus, we will add a bridge VM-1 which will have ens5 as a port.<\/p>\n\n\n\n<p>VM-1 will use the following: <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>IP address: 172.16.100.252\/24 (host1, 172.16.100.253\/24 (host2)<\/li>\n\n\n\n<li>Will run keepalived on both host1 and host2<\/li>\n\n\n\n<li>VRRP address: 172.16.100.254\/24<\/li>\n<\/ul>\n\n\n\n<p>This will be important later on, will add a virtual router that will allow VMs to use 172.16.100.254 to access external network (and the Internet using suitable masquerading rules).<\/p>\n\n\n\n<p>We can of course use the &#8220;onehostmenu&#8221; app, however this is not recommended. If you have a custom BGP configuration which we will when we add nodes to the cluster, the BGP configuration gets overwritten.<\/p>\n\n\n\n<p>We should run &#8220;nmtui&#8221;, this because we are running Alma linux and it uses NetorkManager.<\/p>\n\n\n\n<p>However this being linux we do not have to that.<\/p>\n\n\n\n<p>On each of the hypervisors cd to &#8220;\/etc\/NetworkManager\/system-connections\/&#8221;<\/p>\n\n\n\n<p>Using nano, edit the ens5.nmconnection, it should look like this after editing:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[connection]<br>id=ens5<br>uuid=8bd0d7a7-540f-4f6d-a191-54d0031c8c6d<br>type=ethernet<br>controller=VM-1<br>interface-name=ens5<br>port-type=bridge<br><br>[ethernet]<br><br>[bridge-port]<\/pre>\n\n\n\n<p>Again use nano and create VM-1.nmconnection.<\/p>\n\n\n\n<p>It should look like this after editing:<\/p>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[connection]<br>id=VM-1<br>uuid=b468a409-b72a-445c-9890-99c4f8b003b6<br>type=bridge<br>interface-name=VM-1<br>timestamp=1784999502<br><br>[ethernet]<br><br>[bridge]<br><br>[ipv4]<br>address1=172.16.100.252\/24<br>method=manual<br><br>[ipv6]<br>addr-gen-mode=default<br>method=auto<br><br>[proxy]<\/pre>\n\n\n\n<p>Restart the NetworkManager service. Install keepalived. configure it so keepalived.conf looks like this. Restart the service.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vrrp_instance VI_1 {<br>    state MASTER<br>    interface VM-1<br>    virtual_router_id 51<br>    priority 101<br>    advert_int 1<br><br>    unicast_src_ip 172.16.100.252<br>    unicast_peer {<br>        172.16.100.253<br>    }<br><br>    virtual_ipaddress {<br>        172.16.100.254\/24<br>    }<br>}<\/pre>\n\n\n\n<p>After this look at the network it should look like this.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">7: VM-1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000<br>    link\/ether 00:50:01:00:04:02 brd ff:ff:ff:ff:ff:ff<br>    inet 172.16.100.252\/24 brd 172.16.100.255 scope global noprefixroute VM-1<br>       valid_lft forever preferred_lft forever<br>    inet 172.16.100.254\/24 scope global secondary VM-1<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::5d53:9c29:99c1:7e98\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<\/pre>\n\n\n\n<p>Do the same for host2. Make sure you have the correct IP addresses.<\/p>\n\n\n\n<p>The network for host2 should like this:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">7: VM-1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000<br>    link\/ether 00:50:01:00:05:02 brd ff:ff:ff:ff:ff:ff<br>    inet 172.16.100.253\/24 brd 172.16.100.255 scope global noprefixroute VM-1<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::7cf:5ca7:f3dd:5ab0\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<\/pre>\n\n\n\n<p>Finally enabled the rc.local service on both host1 and host2. There are plenty of recipes for this.<\/p>\n\n\n\n<p>On bot hypervisors, save the \/etc\/frr\/bgpd.conf to bgpd.conf.bak.<\/p>\n\n\n\n<p>Add the following to rc.local:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cp \/etc\/frr\/bgpd.conf.bak \/etc\/frr\/bgpd.conf<br>systemctl restart frr<br>mount -a<br>systemctl restart frr<br>touch \/var\/lock\/subsys\/local<\/pre>\n\n\n\n<p>This will ensure that any custom configuration will be restored after a reboot. We will need a custom BGP configuration when we add the two controllers to the cluster.<\/p>\n\n\n\n<p>Do not mind about the &#8220;mount -a&#8221; for the time being. This is to ensure that our NFS shares are correctly mounted. We will get to that later on.<\/p>\n\n\n\n<h3 id=\"sunstone\" class=\"wp-block-heading\"><em>Sunstone<\/em><\/h3>\n\n\n\n<p>Access the horizon ui at port 2626.<\/p>\n\n\n\n<p>If you got to Hosts under Infrastructure, you should see the hosts.<\/p>\n\n\n\n<p>As per our diagram, 172.16.1.4 and 172.16.1.5.<\/p>\n\n\n\n<p>If you were to click on one of them you will see that they are hypervisors running KVM.<\/p>\n\n\n\n<p>This not an issue on bare metal that has virtualization enabled. It is an issue on EVE-NG.<\/p>\n\n\n\n<p>We need to re-add the hosts as QEMU.<\/p>\n\n\n\n<p>This is very simple, just remove each host then re-add them and choose QUEMU. This will re-add both as QEMU hypervisors and allow you to run VMS.<\/p>\n\n\n\n<p>Finally when you configure a VM template, for your CPU model choose qemu64.<\/p>\n\n\n\n<p>You should have now two hypervisors being monitored and manage by the c1 controller.<\/p>\n\n\n\n<h3 id=\"bgp\" class=\"wp-block-heading\"><em>BGP<\/em><\/h3>\n\n\n\n<p>Before we proceed to adding two more cluster members. We need to take a look at the BGP configuration installed.<\/p>\n\n\n\n<p>The ISO will install FRR and a default BGP configuration. The config files will be under \/etc\/frr.<\/p>\n\n\n\n<p>A typical bgpd.conf looks like this for the controller:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">router bgp 64512<br>  bgp router-id 10.229.128.251<br>  bgp cluster-id 10.229.128.251<br>  no bgp default ipv4-unicast<br>  neighbor fabric peer-group<br>  neighbor fabric remote-as 64512<br>  neighbor fabric capability extended-nexthop<br>  neighbor fabric update-source 10.229.128.251<br>  bgp listen range 172.16.100.2\/32 peer-group fabric<br>  address-family l2vpn evpn<br>    neighbor fabric activate<br>    neighbor fabric route-reflector-client<br>    advertise-all-vni<br>  exit-address-family<br>end<\/pre>\n\n\n\n<p> For the hypervisors:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">router bgp 64512<br>  bgp router-id 10.229.128.252<br>  no bgp default ipv4-unicast<br>  neighbor fabric peer-group<br>  neighbor fabric remote-as 64512<br>  neighbor fabric capability extended-nexthop<br>  neighbor 10.229.128.251 peer-group fabric<br>  !<br>  address-family l2vpn evpn<br>    neighbor fabric activate<br>    advertise-all-vni<br>  exit-address-family<\/pre>\n\n\n\n<p>A couple of things to notice.<\/p>\n\n\n\n<p>On the controller, we listen on \/32 addresses (this belong to the hypervisors).<\/p>\n\n\n\n<p>This will change when we go to a cluster, additionally we listen on the \/24 OP network instead in addition to the EVE_NG management network. We will need to add the additional cluster members as route reflectors so BGP functions correctly, since if the VIP of the cluster moves that node should be able to exchange BGP information.<\/p>\n\n\n\n<p>We make a copy of the final configuration and do make sure that configuration is the one loaded by FRR after a reboot.<\/p>\n\n\n\n<p>If we were to add a third hypervisor, ansible will clover it. This ensures that at least the BGP fabric is working until we add the new hypervisor to it.<\/p>\n\n\n\n<p>Now the next step is to add two more cluster nodes, configure a VIP and reconfigure BGP.<\/p>\n\n\n\n<h2 id=\"cluster-nodes-nfs-and-bgp-setup\" class=\"wp-block-heading\">Cluster Nodes NFS and BGP setup.<\/h2>\n\n\n\n<p>We will add two cluster nodes to our setup. In addition will configure BGP since it is a custom setup.<\/p>\n\n\n\n<p>Rather than waste paper, I will re-direct you to great resource again from OpneNebula.<\/p>\n\n\n\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=cLvmSUUZh1Y\" target=\"_blank\" rel=\"noreferrer noopener\">OpenNebula Front End High Availability<\/a>, this is a YouTube video showing exactly how to create a cluster for OpenNebula.<\/p>\n\n\n\n<p>In our case, we will use 10.229.128.250 as the VIP of the cluster.<\/p>\n\n\n\n<p>You should be able to connect to the VIP and run Sunstone.<\/p>\n\n\n\n<p>Via cli you should see the following:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# onezone show 0<br>ZONE 0 INFORMATION<br>ID                : 0<br>NAME              : OpenNebula<br>STATE             : ENABLED<br><br><br>ZONE SERVERS<br>ID NAME            ENDPOINT<br> 0 10.229.128.231  http:\/\/10.229.128.231:2633\/RPC2<br> 1 10.229.128.232  http:\/\/10.229.128.232:2633\/RPC2<br> 2 10.229.128.233  http:\/\/10.229.128.233:2633\/RPC2<br><br>HA &amp; FEDERATION SYNC STATUS<br>ID NAME            STATE      TERM       INDEX      COMMIT     VOTE  FED_INDEX<br> 0 10.229.128.231  leader     244        16355      16355      0     -1<br> 1 10.229.128.232  follower   244        16355      16355      0     -1<br> 2 10.229.128.233  follower   244        16355      16355      -1    -1<br><br>ZONE TEMPLATE<br>ENDPOINT=\"http:\/\/localhost:2633\/RPC2\"<br>ENDPOINT_GRPC=\"localhost:2634\"<\/pre>\n\n\n\n<p>You can see that the cluster zone 0, hast three members.<\/p>\n\n\n\n<p>Member with ID 0 is the (10.229.128.231) is the leader. It you look at the network issuing &#8220;ip add&#8221;, you will see:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">ens3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000<br>    link\/ether 00:50:01:00:01:00 brd ff:ff:ff:ff:ff:ff<br>    altname enp0s3<br>    inet 10.229.128.231\/24 brd 10.229.128.255 scope global noprefixroute ens3<br>       valid_lft forever preferred_lft forever<br>    inet 10.229.128.230\/24 scope global secondary ens3<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::250:1ff:fe00:100\/64 scope link noprefixroute<br>       valid_lft forever preferred_lft forever<\/pre>\n\n\n\n<p>Notice that in addition of having .231, it has IP .230 which is the VIP.<\/p>\n\n\n\n<p>I recommend that you play with this by shutting a node, the cluster will move the VIP and you should be able to reach the cluster and manage it.<\/p>\n\n\n\n<p>Interestingly enough, Sunstone does not give you the detail information you get by using the command line.<\/p>\n\n\n\n<p>Now that you have a cluster you need to add NFS. Once again this step is well known so I will not go into much detail.<\/p>\n\n\n\n<p>You need to install the NFS server packages on all member of the cluster and configure your fstab file so the shares are mount automatically.<\/p>\n\n\n\n<p>Here I run into issues, using Ubuntu NFS shares get mount during boot.<\/p>\n\n\n\n<p>With Alma linux this was not the case, spent a whole afternoon troubleshooting could not make it work, thus I added the &#8220;mount -a&#8221; on &#8220;rc.local&#8221; to make sure the shares are available.<\/p>\n\n\n\n<p>If you got everything right you should see:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# onedatastore list<br>  ID NAME                                                             SIZE AVA CLUSTERS IMAGES TYPE DS      TM      STAT<br> 109 NFS-SYSTEM                                                      60.8G 61% 0             0 sys  -       shared  on<br> 108 NFS-IMAGE                                                       60.8G 61% 0             2 img  fs      shared  on<br> 107 NFS-FILE                                                        60.8G 61% 0             0 fil  fs      shared  on<br>   2 files                                                           14.6G 98% 0             0 fil  fs      local   on<br>   1 default                                                         14.6G 98% 0             2 img  fs      local   on<br>   0 system                                                              - -   0             0 sys  -       local   on<\/pre>\n\n\n\n<p> Now to something that is not well documented in my opinion.<\/p>\n\n\n\n<p>As I already mentioned, the ISO runs ansible playbooks to automate the configuration.<\/p>\n\n\n\n<p>This carries a problem, if you add two more controllers, or use the &#8220;onefemenu&#8221; to add an interface for example, ansible will clover you BGP configuration.<\/p>\n\n\n\n<p>That is the reason we configure BGP directly after we have three nodes, we make sure the setup works then we copy it to a backup file and finally restore it as mentioned previusly on the rc.local files.<\/p>\n\n\n\n<p>On each controller (which will act a route reflectors) edit the \/etc\/frr\/bgpd.conf to contain:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# cat \/etc\/frr\/bgpd.conf<br><br>!  Route Reflector configuration<br>!<br>hostname 10.229.128.231<br>!<br>log file \/var\/log\/frr\/bgpd.log informational<br>!<br>router bgp 64512<br>  bgp router-id 10.229.128.231<br>  bgp cluster-id 10.229.128.231<br>  no bgp default ipv4-unicast<br>  neighbor fabric peer-group<br>  neighbor fabric remote-as 64512<br>  neighbor fabric capability extended-nexthop<br>  neighbor fabric update-source 10.229.128.231<br> # bgp listen range 172.16.1.5\/32 peer-group fabric<br> # bgp listen range 172.16.1.4\/32 peer-group fabric<br>  bgp listen range 172.16.1.0\/24 peer-group fabric<br>  bgp listen range 10.229.128.0\/24 peer-group fabric<br>  address-family l2vpn evpn<br>    neighbor fabric activate<br>    neighbor fabric route-reflector-client<br>    advertise-all-vni<br>    advertise-svi-ip<br>  exit-address-family<br>end<\/pre>\n\n\n\n<p>For each controller, change the router-id, cluster-id, comment out the \/32 commands and add the range stated.<\/p>\n\n\n\n<p>You can look on the FRR web site about the syntax, briefly you are allowing the BGP process to listen on requests on certain networks and if so allow make BGP peering.<\/p>\n\n\n\n<p>In addition it is very import that you advertise all vnis and all svis (allows you to see the VXLAN and MAC addresses that the hypervisors are advertising).<\/p>\n\n\n\n<p>On each of the hypervisors, edit the same file and have it to it looks like this:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">hostname 10.229.128.235<br>!<br>router bgp 64512<br> bgp router-id 10.229.128.235<br> no bgp default ipv4-unicast<br> neighbor fabric peer-group<br> neighbor fabric remote-as 64512<br> neighbor fabric capability extended-nexthop<br> neighbor 10.229.128.231 peer-group fabric<br> neighbor 10.229.128.232 peer-group fabric<br> neighbor 10.229.128.233 peer-group fabric<br> !<br> address-family l2vpn evpn<br>  neighbor fabric activate<br>  advertise-all-vni<br>  advertise-svi-ip<br> exit-address-family<br>exit<\/pre>\n\n\n\n<p>Once gain change the IP addresses to much your setup.<\/p>\n\n\n\n<p>Notice we are peering with the three controllers (not the VIP address).<\/p>\n\n\n\n<p>After all of this using vtysh (the frr cli) you should see:<\/p>\n\n\n\n<p><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onenode01 ~]#  vtysh<br><br>Hello, this is FRRouting (version 10.5.3).<br>Copyright 1996-2005 Kunihiro Ishiguro, et al.<br><br>onenode01# sh bgp summary<br><br>L2VPN EVPN Summary:<br>BGP router identifier 10.229.128.235, local AS number 64512 VRF default vrf-id 0<br>BGP table version 0<br>RIB entries 0, using 0 bytes of memory<br>Peers 3, using 71 KiB of memory<br>Peer groups 1, using 64 bytes of memory<br><br>Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up\/Down State\/PfxRcd   PfxSnt Desc<br>10.229.128.231  4      64512       373       373        0    0    0 06:10:35            0        0 FRRouting\/10.5.3<br>10.229.128.232  4      64512       374       374        0    0    0 05:07:00            0        0 FRRouting\/10.5.3<br>10.229.128.233  4      64512       374       374        0    0    0 05:06:36            0        0 FRRouting\/10.5.3<\/pre>\n\n\n\n<p>And on the controller:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# vtysh<br><br>Hello, this is FRRouting (version 10.5.3).<br>Copyright 1996-2005 Kunihiro Ishiguro, et al.<br><br>onepoc# sh bgp summary<br><br>L2VPN EVPN Summary:<br>BGP router identifier 10.229.128.231, local AS number 64512 VRF default vrf-id 0<br>BGP table version 0<br>RIB entries 0, using 0 bytes of memory<br>Peers 2, using 47 KiB of memory<br>Peer groups 1, using 64 bytes of memory<br><br>Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up\/Down State\/PfxRcd   PfxSnt Desc<br>*10.229.128.234 4      64512       375       375        0    0    0 06:12:12            0        0 FRRouting\/10.5.3<br>*10.229.128.235 4      64512       375       375        0    0    0 06:12:11            0        0 FRRouting\/10.5.3<br><br>Total number of neighbors 2<br>* - dynamic neighbor<br>2 dynamic neighbor(s), limit 100<\/pre>\n\n\n\n<p>Of course since you are not running any VMs at this point you will not see any L2VPN EVPN routes advertised via VXLAN.<\/p>\n\n\n\n<h2 id=\"access-to-external-networks\" class=\"wp-block-heading\">Access to External Networks<\/h2>\n\n\n\n<p>You now have  a new shiny three node two hypervisor OPenNenula cluster.<\/p>\n\n\n\n<p>Go crazy, download some images, create VMs, create VXLAN networks, etc.<\/p>\n\n\n\n<p>So you have a VM, it can talk to other VMs in their own network however, you want it to be able to reach the Internet.<\/p>\n\n\n\n<p>The default gateway for all VMs trying to access the Internet will be: 172.16.100.254, the VRRP ip!.<\/p>\n\n\n\n<p>Add the following to rc.local on both hypervisors:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">iptables -t nat -A POSTROUTING -s 172.16.100.0\/24 ! -d 172.16.100.0\/24 -o ens3 -j MASQUERADE<\/pre>\n\n\n\n<p>This will ensure that when you create a VM on either, when it tries to get to the Internet will hit the VRRP IP and do masquerading.<\/p>\n\n\n\n<p>When you create a new VM, you need to remember to give it a gateway that needs to be known. This is easily done by creating a virtual router which its default gateway is the .254 and all other virtual networks connect to this virtual routers (the default GW for each virtual network) thus your VMs can reach the Internet.<\/p>\n\n\n\n<p>Since this is also a router (virtual nonetheless), you can have rules to prevent inter VM communications. Remember the virtual router can be any linux appliance, the one OpenNebula uses is a minimal Alpine VM. I did create my own (Alpine also) and it works very well.<\/p>\n\n\n\n<h3 id=\"vms-vxlan-and-testing\" class=\"wp-block-heading\"><em>VMs, VXLAN  and Testing<\/em><\/h3>\n\n\n\n<p>I will not go into how to create VMs, or a VXLAN network however, I want to show what EVPN BGP will show you.<\/p>\n\n\n\n<p>Of course you need to create a few VMS and at least three networks.<\/p>\n\n\n\n<p>First the virtual networks created:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# onevnet list<br> ID USER     GROUP   NAME    CLUSTERS  BRIDGE  STATE LEASES OUTD ERRO<br> 3 oneadmin oneadmin VM-1       0      VM-1     rdy    2     0    0<br> 1 oneadmin oneadmin vxlan-600  0      vxlan600 rdy    2     0    0<br> 0 oneadmin oneadmin vxlan-500  0      vx500    rdy    2     0    0<\/pre>\n\n\n\n<p>I created three networks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A standard network for normal VMs<\/li>\n\n\n\n<li>Two VXLAN networks: vxlan600 and vxlan500<\/li>\n<\/ul>\n\n\n\n<p>Then four virtual machines were created:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onepoc ~]# onevm list<br>ID USER    GROUP    NAME       STAT CPU   MEM  HOST         TIME<br>6 oneadmin oneadmin VM-TEST1   poff 0.25  256M 172.16.1.5   17d 23h06<br>5 oneadmin oneadmin Router-1   runn 0.25  256M 172.16.1.4   17d 23h31<br>3 oneadmin oneadmin vxlan-600-1 poff 0.25 256M 172.16.1.4   18d 17h11<br>2 oneadmin oneadmin vxlan-500-1 poff 0.25 256M 172.16.1.5   18d 17h32<\/pre>\n\n\n\n<p>The VMs created are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>VM-TEST1 (to access Router-1)<\/li>\n\n\n\n<li>Router-1 (the virtual router that allows access to the Internet)<\/li>\n\n\n\n<li>vxlan-600-1 (VMS on VXLAN NET600)<\/li>\n\n\n\n<li>vxlan-500-1 (VMS on VXLAN NET500)<\/li>\n<\/ul>\n\n\n\n<p>If you SSH onto Router-1:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/home\/miguel # cat \/etc\/os-release<br>NAME=\"Alpine Linux\"<br>ID=alpine<br>VERSION_ID=3.21.7<br>PRETTY_NAME=\"Alpine Linux v3.21\"<br>HOME_URL=\"https:\/\/alpinelinux.org\/\"<br>BUG_REPORT_URL=\"https:\/\/gitlab.alpinelinux.org\/alpine\/aports\/-\/issues\"<br>1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN qlen 1000<br>    link\/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>    inet 127.0.0.1\/8 scope host lo<br>       valid_lft forever preferred_lft forever<br>    inet6 ::1\/128 scope host<br>       valid_lft forever preferred_lft forever<br>2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc pfifo_fast state UP qlen 1000<br>    link\/ether 02:00:ac:10:32:33 brd ff:ff:ff:ff:ff:ff<br>    inet 172.16.50.254\/24 scope global eth0<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::acff:fe10:3233\/64 scope link<br>       valid_lft forever preferred_lft forever<br>3: eth1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc pfifo_fast state UP qlen 1000<br>    link\/ether 02:00:ac:10:33:33 brd ff:ff:ff:ff:ff:ff<br>    inet 172.16.51.254\/24 scope global eth1<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::acff:fe10:3333\/64 scope link<br>       valid_lft forever preferred_lft forever<br>4: eth2: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast state UP qlen 1000<br>    link\/ether 02:00:ac:10:64:02 brd ff:ff:ff:ff:ff:ff<br>    inet 172.16.100.2\/24 scope global eth2<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::acff:fe10:6402\/64 scope link<br>       valid_lft forever preferred_lft forever<\/pre>\n\n\n\n<p>A few things to notice:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The router is running alpine linux 3.21 with forwarding enable and IPTABLES installed<\/li>\n\n\n\n<li>eth2 allows access to the Internet. It has IP address 172.16.100.2 and the default GW is 172.16.100.254<\/li>\n\n\n\n<li>eth0 and eth1 are the ports that connect to the VXLAN VMs.<\/li>\n\n\n\n<li>Both have as ip addresses ending in .254, the default gateway for each VM. This is done when you create the VM or via a VM template among other things.<\/li>\n\n\n\n<li>This is as it was pointed out a VM. OpenNebula has now an appliance you can download. It is an Alpine Linux machine. So I decided to make my own.<\/li>\n<\/ul>\n\n\n\n<p>If you start the other VLXLAN VMs, you should see BGP starting to advertise their mac addresses. This is importan because  a VM on VXLAN-500  hypervisor 1 would not be able to talk to another VM on VXLAN-500 on hypervisor 2 unless VTEP and EVPN is working.<\/p>\n\n\n\n<p>As an added bonus since we are using Router-1 VMs on different VXLANS will be able to communicate unless you  add specific IPTABLES rules to deny access.<\/p>\n\n\n\n<p>After starting th other VMS we can test connectivity.<\/p>\n\n\n\n<p>On the Router:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">PING 172.16.100.1 (172.16.100.1): 56 data bytes<br>64 bytes from 172.16.100.1: seq=8 ttl=64 time=2103.056 ms<br>64 bytes from 172.16.100.1: seq=9 ttl=64 time=1101.773 ms<br><br>\/home\/miguel # ping 8.8.8.8<br>PING 8.8.8.8 (8.8.8.8): 56 data bytes<br>64 bytes from 8.8.8.8: seq=0 ttl=114 time=40.752 ms<br>64 bytes from 8.8.8.8: seq=1 ttl=114 time=40.707 ms<\/pre>\n\n\n\n<p>The router can ping VM-1 at IP 172.16.100.1, the router can also ping the Internet.<\/p>\n\n\n\n<p>Now let&#8217;s connect to one of the VMS. We will use VXLAN-500-1 becase it is on hypervisor 2, while the router runs on hypervisor 1. This will ensure the BGP EVPN is actually working.<\/p>\n\n\n\n<p>From the router we SSH onto 172.16.50.50, I created a local user.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vxlan-500-1:~$ ip add<br>1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN qlen 1000<br>    link\/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>    inet 127.0.0.1\/8 scope host lo<br>       valid_lft forever preferred_lft forever<br>    inet6 ::1\/128 scope host<br>       valid_lft forever preferred_lft forever<br>2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1450 qdisc pfifo_fast state UP qlen 1000<br>    link\/ether 02:00:ac:10:32:32 brd ff:ff:ff:ff:ff:ff<br>    inet 172.16.50.50\/24 scope global eth0<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::acff:fe10:3232\/64 scope link<br>       valid_lft forever preferred_lft forever<br><br>vxlan-500-1:~$ ip route<br>default via 172.16.50.254 dev eth0  metric 1 onlink<br>169.254.16.9 dev eth0 scope link<br>172.16.50.0\/24 dev eth0 scope link  src 172.16.50.50<\/pre>\n\n\n\n<p>We should be able to ping the Internet, and we know BGP is also working.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">vxlan-500-1:~$ ping www.google.com<br>PING www.google.com (142.251.154.119): 56 data bytes<br>64 bytes from 142.251.154.119: seq=0 ttl=42 time=52.869 ms<br>64 bytes from 142.251.154.119: seq=1 ttl=42 time=54.530 ms<\/pre>\n\n\n\n<p>So let&#8217;s take a look at BGP EVPN. On the controller:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">onepoc# sh bgp l2vpn evpn<br>BGP table version is 12, local router ID is 10.229.128.231<br>Status codes: s suppressed, d damped, h history, * valid, &gt; best, i - internal<br>Origin codes: i - IGP, e - EGP, ? - incomplete<br>EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]<br>EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]<br>EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]<br>EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]<br>EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]<br><br>   Network          Next Hop            Metric LocPrf Weight Path<br>Route Distinguisher: 10.229.128.234:2<br> *&gt;i [2]:[0]:[48]:[02:00:ac:10:32:33]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [2]:[0]:[48]:[46:ce:d0:b2:0b:fd]:[128]:[fe80::4e9:97ff:feef:66a4]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [2]:[0]:[48]:[7a:b1:b9:46:44:93]:[128]:[fe80::4e9:97ff:feef:66a4]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [3]:[0]:[32]:[10.229.128.234]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:500 ET:8<br>Route Distinguisher: 10.229.128.234:3<br> *&gt;i [2]:[0]:[48]:[02:00:ac:10:33:32]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:600 ET:8<br> *&gt;i [2]:[0]:[48]:[8e:41:5b:98:77:10]:[128]:[fe80::7802:84ff:fe45:d10]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:600 ET:8<br> *&gt;i [3]:[0]:[32]:[10.229.128.234]<br>                    10.229.128.234                100      0 i<br>                    RT:64512:600 ET:8<br>Route Distinguisher: 10.229.128.235:2<br> *&gt;i [2]:[0]:[48]:[02:00:ac:10:32:32]<br>                    10.229.128.235                100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [2]:[0]:[48]:[42:15:35:2c:98:e3]<br>                    10.229.128.235                100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [2]:[0]:[48]:[42:af:f6:d1:11:98]:[128]:[fe80::e8e4:6aff:fee7:e526]<br>                    10.229.128.235                100      0 i<br>                    RT:64512:500 ET:8<br> *&gt;i [3]:[0]:[32]:[10.229.128.235]<br>                    10.229.128.235                100      0 i<br>                    RT:64512:500 ET:8<\/pre>\n\n\n\n<p>It works!. This is a lot to understand so I will not go into details.<\/p>\n\n\n\n<p>Suffice to say that both hypervisors (10.229.128.234, and .235) are advertising VNIs and MAC addresses. The controller acts as a route reflector so each hypervisor knows where to find a machine MAC address and connect to it, VTEPs are working, VXLAN has created tunnels and L2VPN EVPN is advertising routes. Very cool.<\/p>\n\n\n\n<p>We could ask ourselves what interface are the VTEPs using, you see .234 and .235 as the routes in BGP.  These are not technically the actual VTEPs, when you create the VXLAN you tell OpenNebula what interface to use. We told it to use ens4. You can check this by issuing the following on each hypervisors:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[root@onenode00 ~]# ip link show type vxlan<br>11: ens4.500: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vx500 state UNKNOWN mode DEFAULT group default qlen 1000<br>    link\/ether 46:ce:d0:b2:0b:fd brd ff:ff:ff:ff:ff:ff<br>15: ens4.600: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vxlan600 state UNKNOWN mode DEFAULT group default qlen 1000<br>    link\/ether d2:a9:46:da:0c:64 brd ff:ff:ff:ff:ff:ff<br><br>[root@onenode01 ~]# ip link show type vxlan<br>14: ens4.500: &lt;BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vx500 state UNKNOWN mode DEFAULT group default qlen 1000<br>    link\/ether 5a:02:f4:da:3b:68 brd ff:ff:ff:ff:ff:ff<\/pre>\n\n\n\n<p>As you can see, on both hypervisors the VTEPs are using ens4.vni, 500 and 600 respectively.  I will let the reader ponder why we see the .234, .235 addresses rather than the ens4 IP addresses that actually create the tunnels.<\/p>\n\n\n\n<h2 id=\"conclusions\" class=\"wp-block-heading\">Conclusions<\/h2>\n\n\n\n<p>This is it. You now have a working production OpenNebula cluster.<\/p>\n\n\n\n<p>Some caveats:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>I tried using OVS for VXLANs. Run into issues, BGP would not advertise VNIs. I am trouble shooting this. The reason is that OVS is an abstract layer which is supposed to be OS agnostic. However, that is one of the problems at one point it needs to interact with the OS interfaces and kernel (the kernel space not the user space) to correctly send BGP packets from within OVS. This works of course but I myself have not figure it out yet. When I do I will post a solution.<\/li>\n\n\n\n<li>Using the kernel way for doing VXLANs is not an issue per say,  OVS in my opinions is more elegant and scales better.<\/li>\n\n\n\n<li>Using the ISO is fantastic but since it uses ansible, if you want to change network parameters do not use &#8220;onefemenu&#8221; on the controller or &#8220;onehostmenu&#8221; on the hosts. Use &#8220;nmtui&#8221; or as we did configure the network directly.<\/li>\n\n\n\n<li>If you need to add another hypervisor, use the trick we stated, add the new host then immediately run the backup BGP configuration to avoid interruptions on all nodes. Then modify the BGP configuration of the new hypervisor.<\/li>\n\n\n\n<li>Perhaps this could be automated but I did not wanted to go into the rabbit hole and poke around the ansible scripts.<\/li>\n\n\n\n<li>Finally I did not create VRFs. This will be necessary for multi-tenancy on a production system. Your VNIs are already segmenting the network however you need to let the hosts known  that they need to segment the kernel space. You will have to tweak your BGP configuration, create VRFs using FRR, map VLANs, and so on so far. Not difficult to do but you will need to test on a LAB setup until you feel confortable with it. <\/li>\n\n\n\n<li>Refer to the FRR documentation and also a good resource for this is PROXMOX since they do VRFs off the bat when you create VXLANs using their SDN implementation.<\/li>\n<\/ul>\n\n\n\n<p>Well there you have it.<\/p>\n\n\n\n<p>Ciao.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog I will show how to install an OpenNebula cluster from scratch, well almost, using an ISO that allows you to create a minimal configuration. The initial setup can be extended to have an almost functional production setup.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-934","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/934","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=934"}],"version-history":[{"count":91,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/934\/revisions"}],"predecessor-version":[{"id":1041,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/934\/revisions\/1041"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=934"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=934"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=934"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}