{"id":720,"date":"2026-01-02T15:19:32","date_gmt":"2026-01-02T15:19:32","guid":{"rendered":"http:\/\/blog.miguelsarmiento.com\/?p=720"},"modified":"2026-01-02T15:22:17","modified_gmt":"2026-01-02T15:22:17","slug":"pve-vlans-trunks-oh-my","status":"publish","type":"post","link":"https:\/\/blog.miguelsarmiento.com\/?p=720","title":{"rendered":"PVE, VLANs, Trunks, Oh My!"},"content":{"rendered":"

Hello there,<\/p>\n

I thought I was done with PVE, then I noticed that some users had issues with VLANs, trunks, etc.<\/p>\n

In principle it should fairly straight forward to configure VLANs (not VXLANs) however, using the native Linux way could be a bit complicated sometimes.<\/p>\n

OvS on the other hand was created with virtualization in mind and SDN in particular. A key point is that it supports Open Flow and allows for separation of the data plane and control plane.<\/p>\n

Understanding and implementing it is easier said that done, so keep on reading.<\/p>\n

<\/p>\n

Rationale<\/h2>\n

PVE will allow you to create VMs and containers, it will also allow you to access them. It will also allow communications between them using its security tools.<\/p>\n

However, there will be times when you may need access to external devices and would like to separate access to those devices also.<\/p>\n

You can, of course, use SDN in smaller networks, though VLANs are as good as well. PVE, being Linux-based has two ways of achieving this: traditional bridging or using Open VSwitch.<\/p>\n

Both methods are not difficult to do; of course, if you are trying a complicated setup, you go down a rabbit hole very quickly.<\/p>\n

We will create a couple of VLANs using OvS, then we will use a ToR (Top of the rack) switch, and we will trunk the VLANs we created to it. Finally, we achieved separation and security by connecting the ToR to a Cisco ASAv, which will allow the VMs access to the Internet. It will also allow access between VMs, if you add the corresponding rules on the firewall.<\/p>\n

Setup<\/h2>\n

The following figure shows the network setup we are using.<\/p>\n

\"\"<\/a>
Figure1. Network Setup.<\/figcaption><\/figure>\n

The setup consists of:<\/p>\n

    \n
  1. An existing Linux bridge, created by default by PVE. We use it for management.<\/li>\n
  2. Three OVS bridges. One for communication to the NFS backend, one for creating the cluster, and finally one for VLANs and for bridging the VLANs.<\/li>\n
  3. A ToR switch that trunks to each PVE node.<\/li>\n
  4. A Cisco ASAv that connects to the ToR switch.<\/li>\n<\/ol>\n

    Configurations<\/h2>\n

    PVE<\/h3>\n

    This is a standard PVE cluster using a dedicated network for management.<\/p>\n

    Before creating the cluster, add an OvS bridge. Choose the interface you will use for cluster communications. Create the bridge for NFS storage, and create the OvS bridge for connecting to the ToR device.<\/p>\n

    You should see the following in the file “\/etc\/inetwork\/interfaces” on each node after the bridges are created.<\/p>\n

    # network interface settings; autogenerated\r\n# Please do NOT modify this file directly, unless you know what\r\n# you're doing.\r\n#\r\n# If you want to manage parts of the network configuration manually,\r\n# please utilize the 'source' or 'source-directory' directives to do\r\n# so.\r\n# PVE will preserve these directives, but will NOT read its network\r\n# configuration from sourced files, so do not attempt to move any of\r\n# the PVE managed interfaces into external files!\r\n\r\nauto lo\r\niface lo inet loopback\r\n\r\niface ens3 inet manual\r\n\r\nauto ens4\r\niface ens4 inet manual\r\n ovs_type OVSPort\r\n ovs_bridge vmbr1\r\n\r\nauto ens5\r\niface ens5 inet manual\r\n ovs_type OVSPort\r\n ovs_bridge vmbr2\r\n\r\nauto ens6\r\niface ens6 inet manual\r\n ovs_type OVSPort\r\n ovs_bridge vmbr3\r\n\r\nauto vmbr0\r\niface vmbr0 inet static\r\n address 10.229.128.233\/24\r\n gateway 10.229.128.10\r\n bridge-ports ens3\r\n bridge-stp off\r\n bridge-fd 0\r\n\r\nauto vmbr1\r\niface vmbr1 inet static\r\n address 192.168.1.133\/24\r\n ovs_type OVSBridge\r\n ovs_ports ens4\r\n\r\nauto vmbr2\r\niface vmbr2 inet static\r\n address 172.16.1.3\/24\r\n ovs_type OVSBridge\r\n ovs_ports ens5\r\n\r\nauto vmbr3\r\niface vmbr3 inet manual\r\n ovs_type OVSBridge\r\n ovs_ports ens6\r\n\r\nsource \/etc\/network\/interfaces.d\/*<\/pre>\n
    Several things to notice:<\/p>\n
      \n
    1. The ens3 (this will be different for your devices) is assigned to the Linux bridge.<\/li>\n
    2. The other interfaces are declared as being part of OvS and assigned to the particular bridge.<\/li>\n
    3. Bridge vmbr0 is used for management.<\/li>\n
    4. Bridges vmbr1, vmbr2 are for NFS and cluster communications and thus get an IP address.<\/li>\n
    5. Bridge vmbr3 is used for the trunk that will carry VLAN information between VMS and the ToR switch.<\/li>\n<\/ol>\n
      Ease as pie, isn’t it?<\/p>\n
      Now you can create the cluster and attach the NFS backend storage. You already know this drill (I hope you do).<\/p>\n
      VLANs<\/h3>\n
      Now that you have the infrastructure working, let’s create a couple of VMs (or containers, whatever you wish).<\/p>\n
      I will not\u00a0 go through the process; it is straightforward, however, make sure:<\/p>\n
        \n
      • Choose the correct bridge; in our case, it is vmbr3<\/li>\n
      • Assign the VLAN tag (100, 200, etc).<\/li>\n
      • Choose a default gateway for this VM; it will be created on the ToR switch later.<\/li>\n
      • Uncheck the “firewall” box. We are not using PVE to control access.<\/li>\n
      • Create other VMS or VLANs.<\/li>\n<\/ul>\n
        Now start the VMs and open a console. You will not be able to get out of it, you should be able to ping other VMs in the same network.<\/p>\n
        Let’s take a look at what PVE did in the background in terms of networking. On the PVE node you created one the VMs, issue the command shown below.<\/p>\n
        root@pve-3:~# ip add\r\n1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000\r\n link\/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r\n inet 127.0.0.1\/8 scope host lo\r\n valid_lft forever preferred_lft forever\r\n inet6 ::1\/128 scope host noprefixroute\r\n valid_lft forever preferred_lft forever\r\n2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr0 state UP group default qlen 1000\r\n link\/ether 00:50:01:00:03:00 brd ff:ff:ff:ff:ff:ff\r\n altname enp0s3\r\n altname enx005001000300\r\n3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000\r\n link\/ether 00:50:01:00:03:01 brd ff:ff:ff:ff:ff:ff\r\n altname enp0s4\r\n altname enx005001000301\r\n inet6 fe80::250:1ff:fe00:301\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000\r\n link\/ether 00:50:01:00:03:02 brd ff:ff:ff:ff:ff:ff\r\n altname enp0s5\r\n altname enx005001000302\r\n inet6 fe80::250:1ff:fe00:302\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n5: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000\r\n link\/ether 00:50:01:00:03:03 brd ff:ff:ff:ff:ff:ff\r\n altname enp0s6\r\n altname enx005001000303\r\n inet6 fe80::250:1ff:fe00:303\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000\r\n link\/ether f6:b3:2a:11:db:ea brd ff:ff:ff:ff:ff:ff\r\n7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000\r\n link\/ether 00:50:01:00:03:01 brd ff:ff:ff:ff:ff:ff\r\n inet 192.168.1.133\/24 scope global vmbr1\r\n valid_lft forever preferred_lft forever\r\n inet6 fe80::f476:33ff:fe60:ae46\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n8: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000\r\n link\/ether 00:50:01:00:03:02 brd ff:ff:ff:ff:ff:ff\r\n inet 172.16.1.3\/24 scope global vmbr2\r\n valid_lft forever preferred_lft forever\r\n inet6 fe80::e0d7:c8ff:fe53:8540\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n9: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000\r\n link\/ether 00:50:01:00:03:03 brd ff:ff:ff:ff:ff:ff\r\n inet6 fe80::84fb:22ff:fe5d:1a45\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000\r\n link\/ether 00:50:01:00:03:00 brd ff:ff:ff:ff:ff:ff\r\n inet 10.229.128.233\/24 scope global vmbr0\r\n valid_lft forever preferred_lft forever\r\n inet6 fe80::250:1ff:fe00:300\/64 scope link proto kernel_ll\r\n valid_lft forever preferred_lft forever\r\n21: veth100i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000\r\n link\/ether fe:0e:b3:c7:03:f4 brd ff:ff:ff:ff:ff:ff link-netnsid 0<\/pre>\n
        The interesting bit is at the bottom.<\/p>\n
        An interface “vethXXXi0@ifx”, is created, in our case “veth100i0@if2”.<\/p>\n
        100 is the VLAN we are using, and veth is a virtual interface that allows, in this case, containers to communicate with each other. They are normally created in pairs. I will not go too much deeper on it, you can Google it.<\/p>\n
        The fun part starts now. We would like to connect to another device on that VLAN but outside of the cluster.<\/p>\n
        Of course, you do not need a Cisco switch or Cisco ASA; any managed switch that understands trunks will do. And any firewall (like OPNsense or PFsense) will also do.<\/p>\n
        Managed switch and firewall.<\/h3>\n
        ToR<\/h4>\n
        Connect each PVE to the ToR switch as per the network diagram.<\/p>\n
        This is the configuration of the switch:<\/p>\n
        hostname tor\r\n!\r\nboot-start-marker\r\nboot-end-marker\r\n!\r\n!\r\nno aaa new-model\r\nmemory-size iomem 5\r\nip cef\r\n!\r\nno ip domain lookup\r\n!\r\nmultilink bundle-name authenticated\r\n!\r\n!\r\narchive\r\n log config\r\n hidekeys\r\n! \r\n!\r\ninterface FastEthernet0\/0\r\n no ip address\r\n ip virtual-reassembly\r\n duplex auto\r\n speed auto\r\n!\r\ninterface FastEthernet0\/1\r\n no ip address\r\n shutdown\r\n duplex auto\r\n speed auto\r\n!\r\ninterface FastEthernet1\/0\r\n!\r\ninterface FastEthernet1\/1\r\n!\r\ninterface FastEthernet1\/2\r\n!\r\ninterface FastEthernet1\/3\r\n!\r\ninterface FastEthernet1\/4\r\n!\r\ninterface FastEthernet1\/5\r\n!\r\ninterface FastEthernet1\/6\r\n!\r\ninterface FastEthernet1\/7\r\n!\r\ninterface FastEthernet1\/8\r\n!\r\ninterface FastEthernet1\/9\r\n!\r\ninterface FastEthernet1\/10\r\n!\r\ninterface FastEthernet1\/11\r\n!\r\ninterface FastEthernet1\/12\r\n description trunk to ASA\r\n switchport mode trunk\r\n!\r\ninterface FastEthernet1\/13\r\n description trunk to pve1\r\n switchport mode trunk\r\n!\r\ninterface FastEthernet1\/14\r\n description trunk to pve2\r\n switchport mode trunk\r\n!\r\ninterface FastEthernet1\/15\r\n description trunk to pve3\r\n switchport mode trunk\r\n!\r\ninterface Vlan1\r\n no ip address\r\n!\r\ninterface Vlan100\r\n ip address 10.10.100.254 255.255.255.0\r\n ip virtual-reassembly\r\n ip ospf 10 area 0\r\n!\r\ninterface Vlan200\r\n ip address 10.10.200.254 255.255.255.0\r\n ip virtual-reassembly\r\n ip ospf 20 area 0\r\n!\r\nrouter ospf 10\r\n router-id 10.10.100.254\r\n log-adjacency-changes\r\n network 10.10.100.0 0.0.0.255 area 0\r\n!\r\nrouter ospf 20\r\n router-id 10.10.200.254\r\n log-adjacency-changes\r\n network 10.10.200.0 0.0.0.255 area 0\r\n!\r\nip forward-protocol nd\r\n!\r\n!\r\nip http server\r\nno ip http secure-server\r\n!\r\n!\r\ncontrol-plane\r\n!\r\n!\r\n!\r\nline con 0\r\n exec-timeout 0 0\r\nline aux 0\r\nline vty 0 4\r\n exec-timeout 0 0\r\n login\r\n!\r\n!\r\nend<\/pre>\n
        For this kind of switch, you also need to use the “vlan database” to create the VLANs.<\/p>\n
        It is a fairly standard setup. Ports fastE 1\/13, 1\/14, and 1\/15 are configured as trunks. Port fastE 1\/12 is the same, but it is connected directly to Gig0\/0 on the ASA. Do not be confused with the “IP virtual re-assembly” and OSPF setup.<\/p>\n
        I was testing NAT, and then I decided to pair the switch with the ASA running OSPF. You do not need them.<\/p>\n
        You will notice that I created a couple of SVI interfaces, VLAN 100 and VLAN 200.<\/p>\n
        They got IP addresses (but not the gateway addresses, which will be\u00a0configured at the ASA). In this fashion, I can test that I can ping the VMs on the PVE cluster.<\/p>\n
        If everything is set up correctly, you should be able to ping from the VMs to the VLANs configured on the switch.<\/p>\n
        Congratulations, you have trunking configured and can access devices outside PVE. You could, in principle, now have devices outside the cluster. For example, cameras that need to talk to a server on the cluster.<\/p>\n
        ASA<\/h4>\n
        Configuring the ASA is not that difficult., I will not go over how to set up rules or configure NAT. It will depend on your device. The relevant bits are below:<\/p>\n
        interface GigabitEthernet0\/0\r\n no nameif\r\n no security-level\r\n no ip address\r\n!\r\ninterface GigabitEthernet0\/0.100\r\n vlan 100\r\n nameif lan100\r\n security-level 100\r\n ip address 10.10.100.1 255.255.255.0 \r\n ipv6 address fc00:1::1\/64\r\n ipv6 enable\r\n!\r\ninterface GigabitEthernet0\/0.200\r\n vlan 200\r\n nameif lan200\r\n security-level 100\r\n ip address 10.10.200.1 255.255.255.0 \r\n ipv6 address fc00:1:0:1::1\/64\r\n ipv6 enable\r\n!\r\ninterface GigabitEthernet0\/1\r\n nameif outside\r\n security-level 0\r\n ip address dhcp setroute \r\n ipv6 address autoconfig\r\n ipv6 enable\r\n ipv6 nd suppress-ra<\/pre>\n
        We configure each interface with the gateway IPs we set up the VLANs earlier. Notice that on an ASA, you configure sub-interfaces, and the ASA takes care of negotiating the trunking with the switch.<\/p>\n
        You do not need the IPv6 stuff. I decided to add IPv6 so the VMs would get it. Perhaps a topic for another blog.<\/p>\n
        If the firewall can access the Internet and you configured NAT, VMs will be able to access it.<\/p>\n
        In addition, you can now control what the VMs in the cluster can access.<\/p>\n
        Conclusions<\/h3>\n
        There you have it. Creating VLANs using OvS is not that complicated, at least for the simple setup we just showed.<\/p>\n
        Hope you enjoyed the blog.<\/p>\n
        Happy networking.<\/p>\n
        Ciao.<\/p>\n","protected":false},"excerpt":{"rendered":"
        Hello there, I thought I was done with PVE, then I noticed that some users had issues with VLANs, trunks, etc. In principle it should fairly straight forward to configure VLANs (not VXLANs) however, using the native Linux way could be a bit complicated sometimes. OvS on the other hand was created with virtualization in … <\/p>\n
        Continue reading “PVE, VLANs, Trunks, Oh My!”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-720","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/720","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=720"}],"version-history":[{"count":51,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/720\/revisions"}],"predecessor-version":[{"id":772,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=\/wp\/v2\/posts\/720\/revisions\/772"}],"wp:attachment":[{"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=720"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=720"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.miguelsarmiento.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=720"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}