I recently installed proxmox on my server, and now I'm trying to get networking in order.
The problem that I have, is that for a certain VLAN, the DHCP response from my DHCP server doesn't seem to end up in my VM. It does seem to reach proxmox, but there's nothing arriving in the VM.
What I'm trying to achieve, is the following:
From my ISP, I've got a /29 subnet. I have a Mikrotik router running RouterOS 7. I configured a VLAN (id=200) in which I want to add devices and VMs that get an IP from that subnet. Besides that, I've got a standard 192.168.some.thing subnet that is used by every other piece of hardware in my home network.
So, I want to add a VM in proxmox with two virtual interfaces, one that gets an IP from the 192.168 subnet, and another that gets an IP from the /29 subnet of my ISP.
My network topology is as follows:
ISP <-> Mikrotik Router <-> Server running Proxmox <-> switch <-> PCs
My server has two physical interfaces that I bridged together (vmbr0). I did this, because my switch isn't managed, and I read that it strips the VLAN tags from packets. So the switch is running further downstream, and there's a few pieces of hardware behind it. I don't think that it influences my setup too much, but just so you get the whole picture.
So, what I did in proxmox to achieve things there:
I added an extra interface to my VM, and set the VLAN tag to 200 I made the bridge (vmbr0) VLAN aware However, the interface does not seem to get an IP. The strange thing is, that a DHCP lease is provided from my router. Even stranger: it seems to end up in proxmox, but it stops there (it doesn't reach the VM).
What I did to analyze the problem, is running tcpdump in order to gather the DHCP traffic:
tcpdump -i vmbr0v200 -pvn port 67 and port 68
First, I ran this on proxmox. This gave me the following:
10:31:59.579453 IP (tos 0x0, ttl 16, id 0, offset 0, flags [none], proto UDP (17), length 328)
45.xxx.xxx.233.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300, xid 0x6c990843, Flags [none]
Your-IP 45.xxx.xxx.234
Server-IP 45.xxx.xxx.233
Client-Ethernet-Address 9a:01:73:ba:ab:15
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Subnet-Mask (1), length 4: 255.255.255.248
Default-Gateway (3), length 4: 45.xxx.xxx.233
Domain-Name-Server (6), length 8: 8.8.8.8,8.8.4.4
Lease-Time (51), length 4: 600
Server-ID (54), length 4: 45.xxx.xxx.233
So, this all seemed good! A correct IP address was given out, the interface MAC-address was the right one (9a:01:73:ba:ab:15 is of the virtual interface that I put the VLAN tag on). But, nothing on the VM side.
When I ran the same thing on the VM side:
tcpdump -i ens19 -pvn port 67 and port 68
All I got was the request:
14:34:54.988228 IP (tos 0xc0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 322)
0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9a:01:73:ba:ab:15, length 294, xid 0xc62d7779, secs 1342, Flags [none]
Client-Ethernet-Address 9a:01:73:ba:ab:15
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Client-ID (61), length 19: hardware-type 255, 9e:37:69:8c:00:02:00:00:ab:11:f2:16:3b:8b:f0:03:d6:c8
Parameter-Request (55), length 11:
Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), Static-Route (33), NTP (42)
Unknown (119), Unknown (120), Classless-Static-Route (121)
MSZ (57), length 2: 576
Hostname (12), length 10: "kubernetes"
Does anyone know what could go wrong in this setup?
Some additional findings: I know that my router implicitly marks VLAN-less packets with VLAN 1. So on my virtual interface where the 192.168 subnet is, I set the VLAN tag to 1. This works, I do get an IP from my router. I ran tcpdump in my VM, and the output is as follows:
tcpdump: listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:42:37.582434 IP (tos 0xc0, ttl 64, id 11934, offset 0, flags [DF], proto UDP (17), length 322)
192.168.88.157.68 > 192.168.88.1.67: BOOTP/DHCP, Request from c6:13:ea:18:09:cc, length 294, xid 0x120faea6, secs 1805, Flags [none]
Client-IP 192.168.88.157
Client-Ethernet-Address c6:13:ea:18:09:cc
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Request
Client-ID (61), length 19: hardware-type 255, ca:53:09:5a:00:02:00:00:ab:11:f2:16:3b:8b:f0:03:d6:c8
Parameter-Request (55), length 11:
Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Hostname (12)
Domain-Name (15), MTU (26), Static-Route (33), NTP (42)
Unknown (119), Unknown (120), Classless-Static-Route (121)
MSZ (57), length 2: 576
Hostname (12), length 10: "kubernetes"
14:42:37.593791 IP (tos 0x0, ttl 16, id 0, offset 0, flags [none], proto UDP (17), length 328)
192.168.88.1.67 > 192.168.88.157.68: BOOTP/DHCP, Reply, length 300, xid 0x120faea6, Flags [none]
Client-IP 192.168.88.157
Your-IP 192.168.88.157
Server-IP 192.168.88.1
Client-Ethernet-Address c6:13:ea:18:09:cc
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: ACK
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 192.168.88.1
Domain-Name-Server (6), length 8: 192.168.88.1,8.8.8.8
Lease-Time (51), length 4: 600
Server-ID (54), length 4: 192.168.88.1
EDIT:
So, I did some more digging, and it seems that on Proxmox, everything is fine:
nmap --script broadcast-dhcp-discover -e vmbr0
Results in:
Starting Nmap 7.80 ( https://nmap.org ) at 2023-07-06 12:32 CEST
Pre-scan script results:
| broadcast-dhcp-discover:
| Response 1 of 1:
| IP Offered: 192.168.88.155
| DHCP Message Type: DHCPOFFER
| Subnet Mask: 255.255.255.0
| Router: 192.168.88.1
| Domain Name Server: 192.168.88.1, 8.8.8.8
| IP Address Lease Time: 10m00s
|_ Server Identifier: 192.168.88.1
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 0.85 seconds
And nmap --script broadcast-dhcp-discover -e vmbr0v200
Results in:
Starting Nmap 7.80 ( https://nmap.org ) at 2023-07-06 12:32 CEST
Pre-scan script results:
| broadcast-dhcp-discover:
| Response 1 of 1:
| IP Offered: 45.xxx.xxx.236
| DHCP Message Type: DHCPOFFER
| Subnet Mask: 255.255.255.248
| Router: 45.xxx.xxx.233
| Domain Name Server: 8.8.8.8, 8.8.4.4
| IP Address Lease Time: 10m00s
|_ Server Identifier: 45.xxx.xxx.233
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 0.72 seconds
EDIT 2:
Some more relevant information about my setup:
Content of /etc/network/interfaces :
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
iface enp99s0f0 inet manual
iface enx3a0de3575b59 inet manual
iface enp99s0f1 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.88.200/24
gateway 192.168.88.1
bridge-ports enp99s0f0 enp99s0f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
The firewall setup on proxmox itself is all default: I didn't do anything to configure it yet. So proxmox should pass everything through, and on the VM side there should be nothing blocking anything. Also, for the other subnet, DHCP leases are coming through normally.
The following are the relevant parts of my mikrotik configuration: Please note: ether5 is the physical interface where my proxmox server is connected to.
/interface vlan
add interface=bridge name=vlan-public vlan-id=200
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp_pool_freedom ranges=45.xxx.xxx.234-45.xxx.xxx.238
/ip dhcp-server
add address-pool=dhcp interface=bridge lease-time=10m name=defconf
add address-pool=dhcp_pool_freedom interface=vlan-public lease-time=10m name=\
dhcp-freedom
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge interface=vlan-public
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
192.168.88.0
add address=45.xxx.xxx.233/29 interface=vlan-public network=45.xxx.xxx.232
/ip dhcp-server network
add address=45.xxx.xxx.232/29 dns-server=8.8.8.8,8.8.4.4 gateway=45.xxx.xxx.233 \
netmask=29
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8
/ip dns static
add address=192.168.88.1 disabled=yes name=router.lan
add address=8.8.8.8 name="Google DNS 1"
/ip firewall filter
add action=accept chain=forward connection-nat-state=dstnat disabled=yes
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input src-address=!192.168.88.0/24
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=drop chain=forward comment=\
"defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=passthrough chain=prerouting
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
ipsec-policy=out,none out-interface-list=WAN src-address=\
!45.xxx.xxx.232/29
add action=masquerade chain=srcnat out-interface=ether1
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=pppoe-freedom routing-table=\
to-freedom
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=pppoe-freedom \
pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
target-scope=10
add disabled=no distance=1 dst-address=45.xxx.xxx.232/29 gateway=45.xxx.xxx.232 \
pref-src="" routing-table=main suppress-hw-offload=no
Please note that irrelevant entries are omitted in the export above.
/etc/network/interfacesand describe the firewall setup), and show Mikrotik confiuration (export of relevant parts). P.S. Don't rely on any implicit behaviour regarding VLAN ID 1. If you are going to use VLANs, move everything into non-default VLANs (other than 1), and never ever use VLAN ID 1 for anything useful. Think of it it as a catch-all black hole for misconfigured or not configured ports.