Using lxc containers, I want to simulate network components individually. For this, I am trying to embed OVS in an lxc container.
Initially, I started with the following setup, which works properly and the two hosts can ping each other:
+---------------+ +---------------+
| host1 | | host2 |
| testnode-eth0 | | testnode-eth1 |
+-------|-------+ +-------|-------+
| |
vbr0--------------vbr1
host1 and host2 are lxc containers, binding via testnode-ethX to vbrX, which are a connected veth pair. testnode-ethX have IP addresses and correct routing attached to it.
Now I boot up an OVS instance in the middle, naming it switch1:
+---------------+ +-------------------------------+ +---------------+
| host1 | | switch1 | | host2 |
| testnode-eth0 | | testnode-eth1 testnode-eth3 | | testnode-eth2 |
+-------|-------+ +-------|---------------|-------+ +-------|-------+
| | | |
vbr0--------------vbr1 vbr3--------------vbr2
The setup is analog to the above setup. This time, testnode-eth0, switch1 and testnode-eth2 have IP addresses. The switch runs in standalone mode, the configuration looks like this:
3bc3442a-1e83-4e51-ac80-10a618be0a62
Bridge switch1
fail_mode: standalone
Port switch1
Interface switch1
type: internal
Port testnode-eth1
Interface testnode-eth1
Port testnode-eth3
Interface testnode-eth3
ovs_version: "2.13.8"
----
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000000000000001
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(testnode-eth1): addr:00:16:3e:77:1a:af
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
2(testnode-eth3): addr:00:16:3e:a8:d6:0f
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(switch1): addr:00:11:22:33:44:55
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
When I send a ping from host1 to host2, host1 first tries to arp resolve host2. Host1 sends an arp request, which reaches host2 just fine. Host2 then sends an arp reply which I can observe with tcpdump on vrb2 and vbr3. But the reply then never reaches testnode-eth3. By dumping flows, I can see the packet counter increase with arp requests, but no replies are received on the interface. Example:
OFPST_PORT reply (xid=0x2): 3 ports
port LOCAL: rx pkts=147, bytes=5652, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=16, bytes=1216, drop=0, errs=0, coll=0
port "testnode-eth1": rx pkts=123, bytes=5934, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=56, bytes=4208, drop=0, errs=0, coll=0
port "testnode-eth3": rx pkts=24, bytes=1776, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=155, bytes=8366, drop=0, errs=0, coll=0
==> send arp requests from host1 to host2, host 2 replies with equal amount of replies
OFPST_PORT reply (xid=0x2): 3 ports
port LOCAL: rx pkts=153, bytes=5820, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=16, bytes=1216, drop=0, errs=0, coll=0
port "testnode-eth1": rx pkts=129, bytes=6186, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=56, bytes=4208, drop=0, errs=0, coll=0
port "testnode-eth3": rx pkts=24, bytes=1776, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=161, bytes=8618, drop=0, errs=0, coll=0
I can see rx counter on eth1 and tx counter on eth3 increase (arp request), but the arp response is never received on eth3 rx. Which leads me to believe that targeted traffic (arp response) is not forwarded to the lxc container. How can I make vbr3 forward all traffic to testnode-eth3, and how can I make testnode-eth3 accept all packets, even packets not targeted towards it?
What I tried so far:
- Enable ip_forward (host + containers)
- Disable arp_filter (host + containers + all interfaces)
- Add ips to testnode-eth1 and testnode-eth3 (will not work either way)
- Run lxc container with priviledges or unpriviledged
- Specify routing on switch1 specifically towards certain ports or towards the switch1 interace itself. No difference.
- Verified that iptables does not interfere. All chains (INPUT/OUTPUT/FORWARD) ACCEPT always, in containers iptables is not present.
- All interfaces are up
- All interfaces were tried with and without promiscuous mode
- Tried enabling
net.ipv4.conf.*.forwardingeverywhere
Some more info:
- Host containers are on Ubuntu 18.04
- Switch container is on Ubuntu 20.04 with OVS 2.13.8 in standalone mode
- Machine runs on Manjaro 5.19.1-3
Code used to start up the container network:
# Bootstrap bridge interfaces to connect containers
ip link add vbr0 type veth peer vbr1
ip link set dev vbr0 up
ip link set dev vbr1 up
ip link add vbr2 type veth peer vbr3
ip link set dev vbr2 up
ip link set dev vbr3 up
# Create switch container and attach to bridge ends
lxc init ovs switch1
lxc config set switch1 security.privileged true
lxc start switch1
lxc network attach vbr1 switch1 testnode-eth1 testnode-eth1
lxc exec switch1 -- ip link set dev testnode-eth1 up
lxc network attach vbr3 switch1 testnode-eth3 testnode-eth3
lxc exec switch1 -- ip link set dev testnode-eth3 up
# Restart ovs with custom parameters to be in full controll easily
lxc exec switch1 -- service openvswitch-switch stop
lxc exec switch1 -- service ovs-vswitchd stop
lxc exec switch1 -- service ovsdb-server stop
lxc exec switch1 -- rm -rf /var/run/openvswitch
lxc exec switch1 -- mkdir /var/run/openvswitch
lxc exec switch1 -- ovsdb-server --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile --verbose --detach
lxc exec switch1 -- ovs-vsctl init
lxc exec switch1 -- ovs-vswitchd --pidfile --detach
lxc exec switch1 -- ovs-vsctl -- --if-exists del-br switch1 -- add-br switch1 -- set bridge switch1 controller=[] stp_enable=false other_config:datapath-id=0000000000000001 fail_mode=standalone other-config:disable-in-band=true other-config:dp-desc=switch1 other-config:hwaddr=00:11:22:33:44:55 -- add-port switch1 testnode-eth1 -- add-port switch1 testnode-eth3
lxc exec switch1 -- ip link set dev switch1 up
lxc exec switch1 -- ip addr add dev switch1 10.0.0.8/16
lxc exec switch1 -- ip route add 10.0.0.2 via 10.0.0.8
lxc exec switch1 -- ip route add 10.0.0.4 via 10.0.0.8
# Boostrap two simple hosts
lxc init simple-host host1
lxc start host1
lxc network attach vbr0 host1 testnode-eth0 testnode-eth0
lxc exec host1 -- ip link set dev testnode-eth0 up
lxc exec host1 -- ip addr add dev testnode-eth0 10.0.0.2/16
lxc exec host1 -- ip route add 10.0.0.4 via 10.0.0.2
lxc init simple-host host2
lxc start host2
lxc network attach vbr2 host2 testnode-eth2 testnode-eth2
lxc exec host2 -- ip link set dev testnode-eth2 up
lxc exec host2 -- ip addr add dev testnode-eth2 10.0.0.4/16
lxc exec host2 -- ip route add 10.0.0.2 via 10.0.0.4
If you want to try this exact netork for yourself, here is some code to import my images (this switch image is based of Ubuntu 18.04, but is otherwise the same with the same issue):
wget https://friwi.me/testbed_img/ovs-ubuntu-18.04-minimal.tar.gz
wget https://friwi.me/testbed_img/simple-host-ubuntu-18.04-minimal.tar.gz
lxc image import ovs-ubuntu-18.04-minimal.tar.gz --alias ovs --public
lxc image import simple-host-ubuntu-18.04-minimal.tar.gz --alias simple-host --public
If you require any more info, I will gladly provide it. I've been stuck at this exact spot for days and cant seem to find the answer. All help is greatly appreciated.
