Routing VM with two IPs in the same subnet

Question

I have two NICs with IPs in the same subnet.

IP1: 10.10.10.142 - Gateway: 10.10.10.129 - dev: ens192. IP2: 10.10.10.143 - Gateway: 10.10.10.129 - dev: ens256.

I want to configure static route to the IP [192.168.1.5] as following:

192.168.1.5 via 10.10.10.129 dev ens192

192.168.1.5 via 10.10.10.129 dev ens256

Now I have an issue that when [192.168.1.5] send to [10.10.10.143] I'm getting reply from [10.10.10.142].

How can I fix the static route to reply from the same IP which is receiving?

Answer 1

You are correct Tom Yan, and I apologize for the incorrect information provided earlier. The rule I mentioned does not make sense in this context, and your observation about the desired behavior is accurate.

If the goal is to reply to traffic from 192.168.1.5 using the same NIC and IP address to which the original traffic was sent, you should indeed rely on connection tracking (conntrack) and marking packets appropriately.

Here's how you can achieve this using conntrack and marking packets:

1. Delete the conflicting static route:

   If you want traffic from 192.168.1.5 to reach its destination correctly, remove the conflicting static route:

   ip route del 192.168.1.5 via 10.10.10.129 dev ens192
   

2. Configure conntrack and packet marking:

   You can use iptables to mark packets based on their source IP address when they leave the system. For example, if traffic comes from 192.168.1.5 via ens192, mark it as follows:

   iptables -t mangle -A OUTPUT -s 192.168.1.5 -o ens192 -j MARK --set-mark 1
   

   And if traffic comes from 192.168.1.5 via ens256, mark it like this:

   iptables -t mangle -A OUTPUT -s 192.168.1.5 -o ens256 -j MARK --set-mark 2
   

3. Create routing tables for each mark:

   Create two custom routing tables for each mark, which will determine the outgoing interface:

   echo "1 nic1" >> /etc/iproute2/rt_tables
   echo "2 nic2" >> /etc/iproute2/rt_tables
   

4. Add routes for each routing table:

   Now, add the routes to the two custom routing tables, specifying the outgoing interface and gateway:

   For table "nic1":

   ip route add default via 10.10.10.129 dev ens192 table nic1
   

   For table "nic2":

   ip route add default via 10.10.10.129 dev ens256 table nic2
   

5. Create ip rule entries to select the routing table based on marks:

   Create rules that select the routing table based on the mark:

   For traffic marked as "1" (ens192):

   ip rule add fwmark 1 table nic1
   

   For traffic marked as "2" (ens256):

   ip rule add fwmark 2 table nic2
   

6. Flush the routing cache:

   ip route flush cache
   

With these changes, packets from 192.168.1.5 should be marked correctly based on their source interface, and the corresponding routing table and interface should be used for the reply. This should ensure that traffic from 192.168.1.5 receives replies from the same interface it was originally sent through.