Configuring NAT with Traffic Control for Nginx/Apache on localhost, cache miss

Question

Apache is on 127.0.100.2 and Nginx is on 127.0.100.3, with Traffic Control performing DNAT from the former to the latter (for source 127.0.0.1) and SNAT from the latter to the former (for destination 127.0.0.1):

tc qdisc add dev lo root handle 1:0 prio
tc filter add dev lo parent 1:0 protocol ip prio 1 u32 \
  match ip src 127.0.0.1 match ip dst 127.0.100.2 \
  action nat ingress 127.0.100.2 127.0.100.3 
tc filter add dev lo parent 1:0 protocol ip prio 1 u32 \
  match ip src 127.0.100.3 match ip dst 127.0.0.1 \
  action nat egress 127.0.100.3 127.0.100.2 

Cache hits work, misses don't, which I understand: Upon a cache miss, content is delivered by Apache,† but NAT will prevent that. What configuration should be used?

Nginx configuration file:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g 
    inactive=60m use_temp_path=off;
server {
  listen 127.0.100.3;
  location / {
    proxy_cache my_cache;
    proxy_pass http://127.0.100.2;
  }
}

^{† Extra question: Technically speaking, how does Nginx handle a cache miss? Wireshark doesn't reveal anything obvious: I see SYN; SYN,ACK; ACK; HTTP HEAD; ACK between 127.0.0.1 and 127.0.100.3, then a similar TCP handshake between 127.0.0.1 and 127.0.100.2 followed by HTTP GET and payload delivery. Presumably, the final ACK between 127.0.0.1 and 127.0.100.3 is doing something clever, but what?}

Answer 1

What configuration should be used?

Generally it depends on your very circumstances. But obviously if you don't want NAT to jump in between Nginx and Apache, Nginx should connect to Apache using different dest. or src. address. For e. g., you can assign second IP for Apache to listen on, and point proxy_pass there, or make Nginx to connect using a different outgoing IP. That's kinda blunt but should do.

Since tc's NAT is stateless other options (like handling TCP's SYN/ACKs) are kinda cumbersome.

Answer 2

I think you're expecting the nginx connection to apache to come from 127.0.100.3 because you have configured nginx to listen on that IP.

The address bound for listening (incoming) sockets doesn't apply to outgoing connections. With your current configuration, the connection from nginx to apache should come from 127.0.0.1 as that's the primary IP bound to the loopback interface and thus the default for outgoing connections using that interface.

You can use proxy_bind to override that default and bind a specific source address for nginx outgoing connections.Relevant documentation on docs.nginx.com

I believe you'll achieve the behavior you expect by adding proxy_bind 127.0.100.3; to your location / { ... } configuration.

Also, I'd suggest you consider using 127.0.100.4 for your proxy_bind, to ease your filtering rules.