Apache: Truncated or oversized response headers after graceful restart

Question

After graceful restart, some python wsgi processes fail with 500 (internal server error):

[20:06:33] [mpm_prefork:notice] AH00171: Graceful restart requested, doing restart
[20:06:36] [wsgi:error] Truncated or oversized response headers received from daemon process 'foo_vums_di2865'

(The above log lines were cut to make them easier to read)

What could be the reason?

Is there a way to restart apache without creating 500 responses.

Delays of some seconds are ok, but failures should get avoided.

Versions:

• Apache/2.4.34
• wsgi: 4.5.18

Answer 1

1. Possible reason: Not your fault. Upgrade your wsgi module. See second item in Bugs-Fixed-List.

Version 4.5.21

Bugs Fixed

• Installation using pip or setup.py install was failing on MacOS X High Sierra with latest Xcode as Apple doesn’t even include the apxs program at all.

• Speculated that crashes on daemon process shutdown were being caused by a race condition around accessing Python C API when interpreter was being destroyed. There was a check in place to avoid this but may not have been robust enough depending on how memory cache worked for threads running across multi core machine. Now use a dedicated thread mutex to avoid race condition between main process thread and Python interpreter deadlock detection thread.

2. If the problem still occurs, add more logging to apache and your application:

   a. In your apache configuration, set LogLevel to info to generate more messages when the server restart / server error occurs. This also logs mod_wsgi informational stuff or possible segmentation faults.

   b. If these messages don't reveal anything suspicious, add request/response-logging to your application (input headers, response status and response headers).

FYI: Do you restart apache to reflect changes in your application code? If you want to trigger reloading of your code in deamon mode, you can touch the changed script file (update the timestamp). The deamon process is then shutdown and restarted automatically.

Additionally, you could setup a (password protected) page to force the daemon process to be restarted. See Reloading In Daemon Mode / Restarting Daemon Processes