WSGI : Truncated or oversized response headers received from daemon process

Question

System Configuration : Apache2, Django 1.10, Python 3, Ubuntu 16.04 LTS

Django debug=True.

---

/var/log/apache2/error.log

[52:53.057967] [wsgi:error] [pid 4303] [client 1.1.1.22:24409] Timeout when reading response headers from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466726] [wsgi:error] [pid 4305] [client 1.1.1.10:9787] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466729] [wsgi:error] [pid 4304] [client 1.1.1.4:18417] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466726] [wsgi:error] [pid 4307] [client 1.1.1.22:35116] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.466756] [wsgi:error] [pid 4306] [client 1.1.1.22:19242] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467164] [wsgi:error] [pid 4336] [client 1.1.1.4:34187] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467212] [wsgi:error] [pid 4342] [client 1.1.1.22:28212] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py, referer: http://example.org/
[52:58.467282] [wsgi:error] [pid 4331] [client 1.1.1.22:31045] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py
[52:58.467426] [wsgi:error] [pid 4341] [client 1.1.1.70:22784] Truncated or oversized response headers received from daemon process 'example.org': /home/user/dir/project/main_app/wsgi.py, referer: http://example.org/

I do not know the cause of the error. But I have narrowed it down to the Django wsgi process. Since the server is hosting static files properly.

While Cloudflare sometimes shows 502 : Bad Gateway Error, the server itself shows 500 : Internal Server Error.

I have already tried restarting the server and checking Django's (debug) log file. There is no error information in the Django log files (at all).

---

How should I debug the problem? Since Django didn't log anything, I assume the problem could be caused in wsgi.

Note : The server was working fine earlier. I made some changes* (which are reverted back as-is); The Django shell works fine.

Changes*

1. Installed django-pandas, django-model-utils, numpy, scikit-learn
2. A program that utilises above libraries. (This change is reverted to original)

---

In other similar questions, the problem is caused during upload of a big file.

Answer 1

The cause of the problem was numpy.

Python C extension modules, like numpy, are known to cause timeouts when used under mod_wsgi.

Source : Answer by Sean F on Timeout when reading response headers from daemon process

A similar question which I didn't find on initial search answered and explained by author of mod_wsgi says -

Some third party packages for Python which use C extension modules, and this includes scipy and numpy, will only work in the Python main interpreter and cannot be used in sub interpreters as mod_wsgi by default uses. The result can be thread deadlock, incorrect behaviour or processes crashes.

Source : Answer by Graham Dumpleton on Non-responsive apache + mod_wsgi after installing scipy

Solution

Add the below line to your httpd.conf. In my case the file was /etc/apache2/apache2.conf.

WSGIApplicationGroup %{GLOBAL}

Answer 2

As others have mentioned, this is caused by the process crashing for a number of potential reasons. One reason being that some Python dependency isn't thread safe.

If that's the problem, one work around is to switch Apache's MPM type. The prefork type doesn't use threads, so if the problem is numpy crashing due to thread misuse, then that should fix it. The worker and event types use less memory, but also use threads, so they can encounter this error.

To determine which type you're currently using, run:

apachectl -V | grep -i mpm

If you see Server MPM: prefork then you're already using prefork, meaning the cause of the error might be something else. If it says "worker" or "event", then you can switch to prefork by running:

sudo a2dismod mpm_event
sudo a2dismod mpm_worker
sudo a2enmod mpm_prefork
sudo service apache2 restart

Note, the main downside of prefork is that, since it's not using threads, it consumes more memory.

Edit: I've since encountered this error due to other causes. Most recently, the problem is caused by a bug in Ubuntu's precompiled mod-wsgi system package as well as a bug in the Python psycopg2 package.

The solution for this was to switch from the system mod-wsgi to the Python package, as well as switch to the psycopg2-binary package:

sudo apt purge libapache2-mod-wsgi*
sudo apt install apache2-dev
pip uninstall psycopg2
pip install mod_wsgi psycopg2-binary

I also had to update my apache site configuration file by adding to the top:

LoadModule wsgi_module /usr/local/myproject/.env/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so

And change my WSGIDaemonProcess statement to use python-home instead of python-path, to something like:

WSGIDaemonProcess myproject python-home=/usr/local/myproject/.env processes=5 threads=15 display-name=%{GROUP} user=www-data group=www-data

I've encountered this for both Python2.7 and Python3.7 and the solution is the same, but the path to the mod_wsgi.so changes.

Answer 3

I have received a similar error while trying to run opencv using mod_wsgi and apache. I guess the problem was probably multiple threads with underlying C code trying to acquire GIL and failing.

I solved it by setting threads=1 and processes=32(in my case it was appropriate) in WSGIDaemonProcess directive.

PS: Late to the party but thought it could help someone.

Answer 4

In my case I had to change the WSGIDaemonProcess line from:

WSGIDaemonProcess wsgi processes=2 threads=4 display-name=%{GROUP} \
  python-path=/var/www/appname:/var/www/appname/venv/lib/python2.7/site-packages user=wsgi group=wsgi \
  home=/var/www/appname

to:

WSGIDaemonProcess appname user=wsgi group=wsgi processes=2 threads=4 display-name=%{GROUP} home=/var/www/appname

Answer 5

In my case, the problem was pymongo and PHP. As described in this GitHub issue https://github.com/GrahamDumpleton/mod_wsgi/issues/351:

If PHP is loading a client for MongoDB it can [create a conflict]. (...) PHP often preloads all extensions. Doesn't matter if hosted application uses it or not.

The way I solved it:

1. I ran my flask app from mod_wsgi-express: mod_wsgi-express start-server api.wsgi --user=www-data --group=www-data --host=0.0.0.0 --port=8443.
2. Then, in my httpd.conf, I redirected all traffic to my flask app alias using:

SSLEngine on
SSLProxyEngine On
ProxyPass /api http://localhost:8443/
ProxyPassReverse /api http://localhost:8443/