Lighttpd big uploads memory consumption

Question

Consider the following scenario:

• I am running lighttpd-1.4.19 and I am handling big uploads (hundreds of MBs but less than 1GB) through it.
• Server is running on Ubuntu 8.04 LTS.
• Files are temporarily written to /var/tmp.
• PHP5 is handling the end-result through fastcgi.
• During an upload, upload chunks are written in the temporary directory.

The weird thing is that although the chunks are written on disk, free memory on the server is reducing gradually up to the full file size during the upload. I would understand RAM diminishes by the size of each chunk (let's say 1-2MB) until they're written on disk but I don't understand why it is by the whole file size (let's say 300MB).

1. Would you know why the RAM would be taken for the whole file although chunks are written on disk? Is it possible /var/tmp is mounted directly in memory? Versus /tmp?
2. If not, would you know how to diagnose and how to pinpoint which process is taking up the memory? I didn't see the lighttpd nor php-cgi process increasing with ps but maybe I'm not looking at the right thing.
3. Or better yet, would you have a recommendation on a good web server (Apache, Nginx, lighttpd, etc.) and config to handle big uploads without storing everything in RAM?

Thanks!

Answer 1

If the lighttpd or php-cgi are not increasing in size, then don't worry - the increase you're seeing isn't in the RAM consumption of those processes, but is instead in the kernel's use of disk cache. Basically since you just wrote those large files out, the kernel keeps them around in memory on the assumption that you might want to read them again. It will of course drop this cache if that memory is needed by other processes.