Nginx FastCGI buffer spooling and epoll loop blocking

The Observation

A pattern of intermittent 502 Bad Gateway responses and latency spikes appeared on a specific API endpoint utilized by a fleet management system. The environment consists of Nginx 1.24 and PHP-FPM 8.2 running on Debian 12. The hardware is a standard cloud compute instance with 8 vCPUs, 16GB of RAM, and a generic block storage volume provisioned at 3000 IOPS.

The application is a deployment of the Tuning - Car Detailing & Dealer Shop WordPress theme, customized to ingest and export high-resolution vehicle inspection reports via a scheduled cron endpoint. The latency anomalies only occurred during the automated synchronization of these reports.

CPU utilization remained below 20%. Memory usage was static. PHP-FPM slow logs showed no execution delays within the PHP userland. MySQL query execution times were strictly under 50ms. The application logic was executing efficiently, yet the client connection was timing out or receiving a 502 error from the Nginx reverse proxy.

Analyzing the File Descriptor State

When standard application metrics fail to highlight a bottleneck, the investigation must move to the system layer, specifically observing the state of process file descriptors and socket buffers.

I isolated a single Nginx worker process during the synchronization window and utilized lsof combined with watch to sample the open file descriptors every 0.1 seconds.

watch -n 0.1 'lsof -p 48201 | grep "/var/lib/nginx"'

The output revealed a transient pattern:

nginx   48201 www-data   34w   REG  259,1  12482010  12401 /var/lib/nginx/fastcgi/8/01/0000000108 (deleted)
nginx   48201 www-data   35w   REG  259,1  15820100  12402 /var/lib/nginx/fastcgi/9/01/0000000109 (deleted)

The Nginx worker was actively opening and writing to temporary files within the fastcgi spool directory. These files were immediately unlinked (deleted) upon creation to ensure they were removed from the filesystem namespace if the process crashed, but the inodes remained open and consuming disk space until the file descriptor was closed.

The size of these files rapidly grew to 12MB-15MB before disappearing. This indicated that the response payload from PHP-FPM was exceeding the allocated memory buffers in Nginx, forcing Nginx to spool the response to the physical block device.

The Nginx Event Pipe Architecture

To understand why spooling to disk causes 502 errors and latency, it is necessary to examine the Nginx source code, specifically the ngx_event_pipe architecture which handles the transfer of data from an upstream server (PHP-FPM) to the downstream client.

Nginx uses an asynchronous, event-driven model based on epoll. A single worker thread handles thousands of concurrent connections by executing a continuous event loop (ngx_process_events_and_timers).

When Nginx receives data from FastCGI, it reads it into memory buffers defined by the fastcgi_buffers directive.

// src/event/ngx_event_pipe.h
struct ngx_event_pipe_s {
    ngx_connection_t  *upstream;
    ngx_connection_t  *downstream;

    ngx_chain_t       *free_raw_bufs;
    ngx_chain_t       *in;
    ngx_chain_t       **last_in;

    ngx_chain_t       *out;
    ngx_chain_t       *free;
    ngx_chain_t       *busy;

    /* ... */
    ngx_temp_file_t   *temp_file;
    ssize_t            temp_file_write_size;
    /* ... */
};

If the upstream response is larger than the configured memory buffers, Nginx invokes ngx_event_pipe_write_chain_to_temp_file.

// src/event/ngx_event_pipe.c
static ngx_int_t
ngx_event_pipe_write_chain_to_temp_file(ngx_event_pipe_t *p)
{
    ssize_t       size, bsize, n;
    ngx_buf_t    *b;
    ngx_chain_t  *cl, *tl, *out, **ll, **last_out, **last_free;

    /* ... buffer calculation logic ... */

    n = ngx_write_chain_to_temp_file(p->temp_file, out);

    if (n == NGX_ERROR) {
        return NGX_ABORT;
    }

    /* ... */
}

The critical flaw in this specific transaction path is the underlying write system call. While Nginx uses non-blocking I/O for network sockets, standard filesystem writes in Linux (without specific io_uring or asynchronous I/O configurations) are synchronous.

If the underlying block storage device experiences latency, the write system call blocks the execution of the entire Nginx worker process. The worker enters the D state (Uninterruptible Sleep).

During this block, the worker cannot return to the epoll_wait loop. It cannot accept new connections, it cannot read network buffers for existing connections, and it cannot send keep-alives. If the block duration exceeds the upstream or downstream timeout thresholds, or if the kernel network buffers fill up, connections are dropped or 502 Bad Gateway errors are returned.

Block Storage I/O Contention

I attached iostat to monitor the block device during the synchronization window to verify the storage latency.

iostat -x -d 1 /dev/nvme0n1

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme0n1           0.00     0.00    0.00  412.00     0.00 52401.00   254.37     4.12   10.02    0.00   10.02   2.42 100.00
nvme0n1           0.00     0.00    0.00  389.00     0.00 48201.00   247.81    12.41   31.90    0.00   31.90   2.57 100.00
nvme0n1           0.00     0.00    0.00  402.00     0.00 50102.00   249.26    34.10   84.82    0.00   84.82   2.48 100.00

The %util (utilization) was pegged at 100%. More importantly, the w_await (average time in milliseconds for write requests issued to the device to be served) climbed from 10ms to nearly 85ms.

Because the cloud block volume was strictly capped at 3000 IOPS and 125MB/s throughput, the sudden burst of 15MB temporary files from multiple concurrent API requests saturated the storage controller's queue. The Linux kernel, handling the write() system call from Nginx, submitted the bios to the block layer, where they queued in the mq-deadline scheduler.

The Nginx worker thread, waiting for the write() call to return, stalled for 85+ milliseconds per buffer flush.

FastCGI Protocol and Application Payload

To determine why the PHP-FPM payload was reaching 15MB, I captured the network traffic over the Unix domain socket separating Nginx and PHP-FPM using socat as an intermediary proxy, or by utilizing tcpdump if configured over loopback. In this instance, I used ss to inspect the Unix socket memory buffers.

ss -x -a --memory | grep fastcgi

u_str ESTAB 0 8420102 * 419201 * 0 
	 skmem:(r0,rb212992,t0,tb212992,f8420102,w0,o0,bl0,d0)

The receive queue (Recv-Q) for the socket was completely filled. PHP-FPM was attempting to push data faster than Nginx could read it (because Nginx was blocked on disk I/O).

I extracted the raw FastCGI stream. The FastCGI protocol encapsulates data in a specific binary record format.

typedef struct {
    unsigned char version;
    unsigned char type;
    unsigned char requestIdB1;
    unsigned char requestIdB0;
    unsigned char contentLengthB1;
    unsigned char contentLengthB0;
    unsigned char paddingLength;
    unsigned char reserved;
} FCGI_Header;

A hex dump of the FCGI_STDOUT records revealed the exact application payload.

00000000  01 06 00 01 1f f8 08 00  7b 22 76 65 68 69 63 6c  |........{"vehicl|
00000010  65 5f 69 64 22 3a 34 38  32 30 31 2c 22 69 6e 73  |e_id":48201,"ins|
00000020  70 65 63 74 69 6f 6e 5f  64 61 74 61 22 3a 5b 7b  |pection_data":[{|
00000030  22 70 61 72 74 22 3a 22  63 68 61 73 73 69 73 22  |"part":"chassis"|
00000040  2c 22 69 6d 61 67 65 22  3a 22 69 56 42 4f 52 77  |,"image":"iVBORw|
00000050  30 4b 47 67 6f 41 41 41  41 4e 53 55 68 45 55 67  |0KGgoAAAANSUhEUg|

The payload consisted of JSON data containing Base64 encoded images. The synchronization endpoint was fetching complete vehicle inspection records, embedding multiple 2MB-3MB images directly within the JSON array.

Unlike a typical Free Download WooCommerce Theme which standardizes pagination and limits API responses to lightweight product metadata (URLs to images, not the binary data itself), this specific API endpoint was designed to transmit the entire offline-capable report payload in a single HTTP transaction.

This resulted in a 15MB contiguous string being output by the PHP json_encode() function.

Buffer Sizing Mechanics

The default Nginx FastCGI buffer configuration is highly conservative, designed for typical HTML document delivery.

fastcgi_buffers 8 4k|8k;
fastcgi_buffer_size 4k|8k;

On a 64-bit architecture with a 4KB memory page size, the default allocates 8 buffers of 4KB each (32KB total) per connection. If the response exceeds 32KB, Nginx immediately opens a temporary file.

When dealing with a 15MB payload, Nginx exhausts the 32KB memory buffer instantly. It then begins a cycle of reading 32KB from the PHP-FPM socket, writing 32KB to the temporary file on disk, and repeating. For a 15MB file, this requires approximately 480 separate write() system calls.

Under ideal storage conditions, the OS page cache absorbs these writes. The kernel marks the pages as "dirty" and flushes them to disk asynchronously based on the vm.dirty_background_ratio and vm.dirty_ratio sysctl parameters.

sysctl vm.dirty_background_ratio vm.dirty_ratio

vm.dirty_background_ratio = 10
vm.dirty_ratio = 20

However, because the synchronization cron job triggered dozens of these requests simultaneously, the aggregate dirty memory rapidly exceeded the background flush threshold. The kernel flush threads activated, saturating the block device IOPS. Subsequent write() calls from Nginx hit the block layer while the device queue was full, triggering synchronous delays (w_await).

Mitigating the Blocking Sequence

To eliminate the 502 errors and the epoll loop blocking, the disk spooling mechanism must be bypassed entirely. The response payload must reside strictly in RAM until it is transmitted to the network socket.

This requires calculating the maximum expected payload size and allocating sufficient memory buffers in the Nginx configuration.

Maximum expected payload: ~16MB. Desired buffer size per block: 32KB. Number of buffers required: 16MB / 32KB = 512.

Furthermore, Nginx imposes a hard limit on the maximum size of the temporary file it will create.

fastcgi_max_temp_file_size 1024m;

To strictly enforce an in-memory-only pipeline and absolutely prevent Nginx from ever executing a blocking open or write call for FastCGI buffers, the fastcgi_max_temp_file_size directive must be set to 0.

When set to 0, Nginx is prohibited from buffering responses to disk. If the upstream response exceeds the configured fastcgi_buffers, Nginx will read data from FPM only as fast as it can be transmitted downstream to the client (synchronous backpressure). This prevents disk I/O but requires sufficient network bandwidth to the client, otherwise the PHP-FPM process remains occupied waiting for socket buffers to drain.

Because the clients executing the sync job were located on high-bandwidth corporate network segments, applying network backpressure was architecturally sounder than blocking the Nginx event loop on slow disk I/O.

Resolution Configuration

Apply the following directives to the specific location block handling the API endpoint to allocate sufficient memory and disable disk spooling.

location /wp-json/tuning/v1/sync {
    # ... existing fastcgi_pass directives ...

    fastcgi_buffer_size 32k;
    fastcgi_buffers 512 32k;
    fastcgi_busy_buffers_size 64k;
    fastcgi_max_temp_file_size 0;
}