qdisc and QUIC throughput
fq_codel vs fq matters more than I expected on a host running a QUIC server with a handful
of long-lived flows. Default Debian vs default Ubuntu disagree by enough that I spent an hour
chasing a phantom bug before checking tc qdisc show. Setting the qdisc explicitly
in the deploy script saved this from ever being an issue again.
The mechanics: QUIC does its own pacing in userspace, but it relies on the kernel to actually honor the inter-packet spacing it asks for. fq_codel will reorder small packets ahead of large ones (good for latency on a mixed workload, bad for QUIC’s pacing assumptions). fq with SO_TXTIME-respecting earliest-departure will hand the packets out at the times the QUIC stack asked for. Throughput on a 100Mbit link goes from ~70 to ~95Mbit just by switching.
None of this is new — the QUIC people have been saying “use fq” in their
docs for years — but the part I missed was that the default isn’t consistent across
distros. cat /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max is also
worth looking at if you suspect bufferbloat on the egress NIC.
The cargo-culted setup that’s been working: tc qdisc replace dev eth0 root fq
plus net.core.default_qdisc=fq in /etc/sysctl.d/99-qdisc.conf. Belt
and suspenders — the systemd service can’t race the qdisc setup if the kernel default
is already what we want.