W-01
upstreamer
L7 reverse proxy · hyper + tokio
Routes by host and path, balances on EWMA latency, rate-limits each client IP with a token bucket, discovers upstreams from Kubernetes and swaps its whole config lock-free through ArcSwap without dropping a connection. Benchmarked against the proxies people actually run. A bimodal 40 ms tail turned out to be Nagle meeting delayed ACK: six attempts to find, one line to fix.
- Active + passive health checks
- Prometheus metrics, separate port
- jemalloc, ~6% CPU was in malloc
- Per-IP token buckets
| Parameter | Condition | Value |
|---|---|---|
| Throughput50 conns, 500k req | 50 conns, 500k req | 76,663req/s |
| Latency, p99.99lowest of all proxies tested | lowest of all proxies tested | 3.81ms |
| p99.99 before fixNagle + delayed ACK | Nagle + delayed ACK | 41ms |
| Concurrency cap100% 2xx, same as backend | 100% 2xx, same as backend | 4,800conns |
| upstreamer | 76,663 | |
|---|---|---|
| HAProxy | 70,503 | |
| nginx, auto | 69,477 | |
| nginx, 1 worker | 68,637 | |
| Caddy | 66,195 | |
| Apache | 57,518 | |
| 020k40k60k80k |
Same harness as Fig. 0. Backend alone, no proxy: 154,998 req/s.
| attempt 0first run | 41 |
|---|---|
| attempt 1jemalloc decay | 41 |
| attempt 2metrics thread | 41 |
| attempt 3cpu pinning | 41 |
| attempt 4arcswap removal | 41 |
| attempt 5string alloc | 41 |
| attempt 6TCP_NODELAY | 4.15 |
Attempts 1–5 (jemalloc decay, metrics thread, CPU pinning, ArcSwap removal, string alloc) left the tail at 41 ms. Attempt 6 was one line: set_nodelay(true).