fix(queue): recover a Redis connection whose client phpredis marked failed

[loks0n]

What

Connection\Redis runs every command through a call() wrapper: on a phpredis transport error ("went away", "Connection lost", "Connection closed", "Connection refused", "read error on connection") it drops the cached \Redis client and runs the command once more on a fresh connection. Any other RedisException (WRONGTYPE, OOM, ...) still surfaces as before, and a second transport failure propagates.

New e2e test RedisConnectionRecoveryTest, registered in the e2e suite against the compose redis on 16379.

Why

phpredis retries a dropped socket by itself, but once the peer stays away past its retry budget it sets the client to REDIS_SOCK_STATUS_FAILED for good, after which every command throws Redis server <host>:<port> went away. getRedis() cached that object and returned it forever.

On Appwrite Cloud this hit worker-executions when a Dragonfly primary moved during a rollout: the consume side recovered because Broker\Redis::receive() already closes and reconnects, but the command connection behind Locking never did, so every execution job failed at ack with the trace below for 30 minutes until the pod was restarted.

RedisException: Redis server queue-dragonfly:6379 went away
  src/Queue/Connection/Redis.php:126  del
  src/Queue/Connection/Locking.php:119  remove

TDD

The test reproduces the terminal phpredis state deterministically without an outage: a client with OPT_MAX_RETRIES 0 whose own socket the server closes via CLIENT KILL ID, then a command.

• Before the change: RedisException: Connection lost at Redis.php:126, the production line.
• After: the command runs on a fresh connection and later commands keep working. A second test checks that an unreachable server still throws.

composer test (113 unit tests) and SwooleTest against a live worker pass. bin/monorepo check queue --fix applied; the remaining PHPStan findings in Broker/Nats.php and Broker/Pool.php are from registry-resolved siblings in my local vendor and are not in this diff (main CI is green).

Notes

• RedisCluster caches its client the same way and would benefit from the same wrapper; left out to keep this to the reported path.
• A retried write can in principle double-apply if the server executed it before the read failed. phpredis checks EOF before writing, so the failed-state case retries an unsent command; the "read error" case carries the same risk any client has.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previous replay and test-effectiveness findings are resolved, and no new actionable failure was established.

Summary

• Routes Redis operations through centralized transport-error handling.
• Prevents automatic replay of pushes, pops, and counters when execution is uncertain.
• Adds a TCP proxy that can simulate an outage or discard a command reply.
• Registers the Redis recovery scenario in the queue E2E suite.

_{Reviews (3) · Last reviewed commit: "test(queue): lose the reply, not the ser..."}

[github-actions]

Benchmark results

queue — workload shapes across both concurrency axes (4 cores, 600 messages, median of 3)

workload	shape	redis msg/s	nats msg/s
io	1p x 1c	38	38
io	1p x 4c	152	154
io	4p x 1c	152	153
cpu	1p x 1c	203	208
cpu	1p x 4c	210	213
cpu	4p x 1c	486	506
mixed	1p x 1c	66	67
mixed	1p x 4c	181	188
mixed	4p x 1c	260	263

Np x Mc = N consumer processes x M handler coroutines. io yields and should follow
coroutines; cpu does not and should follow processes; mixed is in between.

Shared CI runners — treat absolute numbers as rough, compare modes within a run. Commit 8c4d4b2.