Skip to content

fix(gateways): survive Kafka accept errors and a late Iggy at startup - #4645

Merged
hubcio merged 2 commits into
masterfrom
kafka-process-exits
Oct 28, 2026
Merged

hubcio merged 2 commits into
masterfrom
kafka-process-exits

Conversation

@krishvishal

Copy link
Copy Markdown
Member

Rationale

Three short-lived conditions stopped the Kafka gateway or locked Iggy clients out.

The accept loop retried EINTR, ECONNABORTED, EWOULDBLOCK, EMFILE and ENFILE only. Any other accept() error ended the listener, and the gateway process exited with it. Linux accept(2) also returns the network errors of one pending connection, such as EPROTO, ENETDOWN or EHOSTUNREACH, and its manual says to retry them. ENOBUFS and ENOMEM, a short lack of kernel memory, also stopped the gateway.

At startup, the gateway connected to Iggy once. One connect gives up within 15 s, so a gateway that started before its Iggy node exited. As a sidecar, the gateway starts at the same time as its Iggy node.

Each SASL login opens a short-lived Iggy client to check the password. The gateway shut that client down without a logout. Iggy kept each such session in its clients table until its lease ran out, 36 s by default. A full table refuses every new login, so a burst of Kafka logins locked the other clients of the cluster out.

An accept error or an Iggy that was not up yet stopped the gateway, and SASL checks kept sessions that filled the clients table.
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Oct 25, 2026
@codecov

codecov Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.75%. Comparing base (d6e42a7) to head (f2264c9).

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4645      +/-   ##
============================================
- Coverage     90.85%   89.75%   -1.11%     
+ Complexity     3104     2989     -115     
============================================
  Files          1479     1477       -2     
  Lines        361309   368047    +6738     
  Branches     305179   312629    +7450     
============================================
+ Hits         328277   330342    +2065     
- Misses        26124    30771    +4647     
- Partials       6908     6934      +26     
Components Coverage Δ
Rust Core 90.78% <ø> (-1.41%) ⬇️
Java SDK 77.80% <ø> (-0.34%) ⬇️
C# SDK 83.74% <ø> (+0.14%) ⬆️
Python SDK 85.69% <ø> (+0.01%) ⬆️
PHP SDK 90.67% <ø> (-0.05%) ⬇️
Node SDK 96.44% <ø> (-0.01%) ⬇️
Go SDK 69.97% <ø> (+0.66%) ⬆️
C++ SDK 85.51% <ø> (-0.04%) ⬇️
see 101 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio
hubcio merged commit 6400415 into master Oct 28, 2026
76 checks passed
@hubcio
hubcio deleted the kafka-process-exits branch October 28, 2026 17:21
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants