Skip to content

fix(microservices): log connection error backtraces once per message - #3999

Open
ryanmelt wants to merge 2 commits into
mainfrom
fix/interface-connection-error-logging
Open

ryanmelt wants to merge 2 commits into
mainfrom
fix/interface-connection-error-logging

Conversation

@ryanmelt

@ryanmelt ryanmelt commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Ruby + Python: an unexpected error from connect or read logged a full backtrace on every reconnect attempt, so a persistent fault flooded the log. The connection_failed_messages / connection_lost_messages lists only de-duplicated themselves. The backtrace is now logged only the first time each distinct message is seen; repeats still log the one-line "Connection ... failed due to" / "Connection Lost:" message.
  • Resetting: both lists are cleared on Connection Success and when the user requests a connect (attempting, including connect after a disconnect), so the next failure logs its backtrace again. Automatic reconnect attempts do not clear them.
  • Python handle_connection_failed: the expected-error check used connect_error is Exception, which is always false, so every failure (including connection refused) logged a traceback. It now uses isinstance.
  • Python handle_connection_lost: had no expected-error filter. It now has one.
  • Python error mapping: Python's IOError is an alias for OSError, so Ruby's lists are mapped as ConnectionError (refused/reset/aborted), TimeoutError, and OSError with the same errnos Ruby lists (ENOTSOCK/EHOSTUNREACH for failed, EBADF/ENOTSOCK for lost). A plain OSError such as PermissionError still logs a backtrace.

Fixes the IF-CON-006 deviations in architecture/specs/interface-microservice.yaml.

Test plan

  • bundle exec rspec spec/microservices/interface_microservice_spec.rb (29 examples, 0 failures). 3 of the 5 new examples fail on the old code.
  • uv run pytest test/microservices/test_interface_microservice.py (30 passed). All 5 new tests fail on the old code.
  • ruff check / ruff format --check on the changed source

🤖 Generated with Claude Code

Unexpected connect/read errors logged a full backtrace on every reconnect
attempt. Ruby and Python now log it only the first time each distinct
message is seen. Python also stops logging backtraces for expected network
errors: the `is Exception` check was always false and lost connections had
no filter at all.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.15%. Comparing base (0d8985e) to head (4ca05dd).
⚠️ Report is 28 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3999      +/-   ##
==========================================
+ Coverage   80.06%   80.15%   +0.09%     
==========================================
  Files         901      901              
  Lines       68370    68455      +85     
  Branches     2699     2661      -38     
==========================================
+ Hits        54738    54869     +131     
+ Misses      12976    12918      -58     
- Partials      656      668      +12     
Flag Coverage Δ
frontend 67.08% <ø> (+0.51%) ⬆️
python 80.13% <ø> (-0.01%) ⬇️
ruby-api 82.61% <ø> (ø)
ruby-backend 85.67% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pass # Do not write an exception file for these extremely common cases
# Only log the backtrace the first time we see each distinct error
elif str(error) not in self.connection_lost_messages:
self.logger.error(f"{self.interface.name}: {''.join(traceback.format_exception(error))}")

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All ty errors in interface_microservice.py handled in another PR.

The connection failed/lost message lists were never cleared, so a backtrace
was logged only once for the life of the microservice. They are now cleared
on a successful connect and when the user requests a connect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@ryanmelt ryanmelt added the review-and-merge-ready Ready for a manual review and merge label Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-and-merge-ready Ready for a manual review and merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants