Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,12 @@ TELEGRAM_CHAT_ID_DEFAULT=your-default-chat-id
# alerts keep going to the originating protocol's own chat.
# TELEGRAM_CHAT_ID_CURATION=your-curation-chat-id

# Small parent-vault flow alerts — deposits + withdrawals below the dust threshold,
# aggregated into one Telegram message per run. High-volume, informational only; routed
# to a dedicated group so the noise doesn't spam the protocol's main chat. A standalone
# chat served by the DEFAULT bot; falls back to the originating protocol's own chat when
# unset.
# TELEGRAM_CHAT_ID_SMALL_DEPOSITS=your-small-deposits-chat-id
# Internal Yearn maintenance chat — small parent-vault flow alerts (deposits +
# withdrawals below the dust threshold, aggregated per run) and lender-borrower
# strategy risk alerts. Internal-only: stored under the `yearn-internal` alert key,
# so they never show on the public Yearn page. A standalone chat served by the
# DEFAULT bot; falls back to the yearn chat when unset.
# TELEGRAM_CHAT_ID_YEARN_MAINTENANCE=your-yearn-maintenance-chat-id

# Protocol-specific Telegram settings (legacy per-protocol chats)
TELEGRAM_BOT_TOKEN_AAVE=your-aave-bot-token
Expand Down
4 changes: 2 additions & 2 deletions monitoring.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -466,12 +466,12 @@ protocols:
- name: "Indexer Freshness"
description: "Envio indexer lag per chain; alerts the Envio channel when a chain's newest indexed block is older than 60 minutes or the GraphQL endpoint is down"
- name: "Lender-Borrower Risk"
description: "All configured lender-borrower strategies: collateral and borrow-token prices plus LTV/liquidation risk every 30m; lender and borrow APR availability, 24h net spread, and debt coverage every 6h"
description: "All configured lender-borrower strategies: collateral and borrow-token prices plus LTV/liquidation risk every 30m; lender and borrow APR availability, 24h net spread, and debt coverage every 6h; alerts go to the internal Yearn maintenance chat"
severity: "MEDIUM"
- name: "Large Flows"
description: "Deposit/withdrawal flows >=$500k USD (Katana withdrawals >=$50k; or 10% of vault totalSupply fallback for unpriced tokens)"
- name: "Small Parent Vault Flows"
description: "Deposits and withdrawals with a raw ERC-4626 assets value below 10,000 for non-retired Yearn v3 parent vaults; aggregated into one Telegram message per run and routed to a dedicated group"
description: "Deposits and withdrawals with a raw ERC-4626 assets value below 10,000 for non-retired Yearn v3 parent vaults; aggregated into one Telegram message per run and routed to the internal Yearn maintenance chat"
- name: "Timelock Delay"
description: "Yearn TimelockController getMinDelay() < 7 days across Mainnet, Base, Arbitrum, Polygon, Optimism, Katana"
- name: "Timelock Events"
Expand Down
28 changes: 23 additions & 5 deletions protocols/yearn/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,22 @@

This folder contains monitoring scripts for Yearn vault activity, Safe multisig queues, and timelock operations.

## Alert Routing

Where each script in this folder sends its alerts. "Yearn" is the public channel (`TELEGRAM_TOPIC_ID_YEARN` or `TELEGRAM_CHAT_ID_YEARN`). The DB tag is the alert-history protocol key; `yearn-internal` and `YEARN_TIMELOCK_INTERNAL` never show on the public Yearn monitoring page.

| Script | Alerts | Telegram destination | Fallback when unset | DB tag |
|---|---|---|---|---|
| `lender_borrower.py` | LTV, spread, coverage warnings and monitor errors | Yearn maintenance (`TELEGRAM_CHAT_ID_YEARN_MAINTENANCE`) | Yearn | `yearn-internal` |
| `alert_small_parent_flows.py` | Aggregated small parent-vault flows | Yearn maintenance (`TELEGRAM_CHAT_ID_YEARN_MAINTENANCE`) | Yearn | `yearn-internal` |
| `alert_large_flows.py` | Large deposits/withdrawals | Yearn | — | `yearn` |
| `check_shadow_debt.py` | Shadow debt | Yearn | — | `yearn` |
| `check_stuck_triggers.py` | Stuck TKS triggers | Yearn | — | `yearn` |
| `check_timelock_delay.py` | Timelock min delay below 7 days | Yearn timelock topic (`TELEGRAM_TOPIC_ID_YEARN_TIMELOCK`), mirrored to Yearn timelock internal (`TELEGRAM_CHAT_ID_YEARN_TIMELOCK_INTERNAL`) | — | `yearn` (mirror: `YEARN_TIMELOCK_INTERNAL`) |
| `check_indexer_freshness.py` | Envio indexer lag / outage | Envio (`TELEGRAM_CHAT_ID_ENVIO`) | Errors, then Yearn | `yearn-internal` |

Envio errors raised inside the flow monitors follow the same Envio → errors → Yearn chain; the small-flows monitor stores them as `yearn-internal`. Unhandled crashes from any script go through `run_with_alert` to the errors channel, falling back to Yearn.

## Lender-Borrower Risk

The script `yearn/lender_borrower.py` monitors configured Morpho and Aave-compatible lender-borrower strategies. It currently covers Katana Morpho `vbWBTC/vbUSDC`, Ethereum Spark `wstETH/USDS` reached through its WETH accumulator, and Ethereum Spark `WETH/USDS`; each strategy supplies collateral, borrows a stablecoin, and lends the borrowed balance into a Yearn vault.
Expand All @@ -12,7 +28,7 @@ The script `yearn/lender_borrower.py` monitors configured Morpho and Aave-compat
2. **Net spread**: derives the current borrow APR from Morpho's adaptive IRM or Spark's variable borrow rate and subtracts it from the lender vault APR returned by Yearn's APR oracle. A medium alert fires after at least three samples when the rolling 24-hour average is below `-1%`. A zero lender APR is treated as unavailable data, alerts, and is not stored as a rate sample. Runs every six hours.
3. **Debt coverage**: compares `balanceOfLentAssets() + balanceOfBorrowToken()` with `balanceOfDebt()`. A medium alert fires when the deficit is both at least 10 basis points of debt and worth at least $100. Runs every six hours with the net-spread check.

All breach, unavailable-data, and monitor-error alerts use `MEDIUM` severity and route to the internal curation Telegram channel, falling back to the Yearn channel when curation is not configured. MEDIUM sends Telegram without invoking the HIGH/CRITICAL emergency-dispatch hook. Persistent breaches and errors are deduplicated and reminded once per 24 hours. The monitor is read-only and does not initiate deleveraging.
All breach, unavailable-data, and monitor-error alerts use `MEDIUM` severity and route to the internal Yearn maintenance Telegram chat (`TELEGRAM_CHAT_ID_YEARN_MAINTENANCE`), falling back to the Yearn channel when it is not configured. They are stored in alert history under the `yearn-internal` protocol key, so they do not appear on the public Yearn monitoring page. MEDIUM sends Telegram without invoking the HIGH/CRITICAL emergency-dispatch hook. Persistent breaches and errors are deduplicated and reminded once per 24 hours. The monitor is read-only and does not initiate deleveraging.

### Usage

Expand Down Expand Up @@ -68,9 +84,9 @@ The script `yearn/alert_small_parent_flows.py` alerts on every deposit or withdr
Deposits and withdrawals are processed with independent per-chain `(blockNumber, logIndex)` cursors stored in the monitoring database. The run stages cursor updates in memory and saves them only after the aggregate alert is delivered. A failed send leaves the flows available for the next run. A new deployment starts each stream with a two-hour lookback; that starting timestamp is persisted, so a stream that has not yet seen any event never slides its window forward and a long run gap cannot drop events.

**Routing:** qualifying flows in a run are summarized in one Telegram message
(per chain and direction, sorted chronologically) and sent to a dedicated noisy-channel
group, `TELEGRAM_CHAT_ID_SMALL_DEPOSITS`, so the volume doesn't spam the protocol's
main chat or the curation group. If that chat id is unset the aggregated message falls
(per chain and direction, sorted chronologically) and sent to the internal Yearn maintenance
chat, `TELEGRAM_CHAT_ID_YEARN_MAINTENANCE` (shared with lender-borrower alerts), so the
volume doesn't spam the protocol's main chat or the curation group. If that chat id is unset the aggregated message falls
back to the yearn group.

The aggregated message and this monitor's Envio error messages are stored in alert
Expand Down Expand Up @@ -309,6 +325,8 @@ All chains use the same contract address: `0x88ba032be87d5ef1fbe87336b7090767f36

Optimism is not covered: the Envio indexer stopped indexing it, so its timelock events are no longer available.

Separately, `yearn/check_timelock_delay.py` reads `getMinDelay()` on every chain (including Optimism) and sends a HIGH alert when it drops below 7 days. Like the event alerts, it goes to the public Yearn timelock topic (stored as `yearn`) and is mirrored to the internal chat (`TELEGRAM_CHAT_ID_YEARN_TIMELOCK_INTERNAL`, stored as `YEARN_TIMELOCK_INTERNAL`).

=======

## Indexer Freshness
Expand All @@ -331,7 +349,7 @@ Step 3 covers the inverse trap: an empty result set is not good news. If a chain

### Alerts

This monitor only ever reports Envio indexer problems, so all of its alerts go to the Envio chat (`TELEGRAM_CHAT_ID_ENVIO`) labelled `[yearn]`, alongside the other indexer failures (large flows, timelock). Every other yearn monitor's operational error still goes to the errors channel. If `TELEGRAM_CHAT_ID_ENVIO` is unset these fall back to the errors channel, and from there to the protocol's own chat.
This monitor only ever reports Envio indexer problems, so all of its alerts go to the Envio chat (`TELEGRAM_CHAT_ID_ENVIO`) labelled `[yearn]`, alongside the other indexer failures (large flows, timelock). Every other yearn monitor's operational error still goes to the errors channel. If `TELEGRAM_CHAT_ID_ENVIO` is unset these fall back to the errors channel, and from there to the protocol's own chat. They are stored in alert history under the `yearn-internal` protocol key, so they do not appear on the public Yearn monitoring page.

- **Stale or missing chains** — one message listing every lagging chain with its lag and last indexed block, plus every expected chain the indexer reported no sync state for.
- **Indexer unavailable** — the GraphQL endpoint is unset, unreachable, returned errors, or reported no chains. Sent on every run for as long as it lasts, and — unlike the other Envio messages, which are silent — with a notification, since every Envio-backed monitor is blind until it recovers.
Expand Down
4 changes: 2 additions & 2 deletions protocols/yearn/alert_small_parent_flows.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from utils.alert import Alert, AlertSeverity, send_alert
from utils.chains import EXPLORER_URLS, Chain
from utils.logger import get_logger
from utils.telegram import MAX_MESSAGE_LENGTH, SMALL_DEPOSITS_CHANNEL, resolve_channel, send_envio_error_message
from utils.telegram import MAX_MESSAGE_LENGTH, YEARN_MAINTENANCE_CHANNEL, resolve_channel, send_envio_error_message

load_dotenv()

Expand Down Expand Up @@ -125,7 +125,7 @@ def send_summary(self) -> None:
AlertSeverity.LOW,
message,
ALERT_PROTOCOL,
channel=resolve_channel(SMALL_DEPOSITS_CHANNEL, PROTOCOL),
channel=resolve_channel(YEARN_MAINTENANCE_CHANNEL, PROTOCOL),
)
)
self.truncated = truncated
Expand Down
8 changes: 7 additions & 1 deletion protocols/yearn/check_indexer_freshness.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@
logger = get_logger("yearn.check_indexer_freshness")

PROTOCOL = "yearn"
# Alert-history key: internal-only, so indexer alerts stay off the public Yearn page.
ALERT_PROTOCOL = "yearn-internal"

ENVIO_GRAPHQL_URL = os.getenv("ENVIO_GRAPHQL_URL")

Expand Down Expand Up @@ -278,7 +280,9 @@ def report_recovered(fresh: list[ChainFreshness]) -> None:
if not recovered:
return
names = ", ".join(f"{chain.name} ({format_duration(chain.lag_seconds or 0)} behind)" for chain in recovered)
send_envio_error_message(f"Envio indexer caught up: {names}", PROTOCOL, source="indexer_freshness")
send_envio_error_message(
f"Envio indexer caught up: {names}", PROTOCOL, source="indexer_freshness", alert_protocol=ALERT_PROTOCOL
)
for chain in recovered:
_set_last_alert_timestamp(chain.chain.chain_id, 0)

Expand All @@ -302,6 +306,7 @@ def main() -> None:
PROTOCOL,
disable_notification=False,
source="indexer_freshness",
alert_protocol=ALERT_PROTOCOL,
)
return

Expand All @@ -327,6 +332,7 @@ def main() -> None:
build_alert_message(stale_to_alert, missing_to_alert, max_lag_seconds),
PROTOCOL,
source="indexer_freshness",
alert_protocol=ALERT_PROTOCOL,
)
for entry in stale_to_alert:
_set_last_alert_timestamp(entry.chain.chain_id, now)
Expand Down
37 changes: 32 additions & 5 deletions protocols/yearn/check_timelock_delay.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,12 @@

logger = get_logger("yearn.check_timelock_delay")

# Delay-check violations are an internal security concern, not public topic
# noise — route them to the internal-only chat instead of YEARN_TIMELOCK.
ALERT_PROTOCOL = "YEARN_TIMELOCK_INTERNAL"
PROTOCOL = "yearn"
# Violations go to the public Yearn timelock topic (stored as `yearn`) and are
# mirrored to the internal-only chat, matching protocols/timelock/timelock_alerts.py.
# The mirror keeps its own alert-history key so the Yearn page doesn't list it twice.
PUBLIC_CHANNEL = "YEARN_TIMELOCK"
INTERNAL_PROTOCOL = "YEARN_TIMELOCK_INTERNAL"

TIMELOCK_ADDRESS = Web3.to_checksum_address("0x88ba032be87d5ef1fbe87336b7090767f367bf73")
EXPECTED_MIN_DELAY_SECONDS = 7 * 24 * 60 * 60
Expand Down Expand Up @@ -109,10 +112,34 @@ def main() -> None:
return

message = build_alert_message(violations)
send_alert(Alert(AlertSeverity.HIGH, message, ALERT_PROTOCOL))
send_violation_alerts(message)


def send_violation_alerts(message: str) -> None:
"""Send the violation alert to the public timelock topic and the internal mirror.

Both destinations are attempted even if one fails; the first failure is
re-raised afterwards so ``run_with_alert`` still reports it.

Args:
message: The Markdown alert message.
"""
alerts = (
Alert(AlertSeverity.HIGH, message, PROTOCOL, channel=PUBLIC_CHANNEL),
Alert(AlertSeverity.HIGH, message, INTERNAL_PROTOCOL),
)
error: Exception | None = None
for alert in alerts:
try:
send_alert(alert)
except Exception as exc: # noqa: BLE001 - deliver to the other destination before failing
logger.exception("Failed to send timelock delay alert to %s", alert.channel or alert.protocol)
error = error or exc
if error is not None:
raise error


if __name__ == "__main__":
from utils.runner import run_with_alert

run_with_alert(main, ALERT_PROTOCOL)
run_with_alert(main, PROTOCOL)
12 changes: 7 additions & 5 deletions protocols/yearn/lender_borrower.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@
from utils.chainlink import CHAINLINK_ABI, RoundData
from utils.chains import Chain
from utils.logger import get_logger
from utils.telegram import CURATION_CHANNEL, resolve_channel
from utils.telegram import YEARN_MAINTENANCE_CHANNEL, resolve_channel
from utils.web3_wrapper import ChainManager, Web3Client

PROTOCOL = "yearn"
# Alert-history key: internal-only, so these alerts stay off the public Yearn page.
ALERT_PROTOCOL = "yearn-internal"
logger = get_logger("yearn.lender_borrower")

WAD = 10**18
Expand Down Expand Up @@ -760,8 +762,8 @@ def run_strategy(config: StrategyConfig, *, checks: str = CHECK_ALL, dry_run: bo
Alert(
AlertSeverity.MEDIUM,
message,
PROTOCOL,
channel=resolve_channel(CURATION_CHANNEL, PROTOCOL),
ALERT_PROTOCOL,
channel=resolve_channel(YEARN_MAINTENANCE_CHANNEL, PROTOCOL),
),
plain_text=True,
)
Expand Down Expand Up @@ -795,8 +797,8 @@ def main() -> None:
AlertSeverity.MEDIUM,
f"Lender Borrower Monitor Error ({args.checks})\n"
f"{config.name}\n{error_type}: {exc}\n{config.strategy_url}",
PROTOCOL,
channel=resolve_channel(CURATION_CHANNEL, PROTOCOL),
ALERT_PROTOCOL,
channel=resolve_channel(YEARN_MAINTENANCE_CHANNEL, PROTOCOL),
),
plain_text=True,
)
Expand Down
1 change: 1 addition & 0 deletions tests/test_indexer_freshness.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,7 @@ def test_main_unavailable_alert_notifies(monkeypatch: pytest.MonkeyPatch, envio_

assert len(calls) == 1
assert calls[0]["disable_notification"] is False
assert calls[0]["alert_protocol"] == "yearn-internal"


@pytest.mark.parametrize(
Expand Down
25 changes: 25 additions & 0 deletions tests/test_lender_borrower.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import pytest

import protocols.yearn.lender_borrower as lender_borrower
from protocols.yearn.lender_borrower import (
CHECK_LTV,
CHECK_RATES_AND_COVERAGE,
Expand All @@ -29,7 +30,9 @@
prune_rate_samples,
validate_borrow_price_round,
)
from utils.alert import Alert
from utils.chainlink import RoundData
from utils.telegram import YEARN_MAINTENANCE_CHANNEL

CONFIG = STRATEGIES[0]

Expand Down Expand Up @@ -225,3 +228,25 @@ def test_monitor_errors_are_deduplicated_and_reminded_daily() -> None:
assert not _should_send_error(CONFIG, CHECK_LTV, "ValueError", now + 60)
assert _should_send_error(CONFIG, CHECK_LTV, "ValueError", now + 24 * 60 * 60)
assert _should_send_error(CONFIG, CHECK_LTV, "TimeoutError", now + 60)


@pytest.mark.parametrize(
("chat_id", "expected_channel"), [("maintenance_chat_id", YEARN_MAINTENANCE_CHANNEL), ("", "yearn")]
)
def test_alerts_route_to_internal_maintenance_chat(
monkeypatch: pytest.MonkeyPatch, chat_id: str, expected_channel: str
) -> None:
monkeypatch.setenv("TELEGRAM_CHAT_ID_YEARN_MAINTENANCE", chat_id)
snapshot = _snapshot(current_ltv_wad=70 * WAD // 100)
sent: list[Alert] = []
monkeypatch.setattr(lender_borrower, "_read_snapshot", lambda config, include_rates: snapshot)
monkeypatch.setattr(lender_borrower, "_clear_error_state", lambda *args: None)
monkeypatch.setattr(lender_borrower, "_should_send_alert", lambda *args: True)
monkeypatch.setattr(lender_borrower, "_record_alert_sent", lambda *args: None)
monkeypatch.setattr(lender_borrower, "send_alert", lambda alert, **kwargs: sent.append(alert))

lender_borrower.run_strategy(CONFIG, checks=CHECK_LTV)

assert len(sent) == 1
assert sent[0].channel == expected_channel
assert sent[0].protocol == "yearn-internal"
Loading
Loading