Skip to content

Bug: single broken symlink in skills dir permanently breaks pulls, silently disables backup pruning, and piles up thousands of backup dirs #87

Closed

Description

@Pljas

Summary

SkillHub.pull_skills() (mirror mode) runs shutil.copytree(skills_dir, backup_dir) before every pull. copytree dereferences symlinks by default (symlinks=False), so a single broken symlink anywhere inside the skills directory raises FileNotFoundError. The exception is caught at "backup before pull failed", which returns early — and on that early-return path:

  1. The pull never happens — shared/remote skills silently stop updating.
  2. _prune_backups(keep=3) (called only on the success path) is never reached — with the default 30s reload poll, a new partial backup dir is created every poll cycle and never deleted.

We observed 6,793 backup dirs (~321 GB) accumulate on a production host within ~17 hours, filling a 400 GB disk to 100%.

Environment

  • SkillClaw 0.4.0 (commit 3938f75), deployed via Docker (skillclaw start + skillclaw-evolve-server --interval 300)
  • sharing.backend: local, default skill_reload_mode: poll (30s)
  • Skills dir contained a leftover _staging/ tree from an interrupted skill install with dangling symlinks (.agents/skills/tg-rich-digest -> ../../skills/tg-rich-digest, where the target had been deleted by a previous mirror pull)

Logs

WARNING | skillclaw.skill_hub | [SkillHub] backup before pull failed:
  [('/root/.hermes/skills/_staging/telegram-skills/.agents/skills/tg-rich-digest',
    '/root/.hermes/.skillclaw_backups/skills_.../_staging/.../tg-rich-digest',
    "[Errno 2] No such file or directory: ...")]

(repeats every 30 seconds; no pull is ever performed)

Suggested fixes

  1. shutil.copytree(..., symlinks=True) in the backup step — dangling symlinks are copied as-is instead of being resolved. Also matches user intent: the backup should mirror the tree, not materialize link targets.
  2. Run _prune_backups() on all exit paths (e.g. finally), including backup-failure and mirror-rollback early returns — pruning is exactly what's needed most when pulls fail.
  3. Optionally: make retention configurable (e.g. sharing.skill_backup_keep) and skip _staging/staging-prefixed dirs when backing up.
  4. Consider warning loudly (or surfacing in skillclaw doctor) when pulls fail repeatedly — in our case skills had silently not updated for days.

Opening a PR for (1)+(2).

Activity

  1. Upper9527 commented on Oct 21, 2026

    @Upper9527
    Collaborator

    Fixed on main via #88, with a follow-up in #96 covering one more retention bypass: failure to create the staging directory. Thanks for the detailed production report and the original patch.

    Backups and rollback now preserve dangling symlinks, partial failed backups are removed, and retention runs on backup failure, mirror failure, and staging failure. A staging-creation failure leaves the existing local library untouched. Five regression cases cover these paths, including repeated failures and recovery-backup preservation; the follow-up's full local suite passed (151 passed, 1 credential-dependent test skipped).

    Please update to current main to obtain these changes; this comment does not imply a new PyPI release. Existing old backups are pruned on a subsequent applicable mirror-pull attempt. If filesystem permissions or I/O errors prevent deletion, retention cannot force that deletion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions