Skip to content

Add shared pipeline logging/state-marker bash library for ASPEN/CARLISLE #230

Description

@kopardev

Summary

ASPEN's aspen and CARLISLE's carlisle wrapper scripts each independently
define a nearly-identical set of bash functions for structured logging and
pipeline state tracking (log_info/log_step/log_ok/log_warn/log_error/
log_next/log_divider, json_escape, print_versions,
write_pipeline_state_marker, run_jobby_best_effort, and (after
CCBR/ASPEN#119 lands) a _progress_monitor background-loop function). This is
duplicated code that will drift over time.

Both wrappers already depend on ccbr_tools at runtime (via
module load ccbrpipeliner) for jobby. ccbr_tools's pyproject.toml
already has a [tool.setuptools] script-files mechanism that installs
arbitrary .sh/.bash/.R/.py scripts directly onto PATH on
pip install — this is the natural place to host a shared shell library for
exactly this kind of cross-pipeline utility code.

Proposed work

  1. Add a new shared shell library, e.g. scripts/ccbr_pipeline_logging.sh,
    containing generic/parameterized versions of the functions above.
    • write_pipeline_state_marker and run_jobby_best_effort must be
      parameterized (env vars or args) for PIPELINE_NAME, PIPELINE_VERSION,
      and the snakemake log path — ASPEN keeps snakemake.log at the workdir
      root, CARLISLE keeps it under logs/snakemake.log. Do not hardcode
      either convention.
  2. Add the new file to [tool.setuptools] script-files in pyproject.toml so
    it's installed onto PATH when ccbr_tools/ccbrpipeliner is loaded.
  3. Also include a generic rescript/syncscripts helper (re-sync
    workflow/scripts into an existing workdir's scripts/ folder) — this only
    needs $PIPELINE_HOME/$WORKDIR, no pipeline-specific parameterization.
  4. Document a minimum required ccbr_tools/ccbrpipeliner module version
    contract for consumers of this library.

Related

⚡ Generated using AI ⚡

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions