Skip to content

Propagate PipelineRequestConfigParameters through pull-mode (NATS) tasks #1275

Description

@mihow

Surfaced during PR #1272 occurrence-tracking E2E testing against ADC PR #77.

Problem

PipelineRequestConfigParameters (the config dict in PipelineRequest) only reaches sync HTTP /process workers. The NATS pull-mode task schema does not carry it.

ami/ml/schemas.py::PipelineProcessingTask:

class PipelineProcessingTask(pydantic.BaseModel):
    id: str
    image_id: str
    image_url: str
    reply_subject: str | None = None
    # TODO: Do we need these?
    # detections: list[DetectionRequest] | None = None
    # config: PipelineRequestConfigParameters | dict | None = None

config is commented out. ami/ml/orchestration/jobs.py::queue_all_images_for_job publishes only the per-image fields. Pull-mode workers cannot see include_features, include_logits, or any future per-project / per-pipeline flag.

Why now

PR #1272 needs include_features=true to populate Classification.features_2048 for tracking.

  • Sync HTTP: works via Pipeline.default_config = {"include_features": True}.
  • Pull-mode (ADC): worker must start with AMI_INCLUDE_FEATURES=true in its own env. That makes features a worker-level on/off, not per-project, and silently ignores antenna's ProjectPipelineConfig.

Scope

  • Add config: PipelineRequestConfigParameters | dict | None = None to PipelineProcessingTask.
  • Populate task.config from pipeline.get_config(project_id=...) in ami/ml/orchestration/jobs.py.
  • Update ADC worker (and any other pull-mode worker) to read config.include_features from the incoming task; keep env var as fallback.
  • Document new contract in processing_services/README.md + schema docstring.
  • Test: ProjectPipelineConfig.config={'include_features': true} reaches the ADC worker via NATS task.

Risk / migration

Additive + optional. Existing pull-mode workers that ignore config keep working; defaults stay safe.

Linked

Activity

  1. added
    PSv2Async & distributed ML backend (PSv2): job state, NATS dispatch, result handling. Umbrella #515.
    on Jun 16, 2026
  2. added
    Pipeline APIUpdates to the requests & responses to/from processing service workers for ML pipelines
    on Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PSv2Async & distributed ML backend (PSv2): job state, NATS dispatch, result handling. Umbrella #515.Pipeline APIUpdates to the requests & responses to/from processing service workers for ML pipelines

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions