Skip to content

Backfill legacy jobs_job.logs into JobLog and drop the column #1278

Description

@mihow

Background

PR #1259 introduces the JobLog child table for per-job log persistence. Marks the legacy jobs_job.logs JSONB column as deprecated and leaves it in place as a read-only fallback for jobs created before the table existed.

Job.logs should not be removed yet — older jobs still surface their logs in the UI through the legacy fallback in serialize_job_logs. Removing the column outright would erase pre-migration job logs from the UI.

Approach

Two-step removal once #1259 has soaked:

  1. Backfill existing jobs_job.logs.{stdout, stderr} entries into JobLog rows. One-off Django management command (or data migration), batched, idempotent. Each legacy stdout line becomes one JobLog row with level parsed from the [ts] LEVEL ... prefix where present, raw line otherwise. created_at parsed from the line's prefix when available, falling back to the Job.created_at of the parent.
  2. Drop the column. Schema migration removing Job.logs SchemaField + the JobLogs Pydantic class. After this lands, serialize_job_logs collapses to the JobLog-only path and _legacy_logs_shape goes away.

Acceptance

  • Management command (or data migration) that backfills legacy JSON entries into JobLog rows. Idempotent — running twice does not duplicate rows.
  • Tested against a copy of the demo DB with a representative spread of legacy job rows.
  • Schema migration dropping Job.logs (separate, ships after the backfill is verified in production).
  • serialize_job_logs and friends in ami/jobs/models.py simplified — drop the legacy fallback branch.
  • PR body links Refactor job logging to use separate table #1259 and the soak summary.

Blocked by

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions