Bug Report: Lab AI-300-MLOpsM7 — Pre-built MLflow Model Fails to Deploy

Title

Lab AI-300-MLOpsM7 — pre-built MLflow model in model/ fails to deploy due to aged/incompatible dependencies

Environment

• Lab: AI-300-MLOpsM7 "Deploy-monitor"
• Workspace region: westus
• Deployment path: src/deploy_to_online_endpoint.py → managed online endpoint
• Model location: committed at model/ (deployed via path="./model", type=MLFLOW_MODEL)

Summary

The MLflow model committed at model/ was saved on 2023-02-15 with:

• mlflow_version: 1.30.0
• sklearn_version: 0.24.1
• python_version: 3.7.15
• serialization_format: cloudpickle

Its model/conda.yaml no longer produces a working deployment because several dependencies have since aged out. Using the default lab flow, the deployment fails with no clear learner-side error message — only generic ImageBuildFailure and 502 liveness-probe failures that require digging into storage build logs and container logs to diagnose.

Failures Observed (in order, each one blocking deployment)

1. ImageBuildFailure — Python version no longer available

model/conda.yaml pins python=3.7.15, which conda-forge no longer provides (Python 3.7 is end-of-life). The environment image build fails at conda env create:

PackagesNotFoundError: The following packages are not available from current channels:
  - python=3.7.15

2. Inference server crash — incompatible opentelemetry stack

After pinning python=3.9, the image builds and the container starts, but the inference server crashes on boot:

ImportError: cannot import name 'ReadableLogRecord' from 'opentelemetry.sdk._logs'

Cause: conda.yaml leaves mlflow unpinned, so the build installs mlflow 3.1.4, which pulls an opentelemetry stack (opentelemetry-sdk 1.33.0) incompatible with the inference server's azure-monitor-opentelemetry-exporter. The worker fails to boot, producing repeated 502 liveness/readiness-probe failures and an unrecoverable deployment.

3. numpy / scikit-learn incompatibility

conda.yaml also leaves numpy/scipy unpinned, installing numpy 2.0.2, which is incompatible with the pinned scikit-learn 0.24.1 (built against numpy ~1.19–1.21). This would crash model loading even after the opentelemetry issue is resolved.

Additional Notes

• A generic endpoint name (diabetes-endpoint) in the deploy step also causes region-level name collisions in shared lab regions, since online endpoint names must be unique per Azure region. Using a unique default (or a learner-specific suffix) would avoid this.
• On any failed deployment, the blue deployment enters an unrecoverable state ("Delete and re-create"); the deploy script does not handle deleting a failed deployment before recreating, requiring manual cleanup between attempts.