chore: dev to main merge

[Saswato-Microsoft]

Purpose

This pull request introduces comprehensive improvements to Python package management across development, CI, and deployment environments, with a focus on ensuring reliable pip installs on Microsoft-managed devices. It standardizes the use of the Microsoft Package Feed Proxy as the default pip index, adds robust configuration and documentation, and enhances deployment scripts for better reliability and maintainability.

Python Package Feed Proxy Integration and Configuration:

• Set the Microsoft Package Feed Proxy (https://packagefeedproxy.microsoft.io/pypi/simple/) as the default pip index in the dev container via the PIP_INDEX_URL environment variable in devcontainer.json, with clear override instructions for non-Microsoft environments. (.devcontainer/devcontainer.json, .devcontainer/README.md) [1] [2]
• Updated the dev container post-create script to consistently use the PIP_INDEX_URL for all pip installations, ensuring compatibility with Microsoft-managed networks and allowing overrides. (.devcontainer/post-create.sh) [1] [2] [3] [4] [5]
• Configured CI workflows to use the package feed proxy by default, with the ability to override via environment variables, ensuring pip installs work reliably in GitHub Actions. (.github/workflows/azure-dev.yml, .github/workflows/template-validation.yml) [1] [2] [3] [4] [5] [6] [7]
• Enhanced the PowerShell deployment utility (Run-PythonScript.ps1) to accept and validate a PipIndexUrl parameter, defaulting to the package feed proxy and guarding against malformed URLs. (infra/scripts/utils/Run-PythonScript.ps1) [1] [2] [3] [4] [5]

Fabric Solution Installer Improvements:

• Improved the ontology deployment logic to reliably fetch the ontology ID after creation, including fallback lookups by name, making the deployment process more robust. (infra/fabric/deploy/fabric_solution_installer.ipynb)
• Added a function to check and refresh the readiness of deployed graph models, with retries and user guidance for troubleshooting, increasing deployment reliability. (infra/fabric/deploy/fabric_solution_installer.ipynb)
• Implemented automated publishing of the deployed Data Agent, ensuring it is live and usable after deployment. (infra/fabric/deploy/fabric_solution_installer.ipynb)

Other Maintenance:

• Pinned GitHub Actions to specific commit SHAs for improved security and reproducibility in CI workflows. (.github/workflows/azure-dev.yml, .github/workflows/template-validation.yml) [1] [2] [3] [4] [5]

These changes collectively ensure that Python dependency management is robust, secure, and consistent across all environments, especially for users and CI runners on Microsoft-managed networks.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[Copilot]

Pull request overview

This PR standardizes configurable pip package-feed usage across development, CI, and deployment tooling, while improving Fabric deployment automation.

Changes:

• Adds Microsoft Package Feed Proxy defaults and override support.
• Pins GitHub Actions to commit SHAs.
• Enhances ontology resolution, graph readiness checks, and Data Agent publishing.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Summary / final review notes
src/fabric/datagen/requirements.txt	Adds package-feed configuration for data-generation dependencies.
requirements.txt	Adds the default package-feed configuration.
infra/scripts/utils/Run-PythonScript.ps1	Adds pip-index selection and validation. Critical findings: redact URLs in fallback and deployment logs (2 votes each). Moderate finding: anchor or replace URL validation so malformed values cannot pass (3 votes).
infra/fabric/deploy/fabric_solution_installer.ipynb	Improves ontology and Data Agent deployment. Moderate finding: missing expected agents must fail or be retried rather than silently skipped (2 votes). Nit: update deployment documentation for readiness, retry, publishing, and troubleshooting behavior (3 votes).
.github/workflows/template-validation.yml	Configures the pip proxy and pins workflow actions.
.github/workflows/azure-dev.yml	Configures pip proxy defaults and pins workflow actions.
.devcontainer/README.md	Documents package-feed configuration and overrides.
.devcontainer/post-create.sh	Routes installs through the configured index. Critical finding: do not log raw index URLs because credentials may be exposed (2 votes).
.devcontainer/devcontainer.json	Sets the default pip index environment variable.

Suppressed comments (5)

infra/fabric/deploy/fabric_solution_installer.ipynb:523

• Despite the docstring saying to retry only when the background refresh failed, this unconditionally sleeps and starts a new RefreshGraph job. On every rerun, including when the graph is already ready, this forces another refresh and can race Fabric's initial refresh; inspect/poll the graph model's actual state first and submit a refresh only when needed.

    "    print(f\"⏳ Waiting {initial_wait_seconds}s for '{ontology_name}' graph model to finish its initial load...\")\n",
    "    time.sleep(initial_wait_seconds)\n",
    "\n",
    "    for attempt in range(1, attempts + 1):\n",
    "        response = client.post(f\"v1/workspaces/{workspace_id}/graphModels/{match['id']}/jobs/instances?jobType=RefreshGraph\")\n",

infra/fabric/deploy/fabric_solution_installer.ipynb:595

• Regardless of the HTTP result, this branch prints a success message. A 4xx/5xx response—or a 202 asynchronous publish that has not completed—is therefore reported as a published agent and the notebook can finish in a broken state. Validate the response and poll any asynchronous operation before reporting success.

    "    response = client.post(\n",
    "        f\"v1/workspaces/{workspace_id}/dataAgents/{agent['id']}/staging/publish\",\n",
    "        json={\"publishedDescription\": \"Published by the Microsoft IQ solution installer\"},\n",
    "    )\n",
    "    print(f\"   ✅ Data Agent '{agent['displayName']}' published successfully\")"

infra/fabric/deploy/fabric_solution_installer.ipynb:512

• The graph-model list is queried only once, before the six-minute wait. If Fabric has not made the graph model visible yet after ontology creation, this returns immediately and the readiness/refresh logic is skipped entirely, after which the Data Agent is published anyway. Retry the list until the model appears or fail the deployment rather than bypassing the check.

    "    graph_models_response = client.get(f\"v1/workspaces/{workspace_id}/graphModels\")\n",
    "    if graph_models_response.status_code != 200:\n",
    "        print(f\"  ⚠️ Failed to list graph models for '{ontology_name}': HTTP {graph_models_response.status_code} {graph_models_response.text}\")\n",
    "        return\n",
    "    graph_models = graph_models_response.json().get(\"value\", [])\n",

infra/fabric/deploy/fabric_solution_installer.ipynb:527

• If the RefreshGraph API returns 202 without a Location header, job_url remains None and the next iteration calls client.get(None), aborting the notebook. Treat a missing monitor URL as a failed attempt (and validate polling responses) instead of invoking the client with None.

    "            job_url = response.headers.get(\"Location\")\n",

infra/fabric/deploy/fabric_solution_installer.ipynb:533

• Fabric jobs can report Queued or Running while active and Succeeded when complete (the existing infra/scripts/fabric/fabric_api.py polling logic handles these states), but this loop only polls NotStarted/InProgress and the later check accepts only Completed. A normal successful refresh can therefore be reported as unsuccessful and trigger another POST, potentially starting duplicate refresh jobs before the cell ends with a warning. Include all active states and accept Succeeded as terminal success.

    "            while job_status in (\"NotStarted\", \"InProgress\") and elapsed < 300:\n",
    "                time.sleep(wait_seconds)\n",
    "                elapsed += wait_seconds\n",
    "                job_status = client.get(job_url).json().get(\"status\")\n",

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.