diff --git a/aigw/integrations/llms/azure-foundry.mdx b/aigw/integrations/llms/azure-foundry.mdx index 9ffecfa0..3dc3f857 100644 --- a/aigw/integrations/llms/azure-foundry.mdx +++ b/aigw/integrations/llms/azure-foundry.mdx @@ -604,6 +604,254 @@ The native `/v1/ocr` endpoint requires gateway version **2.18.0** or higher for --- +## Decisions (Microsoft-Decision-1) + +`Microsoft-Decision-1` is a decision model available in Microsoft Foundry. It does not generate text. It reads a piece of text or JSON, answers typed questions about it, and returns probabilities that your application can threshold or rank. Use it for classification, routing, ranking, grading, and yes/no checks. + +The AI Gateway routes these calls through `/v1/decisions`, a separate endpoint from chat completions. You still get the gateway's full feature set: authentication, routing, retries, circuit breaking, caching, logging, and cost tracking. See the [Decisions API](/aigw/product/ai-gateway/decisions-api) for the full request and response reference. + + + `/v1/decisions` does not support streaming. Only before-request [guardrails](/aigw/product/guardrails/decisions-guardrails) apply, scanning the content to judge only (`state`). + + + + This section covers how to call Microsoft-Decision-1 through the AI Gateway. For model details, deployment steps in Azure, regional availability, and responsible-use guidance, see Microsoft's official guide: [Deploy and use Microsoft-Decision-1 in Microsoft Foundry](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision). + + +### Model at a glance + +These values come from [Microsoft's model overview](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision#microsoft-decision-1-at-a-glance). + +| Model name | Model version | Deployment type | API type | +|:---|:---|:---|:---| +| `Microsoft-Decision-1` | 1 | `DataZoneStandard` in selected regions, `GlobalStandard` | Decision | + +### Request model vs response model + +This is the most common source of errors with this model, so read it before you send a request. + +| Field | Value | Example | +|:---|:---|:---| +| `model` in your **request** | The **deployment name** you chose in Azure AI Foundry | `@azure-foundry/decision-1` | +| `model` in the **response** | The underlying model | `microsoft-decision-1` | + + + Do not hardcode `microsoft-decision-1` as the request `model`. Azure looks up a **deployment** with that exact name. If your deployment has a different name, the request fails with `The API deployment microsoft-decision-1 does not exist`. Use the deployment name you see in Foundry. Microsoft describes this in the [Call the decision API](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision) section of its guide. + + +For example, if you deploy the model with the deployment name `decision-1`, send `"model": "@azure-foundry/decision-1"`. If you name the deployment `pi-decision-1`, send `"model": "@azure-foundry/pi-decision-1"`. In both cases the response reports `"model": "microsoft-decision-1"`. + +The AI Gateway logs and prices the call using the response `model`, so cost tracking works the same no matter what you name your deployment. + +### Setup + +1. Deploy `Microsoft-Decision-1` in your Foundry project by following [Microsoft's deployment steps](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision). From the deployment details page, copy the **resource endpoint**, the **deployment name**, and the **API key**. +2. Create your Azure AI Foundry provider by following [Creating Your Azure AI Foundry Provider](#creating-your-azure-ai-foundry-provider) above. Use the URL as shown in Foundry for your resource, for example `https://your-resource-name.services.ai.azure.com/models`. +3. If you manage models in **Model Provisioning**, add your deployment name as the model slug (for example `decision-1`), exactly as it appears in Foundry. See [Adding Multiple Models](#adding-multiple-models-to-your-azure-ai-foundry-provider). + + + For decisions requests, The AI Gateway sends the call to the resource root at `/providers/microsoft/v1/systemone`. Any path in the saved Foundry URL, including a `/deployments/` suffix, is dropped. No `api-version` is sent. The deployment is selected by the `model` field in the request body. + + +Decisions requests work with the same authentication methods as the rest of this integration: API key, Azure Entra ID, and Azure Managed Identity. With an API key, The AI Gateway sends it in the `api-key` header, which is what the Microsoft decision endpoint expects. + +### Send a decisions request + +Microsoft-Decision-1 uses the Jev request shape: `state` for the content to judge, and a `questions` map. Set `model` to `@provider-slug/deployment-name`. The provider holds the Azure credentials, so no provider headers are needed. + +```bash cURL +curl --request POST \ + --url https://aigw.portkey.ai/v1/decisions \ + --header 'Content-Type: application/json' \ + --header "Authorization: Bearer $PORTKEY_API_KEY" \ + --data '{ + "model": "@azure-foundry/decision-1", + "state": { + "current_ask": "hello there, how is your day going?" + }, + "questions": { + "greeting": { + "type": "choice", + "instructions": "Is this a greeting?", + "criteria": { + "yes": "a greeting", + "no": "anything else" + } + } + } +}' +``` + +Response (values are illustrative): + +```json +{ + "model": "microsoft-decision-1", + "answers": { + "greeting": { + "type": "choice", + "choice": "yes", + "probabilities": { "no": 0.0067, "yes": 0.9933 }, + "confidence": 0.9866 + } + }, + "usage": { "input_tokens": 46, "output_tokens": 1 }, + "provider": "azure-ai" +} +``` + +The OpenAI request shape (`input` and a `questions` array) also works. See [Request Shapes](/aigw/product/ai-gateway/decisions-api#request-shapes). + +### Question types + +Each entry in `questions` sets a `type`, which determines the shape of its answer. The `state` can be text or JSON. + +| Type | Question it answers | Answer shape | +|:---|:---|:---| +| `noul` | Is this true? | `{ noul: <0 to 1 probability> }` | +| `choice` | Which option is it? | `{ choice, probabilities, confidence }` | +| `score` | How much? | `{ score, legend, probabilities, confidence }` | + +The response `answers` object is keyed by the question names in your request. + +#### `noul`: gates and filters + +Use for yes/no conditions, such as whether a reply promises a refund or a document contains a prompt injection. `criteria` is optional. Define the true and false conditions to make borderline questions more precise. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": "I refunded your $200. You don't need to contact billing.", + "questions": { + "promises_refund": { + "type": "noul", + "instructions": "Does this reply promise the customer a refund?", + "criteria": { + "true": "Commits to returning money", + "false": "Makes no commitment about money" + } + } + } +} +``` + +#### `choice`: classification and routing + +Use to pick exactly one label from a fixed set, such as routing tickets or classifying intent. Describe each option clearly. The response includes a probability for every option, so a low top probability is a good signal to escalate to a person or another model. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": "The API returns 500 on every call.", + "questions": { + "team": { + "type": "choice", + "instructions": "Which team should handle this ticket?", + "criteria": { + "billing": "Charges, invoices, and refunds", + "engineering": "Bugs, errors, and outages", + "support": "How-to and account questions" + } + } + } +} +``` + +#### `score`: ranking and grading + +Use to place an item on an ordered scale, such as incident severity. List the levels from lowest to highest. The returned score is the probability-weighted average of the level indexes, so a four-level scale runs from `0` through `3` and the result can fall between levels. Prefer scores for relative ordering and thresholds, not as absolute ratings. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": "Checkout is down for all EU customers since 09:00.", + "questions": { + "severity": { + "type": "score", + "instructions": "How severe is this incident?", + "criteria": ["Cosmetic", "Minor", "Major", "Critical"] + } + } +} +``` + +#### Combine questions in one request + +Questions that share the same `state` can go in one request. This reduces request count and keeps the context consistent. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": { + "message": "Checkout is down again. This is my third request.", + "customer_contact_count": 3 + }, + "questions": { + "team": { + "type": "choice", + "instructions": "Which team should handle this request?", + "criteria": { + "billing": "Charges, invoices, and refunds", + "engineering": "Bugs, errors, and outages", + "support": "How-to and account questions" + } + }, + "severity": { + "type": "score", + "instructions": "How severe is this issue?", + "criteria": ["Cosmetic", "Minor", "Major", "Critical"] + }, + "repeat_contact": { + "type": "noul", + "instructions": "Has the customer contacted support before?" + } + } +} +``` + +### Pricing and usage + +Usage is reported in the response as `usage.input_tokens` and `usage.output_tokens`. The pricing entry for `microsoft-decision-1` charges for input tokens only. Output tokens, which hold the typed judgment itself, are priced at zero. + +### Decisions troubleshooting + +| Status | Likely cause | What to do | +|:---|:---|:---| +| `400` | Invalid question type or shape | Check `state`, `questions`, `type`, and `criteria` | +| `401` | Missing, invalid, or expired credential | Check the API key or Entra credentials on the provider | +| `403` | The identity has no access to the deployment | Check role assignments and deployment access in Azure | +| `404` | Wrong endpoint, or the deployment name does not exist | Confirm the `model` value matches your deployment name exactly. See [Request model vs response model](#request-model-vs-response-model) | +| `422` | Unsupported deployment name or question definition | Check the `model` value and each question definition | +| `429` | The deployment rate limit was exceeded | Retry with backoff, or request more quota in Azure. You can also add retries or fallbacks in a [gateway config](/aigw/product/ai-gateway/configs) | + +Upstream errors use the standard error shape, with the message prefixed by `azure-ai`: + +```json +{ + "error": { + "message": "azure-ai error: The API deployment microsoft-decision-1 does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.", + "type": null, + "param": null, + "code": null + }, + "provider": "azure-ai" +} +``` + +### Limitations and responsible use + +From Microsoft's guidance for this model: + +- **No explanations.** The model returns probabilities, not a written rationale. +- **Wording sensitivity.** Scores can change with how you phrase or order questions and options. +- **Bias.** The model might reflect biases from its base model and training data. Do not use its scores as the sole basis for decisions about individuals. +- **Safety filtering.** If you use it as a safety filter, it can miss subtle harmful content or flag benign content. +- **Validate on your data.** Test thresholds against labeled examples from your own workload before you automate a gate. + +See [Known limitations and best practices](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision) in Microsoft's guide for the full list. + +--- + ## Next Steps diff --git a/aigw/product/ai-gateway/decisions-api.mdx b/aigw/product/ai-gateway/decisions-api.mdx index 6857d66e..8a9bce29 100644 --- a/aigw/product/ai-gateway/decisions-api.mdx +++ b/aigw/product/ai-gateway/decisions-api.mdx @@ -174,6 +174,7 @@ Either shape works with any provider. See [Switching Shapes](#switching-shapes). | [OpenRouter](/aigw/integrations/llms/openrouter) | `openrouter` | Jev | Routes to Jev models | | [Workers AI](/aigw/integrations/llms/workers-ai) | `workers-ai` | Jev | Cloudflare model ID, for example `@your-cf-integration-slug/model` | | [OpenAI](/aigw/integrations/llms/openai) | `openai` | OpenAI | `gpt-6-luna` | +| [Microsoft Decision 1](/aigw/integrations/llms/azure-foundry#decisions-microsoft-decision-1) | `azure-ai` | Jev | Azure AI Foundry. `model` is your deployment name | Any provider accepts either request shape. The AI Gateway converts the request to the provider's native shape and converts the answers back. See [Switching Shapes](#switching-shapes). diff --git a/integrations/llms/azure-foundry.mdx b/integrations/llms/azure-foundry.mdx index 455d5d56..513324b7 100644 --- a/integrations/llms/azure-foundry.mdx +++ b/integrations/llms/azure-foundry.mdx @@ -1037,6 +1037,254 @@ The native `/v1/ocr` endpoint requires gateway version **2.18.0** or higher for --- +## Decisions (Microsoft-Decision-1) + +`Microsoft-Decision-1` is a decision model available in Microsoft Foundry. It does not generate text. It reads a piece of text or JSON, answers typed questions about it, and returns probabilities that your application can threshold or rank. Use it for classification, routing, ranking, grading, and yes/no checks. + +Portkey routes these calls through `/v1/decisions`, a separate endpoint from chat completions. You still get the gateway's full feature set: authentication, routing, retries, circuit breaking, caching, logging, and cost tracking. See the [Decisions API](/product/ai-gateway/decisions-api) for the full request and response reference. + + + `/v1/decisions` does not support streaming. Only before-request [guardrails](/product/guardrails/decisions-guardrails) apply, scanning the content to judge only (`state`). + + + + This section covers how to call Microsoft-Decision-1 through Portkey. For model details, deployment steps in Azure, regional availability, and responsible-use guidance, see Microsoft's official guide: [Deploy and use Microsoft-Decision-1 in Microsoft Foundry](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision). + + +### Model at a glance + +These values come from [Microsoft's model overview](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision#microsoft-decision-1-at-a-glance). + +| Model name | Model version | Deployment type | API type | +|:---|:---|:---|:---| +| `Microsoft-Decision-1` | 1 | `DataZoneStandard` in selected regions, `GlobalStandard` | Decision | + +### Request model vs response model + +This is the most common source of errors with this model, so read it before you send a request. + +| Field | Value | Example | +|:---|:---|:---| +| `model` in your **request** | The **deployment name** you chose in Azure AI Foundry | `@azure-foundry/decision-1` | +| `model` in the **response** | The underlying model | `microsoft-decision-1` | + + + Do not hardcode `microsoft-decision-1` as the request `model`. Azure looks up a **deployment** with that exact name. If your deployment has a different name, the request fails with `The API deployment microsoft-decision-1 does not exist`. Use the deployment name you see in Foundry. Microsoft describes this in the [Call the decision API](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision) section of its guide. + + +For example, if you deploy the model with the deployment name `decision-1`, send `"model": "@azure-foundry/decision-1"`. If you name the deployment `pi-decision-1`, send `"model": "@azure-foundry/pi-decision-1"`. In both cases the response reports `"model": "microsoft-decision-1"`. + +Portkey logs and prices the call using the response `model`, so cost tracking works the same no matter what you name your deployment. + +### Setup + +1. Deploy `Microsoft-Decision-1` in your Foundry project by following [Microsoft's deployment steps](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision). From the deployment details page, copy the **resource endpoint**, the **deployment name**, and the **API key**. +2. Create your Azure AI Foundry provider by following [Creating Your Azure AI Foundry Provider](#creating-your-azure-ai-foundry-provider) above. Use the URL as shown in Foundry for your resource, for example `https://your-resource-name.services.ai.azure.com/models`. +3. If you manage models in **Model Provisioning**, add your deployment name as the model slug (for example `decision-1`), exactly as it appears in Foundry. See [Adding Multiple Models](#adding-multiple-models-to-your-azure-ai-foundry-provider). + + + For decisions requests, Portkey sends the call to the resource root at `/providers/microsoft/v1/systemone`. Any path in the saved Foundry URL, including a `/deployments/` suffix, is dropped. No `api-version` is sent. The deployment is selected by the `model` field in the request body. + + +Decisions requests work with the same authentication methods as the rest of this integration: API key, Azure Entra ID, and Azure Managed Identity. With an API key, Portkey sends it in the `api-key` header, which is what the Microsoft decision endpoint expects. + +### Send a decisions request + +Microsoft-Decision-1 uses the Jev request shape: `state` for the content to judge, and a `questions` map. Set `model` to `@provider-slug/deployment-name`. The provider holds the Azure credentials, so no provider headers are needed. + +```bash cURL +curl --request POST \ + --url https://api.portkey.ai/v1/decisions \ + --header 'Content-Type: application/json' \ + --header "x-portkey-api-key: $PORTKEY_API_KEY" \ + --data '{ + "model": "@azure-foundry/decision-1", + "state": { + "current_ask": "hello there, how is your day going?" + }, + "questions": { + "greeting": { + "type": "choice", + "instructions": "Is this a greeting?", + "criteria": { + "yes": "a greeting", + "no": "anything else" + } + } + } +}' +``` + +Response (values are illustrative): + +```json +{ + "model": "microsoft-decision-1", + "answers": { + "greeting": { + "type": "choice", + "choice": "yes", + "probabilities": { "no": 0.0067, "yes": 0.9933 }, + "confidence": 0.9866 + } + }, + "usage": { "input_tokens": 46, "output_tokens": 1 }, + "provider": "azure-ai" +} +``` + +The OpenAI request shape (`input` and a `questions` array) also works. See [Request Shapes](/product/ai-gateway/decisions-api#request-shapes). + +### Question types + +Each entry in `questions` sets a `type`, which determines the shape of its answer. The `state` can be text or JSON. + +| Type | Question it answers | Answer shape | +|:---|:---|:---| +| `noul` | Is this true? | `{ noul: <0 to 1 probability> }` | +| `choice` | Which option is it? | `{ choice, probabilities, confidence }` | +| `score` | How much? | `{ score, legend, probabilities, confidence }` | + +The response `answers` object is keyed by the question names in your request. + +#### `noul`: gates and filters + +Use for yes/no conditions, such as whether a reply promises a refund or a document contains a prompt injection. `criteria` is optional. Define the true and false conditions to make borderline questions more precise. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": "I refunded your $200. You don't need to contact billing.", + "questions": { + "promises_refund": { + "type": "noul", + "instructions": "Does this reply promise the customer a refund?", + "criteria": { + "true": "Commits to returning money", + "false": "Makes no commitment about money" + } + } + } +} +``` + +#### `choice`: classification and routing + +Use to pick exactly one label from a fixed set, such as routing tickets or classifying intent. Describe each option clearly. The response includes a probability for every option, so a low top probability is a good signal to escalate to a person or another model. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": "The API returns 500 on every call.", + "questions": { + "team": { + "type": "choice", + "instructions": "Which team should handle this ticket?", + "criteria": { + "billing": "Charges, invoices, and refunds", + "engineering": "Bugs, errors, and outages", + "support": "How-to and account questions" + } + } + } +} +``` + +#### `score`: ranking and grading + +Use to place an item on an ordered scale, such as incident severity. List the levels from lowest to highest. The returned score is the probability-weighted average of the level indexes, so a four-level scale runs from `0` through `3` and the result can fall between levels. Prefer scores for relative ordering and thresholds, not as absolute ratings. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": "Checkout is down for all EU customers since 09:00.", + "questions": { + "severity": { + "type": "score", + "instructions": "How severe is this incident?", + "criteria": ["Cosmetic", "Minor", "Major", "Critical"] + } + } +} +``` + +#### Combine questions in one request + +Questions that share the same `state` can go in one request. This reduces request count and keeps the context consistent. + +```json +{ + "model": "@azure-foundry/decision-1", + "state": { + "message": "Checkout is down again. This is my third request.", + "customer_contact_count": 3 + }, + "questions": { + "team": { + "type": "choice", + "instructions": "Which team should handle this request?", + "criteria": { + "billing": "Charges, invoices, and refunds", + "engineering": "Bugs, errors, and outages", + "support": "How-to and account questions" + } + }, + "severity": { + "type": "score", + "instructions": "How severe is this issue?", + "criteria": ["Cosmetic", "Minor", "Major", "Critical"] + }, + "repeat_contact": { + "type": "noul", + "instructions": "Has the customer contacted support before?" + } + } +} +``` + +### Pricing and usage + +Usage is reported in the response as `usage.input_tokens` and `usage.output_tokens`. The pricing entry for `microsoft-decision-1` charges for input tokens only. Output tokens, which hold the typed judgment itself, are priced at zero. + +### Decisions troubleshooting + +| Status | Likely cause | What to do | +|:---|:---|:---| +| `400` | Invalid question type or shape | Check `state`, `questions`, `type`, and `criteria` | +| `401` | Missing, invalid, or expired credential | Check the API key or Entra credentials on the provider | +| `403` | The identity has no access to the deployment | Check role assignments and deployment access in Azure | +| `404` | Wrong endpoint, or the deployment name does not exist | Confirm the `model` value matches your deployment name exactly. See [Request model vs response model](#request-model-vs-response-model) | +| `422` | Unsupported deployment name or question definition | Check the `model` value and each question definition | +| `429` | The deployment rate limit was exceeded | Retry with backoff, or request more quota in Azure. You can also add retries or fallbacks in a [gateway config](/product/ai-gateway/configs) | + +Upstream errors use the standard Portkey error shape, with the message prefixed by `azure-ai`: + +```json +{ + "error": { + "message": "azure-ai error: The API deployment microsoft-decision-1 does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.", + "type": null, + "param": null, + "code": null + }, + "provider": "azure-ai" +} +``` + +### Limitations and responsible use + +From Microsoft's guidance for this model: + +- **No explanations.** The model returns probabilities, not a written rationale. +- **Wording sensitivity.** Scores can change with how you phrase or order questions and options. +- **Bias.** The model might reflect biases from its base model and training data. Do not use its scores as the sole basis for decisions about individuals. +- **Safety filtering.** If you use it as a safety filter, it can miss subtle harmful content or flag benign content. +- **Validate on your data.** Test thresholds against labeled examples from your own workload before you automate a gate. + +See [Known limitations and best practices](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision) in Microsoft's guide for the full list. + +--- + ## Next Steps diff --git a/product/ai-gateway/decisions-api.mdx b/product/ai-gateway/decisions-api.mdx index 1822628e..b9d53ca9 100644 --- a/product/ai-gateway/decisions-api.mdx +++ b/product/ai-gateway/decisions-api.mdx @@ -178,6 +178,7 @@ Either shape works with any provider. See [Switching Shapes](#switching-shapes). | [OpenRouter](/integrations/llms/openrouter) | `openrouter` | Jev | Routes to Jev models | | [Workers AI](/integrations/llms/workers-ai) | `workers-ai` | Jev | Cloudflare model ID, for example `@your-cf-integration-slug/model` | | [OpenAI](/integrations/llms/openai) | `openai` | OpenAI | `gpt-6-luna` | +| [Microsoft Decision 1](/integrations/llms/azure-foundry#decisions-microsoft-decision-1) | `azure-ai` | Jev | Azure AI Foundry. `model` is your deployment name | Any provider accepts either request shape. Portkey converts the request to the provider's native shape and converts the answers back. See [Switching Shapes](#switching-shapes).