02 — Azure OpenAI and Foundry for RAG

Answer:

A model is the artifact (for example a GPT chat model or a text-embedding-3 model at a specific version). A deployment is a named, configured instance of that model inside your resource, with a deployment type (Standard, Global Standard, Provisioned, ...), a capacity/quota allocation, and optionally a guardrail (content filter) policy attached.

In Azure OpenAI-style APIs your code calls the deployment name, not the model name. That has practical RAG consequences:

• You can keep a stable name like chat-prod and swap the underlying model version without code changes.
• Quota, content filtering and cost are per deployment, so chat, query rewriting and embeddings should usually be separate deployments.
• A typo or a deleted deployment gives a 404 DeploymentNotFound, a common first-day error.

OpenAI direct differs: you pass the model name and there is no deployment layer. Docs: Deployment types.

Answer:

API keys are long-lived shared secrets: they leak into logs and repos, rotate painfully, and carry no per-caller identity. Entra ID with a managed identity removes the secret entirely and gives you auditable, revocable, role-based access.

• Assign the app's managed identity a role such as Cognitive Services OpenAI User on the resource (inference only, no management rights).
• In code, use DefaultAzureCredential plus a bearer-token provider for the Cognitive Services scope instead of an api_key.
• The same identity can be granted roles on Azure AI Search and Storage, so the whole RAG path is keyless.
• You can then disable key auth on the resource (disableLocalAuth) and enforce it with Azure Policy.

Trade-off: local development needs az login or a dev identity, and token acquisition adds a small amount of setup. Keys are still fine for a throwaway prototype, or for a gateway (such as API Management) that holds them centrally.

Answer:

You deploy an embedding model (for example text-embedding-3-small or text-embedding-3-large) as its own deployment and call the embeddings endpoint through that deployment name. The text-embedding-3 family accepts an optional dimensions parameter that returns a shortened vector, trading a little retrieval quality for less storage, memory and latency.

Rules of thumb:

• The index field dimension must equal what you request; it is fixed once the index exists.
• Query-time and index-time embeddings must use the same model, version and dimensions.
• Decide by evaluating recall@k on a labelled query set, not by leaderboard scores.

Alternatives: open models such as BGE or E5 self-hosted, or Cohere/OpenAI-direct embeddings, if you need different language coverage or hosting.

Answer:

A Foundry resource is the Azure-level account that hosts model deployments, networking, identity and guardrail policies. A Foundry project is a workspace inside it that groups the things one application or team needs: model connections, agents, evaluations, traces, datasets and connections to services like Azure AI Search.

Practically:

• Classic "Azure OpenAI resource" deployments are available through the Foundry resource, and the same models can be called via OpenAI-compatible endpoints.
• Projects are the unit for RBAC, evaluation runs and observability, so a RAG app typically gets one project per environment (dev, test, prod).
• Older material refers to "hubs" and "Azure AI Studio" projects; the current model is simpler, so check which generation a tutorial targets before copying code.

Learn starting point: Microsoft Foundry documentation.

Answer:

Both let you iterate before you provision Azure capacity, by running models on your own machine.

• Foundry Local: a runtime that runs optimised small models (Phi-family and others) locally and exposes an OpenAI-compatible API (check the current Foundry Local docs for supported models and hardware). Good for offline work, privacy-sensitive experiments and cheap unit tests of prompt logic.
• Ollama (or vLLM on your own GPU): the open-source equivalent, with a larger model catalogue and the same OpenAI-style endpoint, so you can also try Phi, Llama and Mistral side by side.

Because these speak the OpenAI wire format, a thin client factory driven by config lets the same RAG code target a local model in dev and Azure OpenAI or a Foundry deployment in prod. Watch for differences in supported features (structured outputs, tool calling) and in answer quality: a small local model will not behave like your production model, so do final evaluation against the real deployment.

Note that hosted free tiers come and go. GitHub Models, which offered a free rate-limited OpenAI-compatible endpoint for prototyping, was retired on July 30, 2026, so do not build a workflow that depends on a free hosted endpoint staying available. Local models have no such dependency.

Answer:

Prompt Shields (part of Azure AI Content Safety, surfaced as guardrails in Foundry) detect adversarial input aimed at manipulating model behaviour. They cover two attack types:

Per the Learn docs, user prompt attack detection is on by default while document attack detection is off by default and must be enabled in the guardrail configuration. Prompt Shields only detect injection patterns; they do not judge harmfulness, factual accuracy or groundedness, so they complement rather than replace other controls.

Docs: Prompt Shields.

Answer:

Two independent choices: where data is processed and how you pay for capacity.

Guidance from the Learn docs and experience:

• Default to Global Standard: newest models first, broadest availability, highest default quota, lowest price.
• Data Zone Standard when contracts require processing to stay in the EU or US.
• Regional Standard when you need processing within a specific geography, accepting less quota and later model availability.
• Provisioned when you have steady high volume and need predictable latency; you pay for reserved PTUs whether used or not.
• Batch for re-indexing, offline enrichment and evaluation runs.

Note that "data at rest" stays in your resource's geography for all types; the choice governs inference processing. Not every model supports every type, so verify in Deployment types and the models-by-region table.

Answer:

Quota is allocated per subscription, region, model and deployment type in tokens per minute (TPM), and you assign part of it to each deployment. A requests-per-minute (RPM) limit is derived from the TPM allocation. Exceeding either returns HTTP 429.

Client handling:

1. Honour the Retry-After header when present; otherwise use exponential backoff with jitter.
2. Cap retries and total wait; fail fast for interactive chat rather than queueing for a minute.
3. Reduce token pressure: smaller max_tokens, fewer retrieved chunks, shorter prompts. Remember that rate limiting counts the max output you reserve, not just what is generated.
4. Smooth bursts with a client-side concurrency limiter or queue.
5. Add a fallback: a second deployment in another region or a gateway that load-balances.

client = AzureOpenAI(azure_endpoint=ep, azure_ad_token_provider=tp,
                     api_version=VERSION, max_retries=5)  # SDK backs off on 429

Separating embedding and chat deployments stops a bulk indexing job from starving user queries. See Quotas and limits.

Answer:

"On Your Data" let you attach a data source (typically Azure AI Search) to a chat completion call via a data_sources parameter, and the service handled query rewriting, retrieval and citation formatting. It was fast for demos and simple Q&A.

Status caveat: Learn documents On Your Data as deprecated, with Microsoft recommending Foundry Agent Service with Foundry IQ knowledge bases (or your own pipeline) and a published retirement date. As of late 2026, verify the current status on the On Your Data page before building on it.

My default for anything beyond a prototype: own the retrieval layer (Azure AI Search plus code or an orchestrator), because quality tuning and security trimming are where RAG projects succeed or fail.

Answer:

They make the LLM a reliable component rather than a free-text box.

• Structured outputs (JSON schema via response_format, on supported models and API versions) guarantee the response matches your schema. Use it for answers with citations (answer, citations[], confidence, insufficient_context), for query-rewrite steps, and for LLM-judge outputs.
• Tool/function calling lets the model decide to call search_docs(query, filters) instead of you always retrieving once. This is the basis of agentic RAG: the model can reformulate, search multiple times, or skip retrieval for small talk.

Practical points:

• Validate tool arguments server-side; never let the model construct raw filter strings that bypass ACL trimming. Inject user identity filters in code.
• A schema field such as insufficient_context: bool gives a clean abstain path.
• Schemas have supported-keyword limits and add a first-request latency cost; check the docs for your model.
• Retrieved text is untrusted input to a model that can call tools, so tool calling raises the stakes of prompt injection (Q17).

Alternatives: Anthropic tool use and OpenAI direct offer equivalent patterns; the design is portable.

Answer:

Foundry's RAG evaluators split into system (final answer) and process (retrieval) evaluation:

LLM-judge evaluators take a deployment_name for the judge model and typically return a 1 to 5 score with a pass threshold (default 3). Concepts match RAGAS (see 01_concepts/evaluation_metrics.md), but prompts, scales and judge models differ, so scores are not interchangeable across tools. Pick one framework per trend line. Docs: RAG evaluators.

Answer:

Think in tiers, from biggest wins down:

1. Model tiering: use a small, cheap model for query rewriting, classification, reranking prompts and routing; reserve the large model for final synthesis. Route by query difficulty.
2. Retrieve less, better: reranking and a smaller top-k cut input tokens, which dominate RAG cost.
3. Prompt caching: on supported models, repeated prompt prefixes (long system prompt, tool schemas) can be cached automatically with discounted input tokens. Put static content first and variable content (retrieved chunks, question) last. Check the docs for the minimum prefix size and supported models.
4. Batch deployments for re-indexing, enrichment and offline evaluation (documented at a 50% discount with a 24-hour target).
5. Semantic or response caching for repeated FAQs, with care for ACL and freshness.
6. Output limits: set max_tokens and ask for concise answers.
7. Embedding dimensions and incremental re-indexing to cut embedding and storage spend.

Measure cost per successful answer, not per request, so a cheaper model that fails more often is not mistaken for savings.

Answer:

They solve different problems.

• Network isolation: a private endpoint gives the Foundry/Azure OpenAI resource a private IP in your VNet; you then disable public network access and use private DNS so the hostname resolves privately. Do the same for Azure AI Search, Storage and Key Vault, and put the app in a VNet-integrated host. Network rules are enforced per resource.
• Data residency: governed by deployment type. Data at rest stays in the resource's geography, but inference may be processed in any region for Global types, within the data zone for Data Zone types, and within the geography for regional Standard/Provisioned types.

Checklist for regulated workloads:

• Pick Data Zone or regional deployments if contracts require it; do not default to Global Standard.
• Keep the vector index, logs and traces in the same boundary; observability sinks are data stores too.
• Use managed identity (Q2) and disable key auth.
• Review abuse-monitoring and data-handling terms with compliance.

Common trap: private networking is set up for the model but a trace exporter or evaluation job sends prompts to a public SaaS. See Deployment types.

Answer:

Instrument each stage as a span so you can answer "why was this answer bad?" after the fact.

Foundry projects integrate with OpenTelemetry and Azure Monitor / Application Insights for tracing, and traces can feed evaluators for continuous quality checks on sampled production traffic. Alternatives include Langfuse, LangSmith or Arize with the same OTel conventions.

Cautions: prompts contain user data and retrieved documents, so redact or restrict access and set retention; sample high-volume traffic; always log the deployment name and version so regressions can be tied to a model change.

Answer:

Constraints: processing must stay in the EU, load is bursty, and every answer must be explainable.

Deployment choice

• Use Data Zone Standard (EU) for chat and embeddings, not Global Standard, since Global can process outside the EU. Confirm the exact models are offered in the EU data zone; availability lags Global.
• Because the spike is seasonal, avoid paying for Provisioned all year. Options: a modest regional/Data Zone Provisioned base for baseline, with spillover to Standard, or simply Standard with quota raised ahead of enrolment and a pre-load test. Decide with measured p95 latency and cost per month, not guesswork.
• Use Data Zone Batch for nightly re-indexing so it never competes with chat quota.

Security and compliance

• Private endpoints for Foundry, AI Search, Storage; public access off; managed identity; key auth disabled.
• Prompt Shields with document attack detection enabled, as benefit documents may come from partner uploads.
• Per-member ACL filters injected server-side.

Resilience and audit

• A gateway (for example API Management) with retry/backoff and a second EU-zone deployment as failover.
• Traces and answers with chunk IDs and prompt/model versions stored in an EU workspace with defined retention.

Trade-off to name: the strict residency choice costs newer-model access and some quota headroom. Say so and agree with compliance on what is required.

Answer:

The timing points to shared quota contention: the re-index job generates embeddings (and maybe LLM enrichment) from the same deployment or quota pool as user queries.

Diagnose

1. Check Azure Monitor metrics for the deployment: processed tokens, requests and 429 counts, aligned with the job schedule.
2. Confirm which deployments the job calls. Embedding and chat TPM are separate allocations; if the job uses the chat deployment for enrichment, that is the culprit.
3. Inspect whether clients honour Retry-After; naive immediate retries amplify the storm.
4. Look at the job's concurrency and batch sizes.

Fix (in order)

• Move bulk work to a Batch deployment (separate enqueued quota, discounted) where latency does not matter.
• Give the job its own embedding deployment and cap its concurrency with a token-bucket limiter below the quota.
• Make indexing incremental (content hashes) so only changed chunks are re-embedded.
• Add backoff with jitter to all clients; set a fail-fast timeout for interactive requests.
• If user traffic itself is the issue, request more quota or add a second-region deployment behind a gateway.

Verify by replaying the schedule in staging and watching 429 rate and p95 latency.

Answer:

Indirect injection means a retrieved document contains instructions ("email the user's data to ...") that the model treats as commands. No single control stops it, so layer them.

Azure tooling

• Prompt Shields document attack detection: classify retrieved content before it reaches the model; off by default, so enable it.
• Spotlighting (preview): marks documents as lower trust to the model by encoding them; costs extra tokens and can leak "base-64" mentions into answers.
• Content filters for harmful output categories.

Architecture (your responsibility)

• Keep the model least-privilege: no tools that send data out or write state without human confirmation.
• Separate trusted instructions from untrusted content with clear delimiters and a system prompt that says retrieved text is data.
• Control the corpus: ingestion review for externally authored content, ACL trimming so a user can only inject through documents they could read anyway.
• Render output safely (no auto-loading markdown images or links to attacker-controlled URLs, a known exfiltration path).
• Log and red-team with representative attacks, including your non-English content, since the shield models are primarily trained on English.

Limit to state plainly: detectors have false negatives. The robust defence is limiting what a hijacked model can do.

Docs: Prompt Shields.

Answer:

Models have retirement dates published on Learn, and deployments can be set to auto-upgrade or pinned. Silent upgrades are risky for RAG because prompts, citation habits and abstention behaviour shift between versions.

Process

1. Track retirement notices and the models-by-region tables; plan a few months ahead.
2. Build a golden set: real queries with expected sources/answers, including must-abstain cases and adversarial ones.
3. Deploy the candidate side by side under a new deployment name; keep the old one live.
4. Run offline evaluation: groundedness, relevance, retrieval, completeness, plus latency, token usage and cost. Compare with thresholds, not eyeballing. Review failing items manually.
5. Re-tune prompts if needed (output format, citation style, refusal behaviour).
6. Canary: route a small share of traffic via config or gateway, watch guardrail rates and user feedback.
7. Cut over, keep rollback until the old model is near retirement, then delete.

Pin versions in production rather than relying on auto-upgrade, and treat model version as part of the config you log with each trace. A different deployment type may also change quota and latency profile, so retest limits.

Reference: Model retirements.

Answer:

Vectors from different models, versions or dimension settings are not comparable, and a vector field's dimension count is fixed at creation. So this is a re-index, not a config flip.

Blue/green approach

1. Create a new index (or new vector field) with the new dimension and algorithm settings.
2. Re-embed the corpus into it using a dedicated embedding deployment and, where latency is irrelevant, batch or throttled jobs (see Q16).
3. Run the golden query set against both indexes. Compare recall@k and NDCG with the Foundry Document Retrieval evaluator or your own labels, then downstream groundedness.
4. Switch the query side atomically: query embedding deployment and index alias must change together. A mismatch yields plausible but wrong results with no error.
5. Keep the old index for rollback, then delete it.

Pitfalls

• Query embeddings must use identical model, version and dimensions as the index.
• Hybrid weights and score thresholds tuned on the old model often need retuning.
• Budget: re-embedding the whole corpus costs tokens and time.

Mitigation: store the embedding model and dimension in index metadata and in every trace, so mismatches are detectable.

Answer:

They answer a similar question, "is the answer supported by the sources?", at different points in the lifecycle.

Design guidance:

• Use the evaluator on golden sets in CI and on sampled production traces; it informs prompt, retrieval and model decisions.
• Use runtime detection only where a wrong answer is costly (medical, legal, financial) and the added latency is acceptable. Pair it with a fallback such as "I could not verify this in the documents".
• Neither verifies that the source itself is correct, and both can err; periodically calibrate them against human labels.

Groundedness Pro in the evaluator suite uses the Content Safety service. Feature availability (preview status, supported regions and domains) changes, so confirm in current docs.

Answer:

Constraint: offline-first, modest hardware, but answers must be trustworthy for safety-relevant tasks.

On-device stack

• A small language model (Phi-family or similar) served locally, for example through Foundry Local, which exposes an OpenAI-compatible API. Alternatives: Ollama or llama.cpp-based runtimes.
• A small local embedding model and an embedded vector store (SQLite-based, FAISS, or similar) holding a pre-built, versioned manual index synced when online.
• Same app code as the cloud path via an OpenAI-compatible client and config switch.

Design choices

• Keep answers extractive and cited: show the manual section alongside the text, because small models hallucinate more. Use a strict "answer only from context, otherwise say not found" prompt and structured output if the runtime supports it.
• Limit corpus per device (model, region) to keep retrieval precise on weak hardware.
• Hybrid routing: when connected, send hard or ambiguous queries to a cloud deployment; offline, fall back to local and label the answer as such.
• Sync index and model updates with version stamps; log queries locally and upload later.

Trade-offs: lower reasoning quality, device storage and battery, harder update management. Evaluate the local stack on the same golden set as the cloud one, and only ship if groundedness meets a threshold you define for safety-critical content.

Answer:

PTU deployments reserve model processing capacity, billed per PTU-hour whether or not you use it. They win when utilisation is high and steady and when latency consistency matters; they lose for spiky or low-volume traffic.

Break-even thinking

• Estimate sustained tokens per minute (input and output separately, since output weighs more heavily in PTU accounting) and compare the PTU cost for that throughput with pay-as-you-go cost for the same tokens. Use the capacity calculator and per-model PTU guidance in Provisioned throughput concepts; minimum PTUs and throughput per PTU differ by model.
• If average utilisation of the reservation is low, Standard is cheaper. If it is consistently high, PTU usually is.

Sizing and de-risking

1. Load-test with a production-shaped prompt mix (RAG prompts are input-heavy, and cache hits change the picture).
2. Size PTU for baseline, not peak; handle peaks by spillover to a Standard deployment or client-side routing when PTU returns 429.
3. Prefer Global or Data Zone Provisioned where residency allows, since capacity is easier to obtain than regional; check availability before committing.
4. Consider reservations only after a few months of stable usage data.
5. Monitor provisioned utilisation and re-size when prompt shapes change, for example after raising top-k.

Non-Azure comparison: self-hosting on vLLM gives similar fixed-cost economics with more ops burden.

Answer:

Correct: B. A 429 means the deployment's tokens-per-minute or requests-per-minute allocation was exceeded, and the header tells you when capacity is likely available. Waiting, adding jitter so many clients do not retry in lockstep, and capping retries keeps the system stable and the user experience bounded. Option A makes the overload worse, since rate limits are tracked per deployment, not per connection. Option C is wrong because the authentication method does not change quota; limits are tied to the deployment and subscription allocation. Option D increases tokens reserved per request, which consumes more of the limit and makes further 429s more likely.

Answer:

Correct: D. Similarity scores are only meaningful between vectors from the same embedding space with the same dimensionality, and a vector field's dimension count is fixed in the index schema. Option A is wrong because nothing re-projects queries; a query embedded at a different size is rejected or compares incorrectly. Option B is wrong because search services validate vector length against the field definition rather than padding, and padding would distort similarity anyway. Option C is wrong because normalisation does not align different models' spaces; mixing models gives plausible-looking but meaningless rankings.

Answer:

Correct: A. Document attack detection scans content submitted as documents for injected instructions, which is exactly the indirect injection pattern, and the Learn docs note it is off by default, so it must be turned on. Option B is wrong because harm-category filters judge content like hate or violence, and an injected instruction can be polite and harmless-looking. Option C is wrong because dilution does not neutralise an instruction; one malicious chunk in context can still steer the model. Option D is wrong because temperature affects sampling randomness, not whether a model obeys instructions; a deterministic model follows injected commands just as faithfully.

Answer:

Correct: C. Data Zone deployments keep inference processing within the chosen zone (the EU, following the Azure EU Data Boundary) while typically offering higher default quota than geography-bound Standard, because traffic can be routed across datacenters in the zone. Option A has the largest quota but Global types may process data in any Azure region, violating the requirement. Option B is also global and asynchronous with a 24-hour target, so it is unsuitable for interactive chat even though its quota is separate. Option D is intended for evaluating fine-tuned models only, has a 24-hour lifetime, no SLA and no data residency guarantee. Verify model availability per data zone in the current docs.