Deploying a Gen AI App on the Cloud: A Walkthrough

A prototype that calls a language model is a few lines in a notebook. A service other people rely on needs more: limits on what users can send, a key that is not in the code, sensible behaviour when the model is slow or down, tests that run in seconds, and a way to run it in the cloud. None of this is specific to AI, but the model call makes each of these points sharper.

This walkthrough builds that service with FastAPI, checks it locally, and deploys it to Google Cloud Run.

The service

The application has two endpoints: a health check for the platform, and /ask for questions. Its design choices are the point:

• Configuration from the environment. Model name, address and key come from environment variables, so the same container runs locally against a test model and in production against a hosted one.
• Input limits. Questions must be 3 to 1,000 characters. Longer input costs money, slows replies and widens the room for abuse.
• One place that talks to the model, wrapped as a dependency, with a timeout and limited retries.
• Failures become clear messages. If the model times out, the user gets a 503 and a sentence they can act on, and the details go to the log.

import logging
import os
import uuid

from fastapi import Depends, FastAPI, HTTPException
from fastapi.testclient import TestClient
from openai import OpenAI
from pydantic import BaseModel, Field

# Configuration comes from the environment, so the same container runs anywhere
MODEL = os.environ.get("MODEL_NAME", "llama3.2")
BASE_URL = os.environ.get("MODEL_BASE_URL", "http://localhost:11434/v1")
API_KEY = os.environ.get("MODEL_API_KEY", "not-needed-locally")     # a secret in production
log = logging.getLogger("ask-service")


class Question(BaseModel):
    question: str = Field(min_length=3, max_length=1000)     # refuse empty and oversized input


class Reply(BaseModel):
    answer: str
    model: str


def get_model():
    client = OpenAI(base_url=BASE_URL, api_key=API_KEY, timeout=60, max_retries=2)

    def complete(question: str) -> str:
        r = client.chat.completions.create(
            model=MODEL, temperature=0, seed=42, max_tokens=120,
            messages=[{"role": "system", "content": "Answer in at most two sentences."},
                      {"role": "user", "content": question}])
        return r.choices[0].message.content

    return complete


app = FastAPI(title="ask-service")


@app.get("/healthz")
def health():
    return {"status": "ok"}


@app.post("/ask", response_model=Reply)
def ask(body: Question, complete=Depends(get_model)):
    request_id = uuid.uuid4().hex[:8]
    try:
        answer = complete(body.question)
    except Exception:
        log.exception("model call failed, request %s", request_id)
        raise HTTPException(status_code=503, detail="The assistant is unavailable. Please try again.")
    return Reply(answer=answer, model=MODEL)


# ---- tests: the model is replaced, so they are fast, free and repeatable ----
def fake_model():
    return lambda question: f"(stand-in answer to: {question})"


def broken_model():
    def fail(question):
        raise TimeoutError("model did not answer")
    return fail


client = TestClient(app)
print("health:", client.get("/healthz").status_code, client.get("/healthz").json())

app.dependency_overrides[get_model] = fake_model
r = client.post("/ask", json={"question": "What is RAG?"})
print("normal question:", r.status_code, r.json())
r = client.post("/ask", json={"question": "Hi"})
print("too short:", r.status_code, r.json()["detail"][0]["msg"])
r = client.post("/ask", json={"question": "x" * 5000})
print("too long:", r.status_code, r.json()["detail"][0]["msg"])

app.dependency_overrides[get_model] = broken_model
r = client.post("/ask", json={"question": "What is RAG?"})
print("model down:", r.status_code, r.json())

# ---- one live request through the real model client, here a local OpenAI-compatible server ----
app.dependency_overrides.clear()
r = client.post("/ask", json={"question": "What is retrieval-augmented generation?"})
print("live model:", r.status_code, r.json())

health: 200 {'status': 'ok'}
normal question: 200 {'answer': '(stand-in answer to: What is RAG?)', 'model': 'llama3.2'}
too short: 422 String should have at least 3 characters
too long: 422 String should have at most 1000 characters
model down: 503 {'detail': 'The assistant is unavailable. Please try again.'}
live model: 200 {'answer': 'Retrieval-Augmented Generation (RAG) is a text generation technique that combines the strengths of both retrieval and generation models to improve the quality and relevance of generated text. It involves first retrieving relevant information from a large database or knowledge base, and then using this retrieved information as input to generate new text, often with improved coherence and accuracy.', 'model': 'llama3.2'}

What the run shows

• Tests do not need the model. FastAPI lets tests replace a dependency through app.dependency_overrides (FastAPI: testing dependencies). The stand-in model made the normal, invalid and failure cases run in a fraction of a second, without cost or flakiness.
• Bad input is rejected before it reaches the model. A two-character question and a 5,000-character one both returned 422 with a reason, and neither cost a token.
• An outage is handled. When the stand-in raised a timeout, the user received a 503 and "The assistant is unavailable. Please try again." instead of a stack trace.
• The real path works too. With the overrides cleared, the same endpoint called the model through the OpenAI client, here pointed at a local OpenAI-compatible server.

Packaging it

A container makes the service run the same everywhere. FastAPI's documentation builds the image from an official Python image and starts the app with fastapi run, which uses Uvicorn, in the exec form of CMD (FastAPI: Docker). Cloud Run sends requests to port 8080 by default and passes the port in the PORT variable (Cloud Run: container contract), so the app listens there:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
CMD ["fastapi", "run", "main.py", "--port", "8080"]

Deploying to Cloud Run

The model key belongs in Secret Manager, not in the image or the code. Cloud Run can expose a secret to the service as an environment variable (Cloud Run: secrets), and gcloud run deploy --source . builds the image from the directory, using the Dockerfile if there is one (Cloud Run: deploying from source):

printf "%s" "$MODEL_API_KEY" | gcloud secrets create model-api-key --data-file=-

gcloud run deploy ask-service --source . --region asia-south1 \
  --set-env-vars MODEL_NAME=your-model,MODEL_BASE_URL=https://api.example.com/v1 \
  --update-secrets MODEL_API_KEY=model-api-key:latest

The service account Cloud Run uses must be allowed to read the secret. Without that permission, the deployment fails.

Scaling and slow responses

Model calls are slow by web standards: seconds, not milliseconds. Two Cloud Run settings matter. Each instance receives several requests at once, 80 by default, which suits a service that spends most of its time waiting on a remote model (Cloud Run: concurrency). And Cloud Run starts more instances under load, by default up to 100 per revision; a lower maximum caps the cost of a traffic spike or an abusive client (Cloud Run: maximum instances).

For long answers, stream tokens to the user as they arrive, so the wait feels shorter. Keep the model client's timeout below the platform's request timeout, so a slow model produces the clean 503 above rather than a dropped connection.

After launch

A deployed service needs watching. Log each request's latency, token counts and outcome, without logging personal data. Set alerts for error rates and for spending. Keep an evaluation set and run it before every change of model or prompt. See evaluating LLM outputs. Watch the bill per request, and see cost optimization for LLM applications. Add input and output checks appropriate to the content. See guardrails for LLMs.

Managed platforms like Cloud Run suit most LLM front ends, because the heavy work happens at the model provider. Hosting an open model yourself is a different problem, with GPUs and batching. See vLLM and model serving.

Common mistakes

• API keys in code or in the container image.
• No input limits, so one user can send a novel per request.
• Raw errors returned to users.
• Tests that call the real model, making them slow, costly and flaky.
• No timeout, so a slow model holds connections open.
• Logging full prompts with personal data.

Deployment strategies, cloud deployment, Docker and API integration make up the Deployment and Production module of the Generative AI course.

Sources

• Testing Dependencies with Overrides (FastAPI documentation) — FastAPI
• FastAPI in Containers - Docker (FastAPI documentation) — FastAPI
• Deploying from source code (Cloud Run documentation) — Google Cloud
• Container runtime contract (Cloud Run documentation) — Google Cloud
• Maximum concurrent requests for services (Cloud Run documentation) — Google Cloud
• Set maximum instances for services (Cloud Run documentation) — Google Cloud
• Configure secrets for services (Cloud Run documentation) — Google Cloud