One base_url, many models: migrating to OpenAI-compatible routing

Teams that already use the official SDKs tend to assume that adopting an aggregation gateway means rewriting the client layer. It does not. With an OpenAI-compatible protocol, three things change: base_url, api_key and model. Everything else — the message structure, stream parsing, tool calls, retry logic — stays exactly as it is.

What is compatible is the shape, not the capability

An OpenAI-compatible layer unifies the shape of requests and responses: /v1/chat/completions, /v1/embeddings, the messages array, data: events on a stream, tool_calls, the usage field. The gateway translates each request into the upstream provider's native protocol and normalises the reply back into the OpenAI shape. "It works" therefore means the interface lines up — it does not mean every model has the same context window, image input support or structured-output behaviour. Those still have to be checked model by model.

The same holds for the Anthropic protocol (/v1/messages): two headers, x-api-key and anthropic-version, plus the gateway's path.

Migration order

1. Change base_url first, nothing else. The field name differs per SDK: base_url in Python, baseURL in Node, api_base in some libraries, or the OPENAI_BASE_URL environment variable. Point it at the gateway and stop at /v1 — the SDK appends /chat/completions itself, and one segment too many returns a 404.
2. Swap the key. A gateway key belongs on the server. Anything that puts it in a front-end bundle, a mobile client, a mini program or a Git repository counts as a leak.
3. Swap the model. Use the gateway's model IDs and pull the live list from /v1/models. Do not carry over upstream aliases; providers rename and retire models all the time.
4. Verify in order: a short non-streaming prompt, then streaming, then a tool call. At every step check that usage comes back — that is what you will reconcile the bill against.

Five mistakes worth avoiding

• A doubled /v1. base_url already ends in /v1 and the SDK appends it again, so the request lands on /v1/v1/chat/completions.
• Streaming code that only handles [DONE]. Some implementations ignore everything but the terminator; a compatibility layer may also interleave comment lines (keep-alives). Filter by the data: prefix instead of assuming every line is JSON.
• Timeouts. Image and video jobs routinely take tens of seconds to minutes, while a client default of 60 seconds disconnects first. A broken transport is not an upstream failure, and blind retries can bill you twice.
• Matching on error text. The gateway normalises errors into the OpenAI error structure, but upstream wording varies. Branch on the HTTP status code plus error.code; never regex the message.
• Unconditional retries. Retry only 429 and 5xx, with exponential backoff and jitter. Every retry is a new billable request.

Once the migration is in, confirm the request really went end to end rather than merely "not failing": log the request identifier the gateway returns so you can find the same call in the console's request log later, and check that the response carries both input and output numbers in usage. A missing usage means the path was never completed. Those two checks are worth far more than a 200 status — a 200 only proves this one call worked, while reconcilable usage proves you can investigate every future failure.

Rollout and rollback

Put base_url, api_key and model into environment variables so the same code can run two configurations: one direct to the provider, one through the gateway. Roll out internally first, then one non-critical endpoint, then everything. Keep the old configuration for at least one release cycle; when something goes wrong you flip one environment variable instead of editing code or shipping a build.

For exact paths, error codes and field definitions see the API reference and the quickstart. Model IDs and current prices live on the pricing page; the authoritative charge is the one on your invoice.