Key management and rate limits: least privilege for a team

The easy option is to paste one key into the team channel and let everyone use it. The price is steep: usage cannot be attributed to anyone, a leak forces a company-wide swap, and a single batch job can burn the whole team's budget. Key management is not about convenience — it is about being able to stop the bleeding quickly when something goes wrong.

Four least-privilege rules

1. One key per purpose. Name it after who uses it and where: prod-chat-backend, ci-eval-runner. The name goes into logs; the key itself never does.
2. Isolate by environment. Development, staging and production each get their own key, and the development key carries a low cap. Never use a production key for local debugging.
3. Grant only what is needed. Set only the spend cap and expiry the task requires; restrict source IPs where the gateway allows it; whitelist only the models actually in use. Give temporary work a short-lived key that expires on its own — more reliable than remembering to delete it.
4. Revoke on suspicion. When a leak is suspected, do not "rotate the value and carry on" — delete the key and create a new one. Offboarding, handovers and going public all warrant a rotation.

Rotation can be zero-downtime: create the new key, deploy it, confirm traffic is running on it, then delete the old one. The overlap is your double-write window.

Rate limiting is two dimensions, not one

Most people watch requests per minute (RPM) only, but concurrency — requests in flight at the same moment — is a separate ceiling. RPM limits frequency; concurrency limits how many requests are pressing at once. A 30-second request occupies concurrency for far longer than it occupies RPM. Hit either limit and you get a 429 or a refused connection.

So diagnose the bottleneck first: is traffic too dense (raise RPM or slow down), or is each request too slow (lower concurrency, move work to async jobs, reuse connections)?

Treat 429 as normal operation

A robust client must assume some fraction of requests will be limited:

• Retry only 429 and 5xx, with exponential backoff plus jitter. Fixed intervals make every client hit the wall in the same second again.
• Honour retry hints in the response when present, and bound the total number of retries and total wait time.
• Turn batch work into a queue: cap concurrent workers and push failures into a retry queue instead of firing thousands of requests at once.
• Separate "I am too fast" from "the upstream is congested". The first calls for slowing yourself down; the second calls for backing off, not for pushing more pressure upstream.

Observability and team process

Look at usage and error rate per key and you can find the service burning the quota within ten minutes. Log and alert on the key name, not the key value, so every usage record traces back to an owner. Review idle keys weekly and delete them — an unused key is pure risk.

Keys live server-side only: environment variables or a secrets manager. Never commit them, never bundle them into a front end, never paste them into a chat screenshot or a ticket.

Actual caps, RPM and concurrency defaults are shown in the console and in the billing documentation. For auth headers, error codes and /v1/models, see the API reference. When onboarding a new service, walk the full key-creation flow in the quickstart.