Connect LiteLLM to OpenLLM Buddy

Use LiteLLM with OpenLLM Buddy: call it from the Python SDK or add it to the LiteLLM proxy config for a unified gateway with keys and budgets.

SDK & framework · Integration guide

What is LiteLLM?

LiteLLM translates one OpenAI-style interface to 100+ providers. Run it as a Python library or as a proxy server with virtual keys, budgets, and logging. Visit LiteLLM.

Why connect OpenLLM Buddy?

Put OpenLLM Buddy behind LiteLLM to give every team member their own key and budget, to fall back between models, or to hand tools a stable URL while you swap models behind it.

Before you start, create an API key in the console and deploy a model. If anything fails, run the curl test in Step 4 first: no tool can fix a broken endpoint.

Call it from the Python SDK

Prefix the model with openai/ and pass api_base so LiteLLM treats it as a generic OpenAI-compatible server.

sdk.py

python

import os
import litellm

resp = litellm.completion(
    model="openai/gemma4:26b",
    api_base="https://openllmbuddy-proxy.botbuddytech.workers.dev/v1",
    api_key=os.environ["OPENLLMBUDDY_API_KEY"],
    messages=[{"role": "user", "content": "Hello from LiteLLM"}],
)
print(resp.choices[0].message.content)

Or add it to the proxy

For a shared gateway, install the proxy and describe the model in config.yaml.

config.yaml

yaml

model_list:
  - model_name: buddy
    litellm_params:
      model: openai/gemma4:26b
      api_base: https://openllmbuddy-proxy.botbuddytech.workers.dev/v1
      api_key: os.environ/OPENLLMBUDDY_API_KEY

model_name is the alias your clients use. You can list several deployments under the same alias for load balancing.

Run the proxy and call it

Start the proxy, then use any OpenAI client against http://localhost:4000.

Terminal

bash

pip install 'litellm[proxy]'
export OPENLLMBUDDY_API_KEY=YOUR_API_KEY
litellm --config config.yaml --port 4000

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "buddy", "messages": [{"role": "user", "content": "Reply with OK"}]}'

Set a master_key in general_settings before exposing the proxy beyond your machine.

Test the endpoint with curl

Confirm the key, URL, and model id work before debugging LiteLLM. A short JSON reply means the endpoint is healthy.

Terminal

bash

curl -sS https://openllmbuddy-proxy.botbuddytech.workers.dev/v1/chat/completions   -H "Authorization: Bearer YOUR_API_KEY"   -H "Content-Type: application/json"   -d '{"model": "gemma4:26b", "messages": [{"role": "user", "content": "Reply with OK"}], "max_tokens": 16}'

Troubleshooting

`LLM Provider NOT provided` — the openai/ prefix is missing from the model name.
Wrong model in logs — the id after openai/ is sent to the endpoint unchanged, so it must match your deployment exactly.
401 or 403 — the API key is wrong, revoked, or missing the Bearer prefix. Create a fresh key in the console.
404 on /chat/completions — the base URL must end in /v1, with no trailing /chat/completions and no extra path.
Model not found — the model id must match your deployment exactly. Run the curl test and copy the id you used.
Timeouts on the first request — a GPU that was idle may still be loading the model. Retry after a minute.
Connection refused from Docker or a remote host — use the public HTTPS base URL, never localhost.

Connection details

Use these values wherever LiteLLM asks for an OpenAI-compatible provider.

Reference

text

Base URL:  https://openllmbuddy-proxy.botbuddytech.workers.dev/v1
Chat URL:  https://openllmbuddy-proxy.botbuddytech.workers.dev/v1/chat/completions
API key:   your key from the console
Model id:  gemma4:26b