Connect LlamaIndex to OpenLLM Buddy

Connect LlamaIndex to OpenLLM Buddy using the OpenAILike LLM class with a custom api_base for RAG and agent workflows.

SDK & framework · Integration guide

What is LlamaIndex?

LlamaIndex is a data framework for retrieval-augmented generation: loaders, indexes, query engines, and agent workflows over your own documents. Visit LlamaIndex.

Why connect OpenLLM Buddy?

The OpenAILike class is made for third-party OpenAI-compatible servers, so your documents stay in your pipeline and the model runs on your OpenLLM Buddy GPU.

Before you start, create an API key in the console and deploy a model. If anything fails, run the curl test in Step 4 first: no tool can fix a broken endpoint.

1

Install

Install LlamaIndex and the OpenAI-like LLM package.

Terminal
bash
pip install llama-index llama-index-llms-openai-like
2

Create the LLM

is_chat_model=True makes it use /v1/chat/completions.

llm.py
python
import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="gemma4:26b",
    api_base="https://openllmbuddy-proxy.botbuddytech.workers.dev/v1",
    api_key=os.environ["OPENLLMBUDDY_API_KEY"],
    is_chat_model=True,
    context_window=32768,
)

print(llm.complete("Define vector search in one line."))

Set context_window to your deployment's real context length.

3

Use it as the default LLM

Register it in Settings so query engines and agents pick it up automatically.

rag.py
python
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader

Settings.llm = llm
# Embeddings are separate: configure a local or hosted embedding model.

docs = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("What do these docs say about pricing?"))

OpenLLM Buddy serves chat models. Choose an embedding model separately, for example a local HuggingFace one.

4

Test the endpoint with curl

Confirm the key, URL, and model id work before debugging LlamaIndex. A short JSON reply means the endpoint is healthy.

Terminal
bash
curl -sS https://openllmbuddy-proxy.botbuddytech.workers.dev/v1/chat/completions   -H "Authorization: Bearer YOUR_API_KEY"   -H "Content-Type: application/json"   -d '{"model": "gemma4:26b", "messages": [{"role": "user", "content": "Reply with OK"}], "max_tokens": 16}'

Troubleshooting

  • Asks for an OpenAI key for embeddings — the default embedding model is OpenAI's. Set Settings.embed_model to a local model.
  • Context errors — keep context_window at or below what the deployment really supports.
  • 401 or 403 — the API key is wrong, revoked, or missing the Bearer prefix. Create a fresh key in the console.
  • 404 on /chat/completions — the base URL must end in /v1, with no trailing /chat/completions and no extra path.
  • Model not found — the model id must match your deployment exactly. Run the curl test and copy the id you used.
  • Timeouts on the first request — a GPU that was idle may still be loading the model. Retry after a minute.
  • Connection refused from Docker or a remote host — use the public HTTPS base URL, never localhost.

Connection details

Use these values wherever LlamaIndex asks for an OpenAI-compatible provider.

Reference
text
Base URL:  https://openllmbuddy-proxy.botbuddytech.workers.dev/v1
Chat URL:  https://openllmbuddy-proxy.botbuddytech.workers.dev/v1/chat/completions
API key:   your key from the console
Model id:  gemma4:26b