Red Hat OpenShift AI AutoRAG is a Technology Preview in OpenShift AI 3.5 that runs retrieval configurations against an evaluation dataset and ranks the results. It can replace weeks of guesswork around chunking, embeddings, and retrieval settings, but only if you already have representative questions with trustworthy expected answers. The first setup also brings an OpenShift 4.20 cluster, an Open GenAI Stack server, a pipeline server, object storage, a vector database, and model endpoints into the picture.

What Red Hat OpenShift AI AutoRAG actually changes

Most RAG teams tune one variable at a time. They pick a chunk size, test five questions, change the embedding model, and keep whichever result looks less wrong. The process feels empirical, but it rarely produces an audit trail or a fair comparison.

AutoRAG turns that work into a pipeline. According to Red Hat's September 11 implementation guide, it ingests documents and an evaluation dataset, tests candidate RAG configurations, and presents a ranked result. The guide uses Qwen3-8B, embeddinggemma-300m, Milvus, MinIO, PostgreSQL, and Open GenAI Stack, previously called Llama Stack.

The important change is not that the platform can build a vector index. Plenty of libraries do that. It is that the experiment, artifacts, and selected configuration live inside the same OpenShift environment where the models can run.

That makes AutoRAG interesting for platform teams that already operate OpenShift AI. It does not make it a shortcut for teams that just need search over a small handbook.

The benchmark is the product

AutoRAG can optimize only what the evaluation set measures. A dataset full of simple fact lookups will reward a pipeline that handles simple fact lookups. It tells you very little about multi-document questions, fresh content, access-controlled records, or answers that should be refused.

The upstream AutoRAG paper supports the general premise that the best RAG modules vary by dataset. That is useful independent evidence for automated configuration search, but it does not validate Red Hat's specific Technology Preview or guarantee that a winning offline configuration will behave well in production.

We would build the evaluation set before installing anything. Start with real queries from support tickets, search logs, product specialists, and failed agent traces. Keep a held-out set that the optimizer never sees. Otherwise, the leaderboard can become a very expensive way to overfit 30 questions.

A practical first set should cover four shapes:

• direct facts contained in one chunk
• answers that require two or more documents
• near matches where the system should say it lacks evidence
• recently changed facts that expose stale ingestion

Record the source document and answer evidence, not just a reference sentence. Two people can phrase a correct answer differently. Retrieval quality and grounded generation quality should remain separate measurements.

The AutoRAG architecture has seven moving parts

Red Hat's published demo requires OpenShift 4.20 or later and the OpenShift AI 3.5 operator. A GPU is required for the example's local model deployment, although Red Hat says a model provider endpoint can be used instead.

PostgreSQL also stores operational configuration, while the OpenShift AI pipeline server persists pipeline artifacts. The separation is sensible for a production platform, but it explains the real cost: AutoRAG is not one container that accepts a folder and returns a YAML file.

The vendor guide later adds an MCP lifecycle operator, MariaDB, an MCP server, and a Streamlit application. Those components demonstrate how retrieval and live tools can coexist. They are not required to prove whether AutoRAG improves retrieval, so we would leave them out of the first evaluation.

If you are deciding how to fit automated RAG evaluation into an existing agent platform, our AI development team can help define the evaluation contract and production architecture before the cluster grows around a weak benchmark.

Set up the smallest useful evaluation

Red Hat's public demo repository contains the manifests referenced by the guide. Pin the demo commit before applying it in a shared cluster so a later repository update cannot change what your run installs.

git clone https://github.com/dialvare/autorag-demo.git
cd autorag-demo
git checkout a2c4c26e9a83448dade51e5388905711b2f46da9

oc new-project ogx
oc label namespace ogx opendatahub.io/dashboard=true

The dashboard features are controlled through OdhDashboardConfig. Set genAiStudio, autorag, and mcpCatalog to true. In the DataScienceCluster, set the ogx component to Managed. These are cluster-level changes, so confirm them with the platform owner rather than hiding them inside an application deployment.

The demo then applies three storage manifests:

oc apply -f 2-storage/postgresql-setup.yaml -n ogx
oc apply -f 2-storage/milvus-setup.yaml -n ogx
oc apply -f 2-storage/minio-setup.yaml
oc apply -f 2-storage/minio-job.yaml

Do not copy the demo credentials into production. The published walkthrough uses minioadmin and minioadmin123, which are suitable only for a disposable environment. Replace static credentials, use separate namespaces and service accounts, and decide who may read source documents, benchmark answers, and pipeline artifacts.

After deploying the language and embedding model endpoints, create the OGX secret and apply the supplied server manifest. The example points to in-cluster vLLM endpoints for Qwen3-8B and embeddinggemma-300m.

oc apply -f 4-ogx/deployment.yaml -n ogx
oc apply -f 5-autorag/pipeline-server.yaml -n ogx
oc apply -f 5-autorag/knowledge-connection.yaml -n ogx

oc get ogxserver,pipelineserver -n ogx
oc get pods -n ogx

These commands and paths match the pinned demo tree. They do not prove your cluster has the operators, storage classes, GPU profile, or permissions needed to make the resources Ready. Treat oc apply success as admission, not readiness.

In the dashboard, create an AutoRAG run against the input_data folder and benchmark_data.json, select the Milvus vector provider, and connect the OGX service. Preserve the run configuration and scores outside the UI so the production choice can be reviewed later.

What the winning configuration does not tell you

A top-ranked run is an offline result. It does not measure ingestion lag, index rebuild time, concurrent query latency, model cold starts, or the cost of rerunning the search after documents change.

It also does not settle permissions. If two employees can ask the same question but are allowed to see different source documents, a single shared index may be the wrong architecture regardless of its retrieval score. AutoRAG can help choose retrieval settings. It cannot design tenancy boundaries for you.

The largest operational pain is configuration promotion. The Red Hat walkthrough has the user manually copy the winning vector store, number of chunks, search mode, and ranker values into the application. That is acceptable for a demo. In production, those values need a versioned artifact, an approval step, a rollback path, and the exact dataset and model versions that produced them.

Our position is to treat the result like a model release. Give it an identifier, deploy it to staging, replay the held-out evaluation set, then shadow a slice of production traffic before replacing the current retriever.

When AutoRAG is worth the infrastructure

AutoRAG is worth evaluating when a team already runs OpenShift AI, owns a meaningful domain corpus, and has enough repeated queries to justify systematic tuning. It is especially relevant when data must stay inside the cluster or when several teams need a common evaluation process.

It is not worth using for a small documentation bot with a few hundred stable pages and no labelled evaluation set. A fixed chunking strategy, one embedding model, Postgres with pgvector, and a carefully reviewed test set will be cheaper to operate and easier to debug.

Technology Preview status is another honest limit. Red Hat explicitly labels AutoRAG as a Technology Preview in OpenShift AI 3.5. We would not put a critical production dependency on its UI or resource behavior until the support and upgrade contract is clear.

For a broader view of routing models after retrieval, see our guide to NVIDIA NeMo Switchyard. If the project needs a wider agent architecture rather than only RAG tuning, our AI agent development service covers tool boundaries, evaluations, observability, and deployment.

Red Hat OpenShift AI AutoRAG FAQ

Is Red Hat OpenShift AI AutoRAG generally available?

No. Red Hat describes AutoRAG in OpenShift AI 3.5 as a Technology Preview. That makes it appropriate for controlled evaluation, but teams should verify support, upgrades, and API stability before making it a critical production dependency. Keep the chosen configuration portable outside the preview interface.

Does AutoRAG replace a vector database?

No. AutoRAG evaluates retrieval configurations, while a vector database stores and searches embeddings. Red Hat's example uses Milvus as a remote vector provider. You still own index lifecycle, capacity, backups, access boundaries, and the process for rebuilding embeddings when documents or models change.

Do we need a GPU to use AutoRAG?

The Red Hat demo uses a GPU for Qwen3-8B and CPU serving for embeddinggemma-300m. Red Hat says you can instead connect a preferred model provider endpoint. The practical requirement depends on where inference runs, data residency rules, expected evaluation volume, and the latency you can tolerate.

Start with the benchmark, not the cluster. If the held-out questions cannot tell two retrieval configurations apart, more automated search will only produce a more confident guess.