The pilot is not the hard part any more

Every credible tool can produce an impressive demonstration. Upload some documents, ask a question, get a cited answer — that capability is now table stakes, and evaluating on it tells you almost nothing. The failures happen months later, and they are always the same shape: the pilot worked, and then it could not be extended to a second team, or it could not be signed off, or nobody could explain what it did.

So run the evaluation upside down. Start at the bottom of the stack — where does it run, who can see what, what does it record — and only then look at the agents. Vendors demonstrate top-down because the top is the exciting part. Buy bottom-up.

The seven layers, and the question each one answers

1. Deployment and data residency

Decide this first, because it eliminates more candidates than anything else and no amount of feature quality compensates for failing it. Can the platform run inside your network, or in a region you specify? If self-hosting is required, is the on-premises build the same product as the hosted one or a lagging subset? What crosses the boundary at runtime — and specifically, where are document embeddings computed and stored?

Orckai deploys with Docker Compose on your own infrastructure or runs hosted, with the same feature set either way; the self-hosting guide covers the checklist in detail.

2. Tenancy, RBAC, and identity

The second team is where architecture shows. Ask how isolation is enforced — a filter the application is expected to remember, or a scope applied at the data layer? Can one team's knowledge base be invisible to another's agents by construction, not by configuration discipline? Are roles and permissions first-class objects you can inspect, or a fixed set of tiers?

Then ask about identity: SSO against your provider, and what happens to a user's access when they leave. A platform with its own separate user directory becomes a compliance problem at the first departure.

3. Audit and execution history

The test question is specific: an agent gave a customer a wrong answer three weeks ago. Show me what it was asked, what it retrieved, which tools it called with which arguments, what it replied, who owned the agent, and what it cost.

Platforms that can answer that have a real audit log — actor, action, IP address, request id — and retained per-step execution records. Platforms that cannot will show you an analytics dashboard instead. Notice the substitution when it happens.

4. Cost metering and hard limits

Model spend is the line item that surprises finance in month three. Ask whether usage is attributed per team or arrives as one invoice; whether limits are alerts or enforcement; and what happens when a workflow retries in a loop over ten thousand records at three in the morning.

An alert is not a control. Orckai meters per organization with a visible balance and stops cleanly at zero, and teams that prefer to use their own provider key can switch to direct billing at any point.

5. System access and tool permissions

This is where an AI project stops being a chatbot and starts being useful — and where it can go most wrong. The questions: how does an agent reach a database, and who decided what it may see? Is access read-only by default? Does the model ever hold a credential? Is the tenant filter applied in SQL, or supplied in a prompt the model could be argued out of?

Orckai generates MCP servers that publish specific tables or endpoints as named, typed, bounded tools, with credentials held server-side. The reasoning is set out in AI agents for databases.

6. Retrieval you can defend

Grounding matters less than provability. Can a user see which document a sentence came from? Does the retrieval respect permissions, so a document someone cannot open cannot be quoted at them? When a document is deleted or replaced, does the index follow — and how quickly?

7. The agents themselves

Last, not first. By this point the field is usually down to two or three, and the remaining questions are about fit: which models, whether non-engineers can safely make changes, how a change is reviewed and rolled back, and whether the platform can also run the scheduled and event-driven work that inevitably follows the first chat use case.

A requirements table you can paste into an RFP

Three mistakes worth avoiding

Buying for one use case. The first project is support chat, or document processing, or an internal assistant. The second and third arrive within a year, from departments who saw the first. A tool scoped to the first use case means buying again, and running two vendors, and explaining why.

Treating open source as a cost answer. The licence is one line of the total. Deployment, upgrades, security patching, capacity, and everything the project does not include are the rest, and they recur. Open source can absolutely be the right answer — just cost it honestly against a subscription rather than against zero.

Letting the pilot pick the platform. A pilot that succeeded on a prototyping tool creates enormous pressure to productionise it. That pressure is how organisations end up running unmonitored, single-tenant AI services on infrastructure nobody owns. Decide the platform on the seven layers, then pilot on it.

Where Orckai sits

Orckai is built for the case where an AI capability has to be shared across teams, reach real systems, and survive review. Organization-scoped data with RBAC, an audit log carrying actor and request id, per-organization usage metering with enforced limits, generated permissioned connections to PostgreSQL, MySQL, SQL Server, Oracle, MariaDB and REST APIs, a knowledge base with inline citations, an embeddable widget, a workflow engine for the work that follows, and deployment hosted or on your own infrastructure.

It is not the right answer for everyone — a single team with a novel reasoning problem is better served by a framework, and the field guide lays out where each category fits. But if the questions above are the ones your review will ask, this is the shape of platform that answers them.

Frequently Asked Questions