Product, software, and practical AI

What an AI-built prototype can show—and what remains untested

A convincing demo can provide evidence that a workflow is useful. By itself, it does not establish security, maintainability or operational readiness.

Edo Ceder23 July 20266 min read

An AI coding agent can produce an experience that looks remarkably complete. The application may have polished screens, a database, authentication, integrations, and a conversational assistant. A user can click through it and reach a useful result.

That is real progress. It deserves neither dismissal nor blind confidence.

The mistake is treating “it works” as one undivided claim. A tested prototype can provide evidence that an idea may create value while leaving important questions about reliability, safety, ownership, and operations unanswered.

The right response is not to diminish the prototype. It is to be precise about the evidence it has created.

What a tested prototype can show

Depending on how it was tested, a prototype may provide evidence that:

The idea can take a concrete form. People can see its information, actions, sequence, and result; ambiguities become visible on screen.
A tested workflow is understandable under the conditions observed. Task success and confusion show where terminology, hierarchy and decisions work or fail.
The result may create value. Completing the flow, returning, providing data, or involving someone else is more useful evidence than general enthusiasm.
Business rules are emerging. Real use exposes approvals, missing information, conflicting responsibilities, unusual states, and workarounds.
Product questions are more concrete. Screens, prompts, sample data, observations and decisions give the next team material to inspect rather than only an abstract brief.

These are substantial achievements. None of them, however, automatically answers the production questions below.

A demo does not prove security

Authentication is not the same as authorization. A prototype may allow users to sign in while failing to enforce which records or actions each person is permitted to access.

Inspect whether permissions are enforced at the server and data layers, request identifiers can expose another user’s records, administrative actions are separated, secrets remain outside the code, files are restricted, and sensitive information reaches model or logging providers.

Security depends on the actual implementation and the sensitivity of the product—not on whether the interface appears private.

A successful flow does not prove data integrity

A demo may cover only one happy path. A business system must also preserve correct state when users, networks and providers behave differently.

Test double clicks, concurrent edits, provider timeouts after an action completes, partial failure, invalid agent proposals, duplicate webhooks, and interrupted browser sessions.

Production systems need explicit rules for state changes, validation, idempotency where appropriate, and a way to reconcile partial failures. A safer boundary is to let the model interpret and propose while policy and execution code independently validate scope, permissions, current state and approvals before any critical change.

One successful integration does not prove reliability

External services can time out, reject requests and enforce rate limits. Model calls can fail, be incomplete or refused, and even schema-conforming output can still be wrong. Some integrations—payment webhooks are a clear example—are asynchronous and may be retried, duplicated or delivered out of order.

Reliability requires explicit decisions about timeouts, bounded retries, idempotency and duplicate handling, fallback behavior and, where appropriate, queues, visible status, alerts and recovery.

The objective is not to prevent every failure. It is to make failures contained, visible, understandable, and recoverable.

Generated code does not prove maintainability

The question is not whether AI wrote the code. Human-written code can also be fragile, repetitive, or poorly understood.

The practical questions are whether the structure is understandable, rules have one clear home, builds and deployments are reproducible, dependencies are deliberate, changes remain contained, critical behavior is tested, and unusual decisions are documented.

During review, look for duplicated patterns and competing implementations. Their presence—not whether a human or AI wrote them—can create maintenance risk.

A successful small-scale demo does not establish cost or capacity

A small pilot does not establish how latency, provider limits, support workload or cost will behave at higher volume.

AI-native products can involve model tokens, retrieval, media processing, browser sessions, email searches, vector storage, or several agent steps for one user request. Measure the full workflow rather than only the final model call.

Measure cost per completed task, repeated calls, limits on loops and retries, concurrent use, and which steps can be cached, batched, queued, made deterministic, or routed through a lower-cost path.

Scale is not merely a database benchmark. It includes provider limits, operational workload, support, and the cost of correcting failed automation.

A working application does not prove it can be operated

Once users depend on a product, someone must know which version is live, whether deployment succeeded, when background work fails, how to trace important actions safely, whether recovery is possible, who owns every account, and what happens when the original builder is unavailable.

Operational readiness includes observability, ownership, support procedures, access control, and a repeatable release process. These elements may be intentionally minimal at first, but they cannot remain accidental.

Readiness is proportional to consequence

Not every prototype needs enterprise infrastructure. The level of engineering should match what the product is being asked to carry.

Consider three dimensions:

Impact of an error. Is the result a disposable draft, or can it affect money, health, access, employment, or a customer commitment?
Sensitivity of the data. Is the system handling public information, internal operations, or personal and regulated data?
Difficulty of recovery. Can a user simply try again, or would a failure create an irreversible or hard-to-detect state?

A low-consequence internal prototype may be ready for a controlled pilot with a short checklist. A product that changes critical records or handles sensitive data needs stronger boundaries, tests, review, and operational controls.

The goal is proportional engineering—not maximum ceremony.

Turn the gaps into a staged plan

A readiness review should not end with a frightening list. A practical risk-based sequence is:

Before any external pilot

Clarify ownership, remove exposed secrets, protect user access, create safe test data, make important limitations explicit, and ensure there is a person responsible for support.

Before relying on the product operationally

Protect critical state changes, add repeatable tests for central flows, make failures visible, establish backups and recovery appropriate to the data, and document the release process.

Before expanding use

Measure performance and cost, review permission boundaries, reduce fragile manual intervention, strengthen monitoring, and make sure the team can maintain the product without depending on one person’s memory.

Some changes will require code. Others require clearer product rules, reduced scope, different operational procedures, or a human approval step. Production readiness is a property of the whole system, not only the repository.

Respect the prototype by evaluating it honestly

A tested AI-built prototype can provide evidence about whether users understand the problem, can complete the workflow and care about the result. It can also reveal business rules that were absent from the original brief and give a future team a better starting point.

It does not automatically prove security, correctness under failure, maintainability, affordable operation, or safe scale. Those are separate claims and should be tested separately.

That distinction protects both the business and the value already created. Instead of arguing about whether AI-generated code is “real software,” inspect the actual product, match the engineering to the consequences, and decide what must be true for the next stage.