Prompt engineering for business: Beyond the chatbot demo

Anyone can write a chatbot demo in 20 minutes. The hard part is making the prompt survive a customer who asks the same question three different ways, a payment that fails mid-conversation, or a model update that quietly shifts behaviour overnight. Production prompt engineering for business is closer to writing defensive code than writing clever copy. Here's how we approach it at Apex-OS.

Version every prompt. Production prompts belong in source control alongside your application code — tagged, reviewed, and deployed through the same CI/CD pipeline. We store prompts as versioned templates in a prompts/ directory, with semantic versioning (1.0.0 → 1.1.0 for additive changes, 2.0.0 for breaking). Each prompt has a CHANGELOG entry explaining why it changed. This makes rollback trivial — when a new prompt causes regressions, we revert to 1.4.2 in under a minute.

Force schema-validated outputs. Free-form text responses are a liability. Every transactional AI call we make returns a structured JSON object that's validated against a Zod schema before it touches business logic. The schema enforces types, enums, and required fields; any validation failure routes to a fallback handler or human review. This pattern alone has eliminated hallucinated order numbers, fake customer IDs, and the entire class of "the AI said something we can't act on" bugs.

Build an eval suite before scaling. An eval suite is a fixed set of test cases — typical inputs, edge cases, and adversarial prompts — that you run against every prompt version before deployment. Our standard suite has 50-200 cases per prompt, drawn from real customer interactions (anonymised). Each case has an expected behaviour; pass rate must stay above the previous version's baseline before we ship. This catches the silent regressions that human spot-checks always miss — like a prompt that handles 90% of cases perfectly but breaks on plurals.

Monitor and log everything. In production, every AI call gets a structured log: prompt version, model, input (truncated for PII), output, latency, confidence, and the rule path taken. When a customer disputes an AI-generated invoice, we can replay exactly what happened and why. This is non-negotiable for compliance — the moment an AI decision touches money, you need an audit trail. We send a daily summary of AI decisions flagged for human review to the ops Slack; usually 2-3 a day, all minor.