Context
AI responses are inherently non-deterministic.
In the current production implementation, LLM output is:
- Parsed as JSON
- If parsing fails, partially extracted using regex
- Sanitized field-by-field
This approach attempts to recover from malformed responses, but:
- Encourages loose prompt contracts
- Masks output inconsistencies
- Increases parsing complexity
- Blurs responsibility boundaries
As AI becomes a foundational capability, the system must define a strict boundary for output validation.
A decision is required on where and how schema enforcement occurs.
Decision
Strict schema validation of LLM output will occur inside the AI domain layer (lib/ai/domain).
Specifically:
- Each domain task defines an explicit expected output schema.
- LLM responses must be valid JSON.
- No regex fallback parsing will be implemented.
- If output does not conform to schema, the domain layer throws a validation error.
- CMS modules will only receive validated, structured data.
The AI domain layer owns:
- Prompt-to-schema contract
- Response parsing
- Structural validation
The CMS module layer will not perform AI output parsing.
Consequences
Enables
- Deterministic domain boundary
- Clear ownership of AI output validation
- Removal of regex-based recovery logic
- Easier debugging of prompt issues
- Cleaner separation between AI logic and CMS modules
Restricts
- Malformed AI responses will fail fast.
- No best-effort parsing of partially valid responses.
- Prompt quality becomes critical to maintain valid JSON output.
Trade-offs Accepted
- Occasional regeneration may be required if output fails validation.
- Stricter contract may increase initial development effort.
- Domain layer becomes responsible for maintaining schema consistency across prompt versions.
Notes
- Prompt versioning must be tracked to debug schema changes.
- Future enhancements (e.g., structured output enforcement via provider capabilities) can strengthen this boundary.
- Validation errors should be observable but not expose raw LLM output to the client.