Choose an AI model for your SaaS
A support example shows how to compare tasks, allowed data, response quality and full cost before releasing an AI feature to customers.
The right model for SaaS handles a defined task within the product’s boundaries. A report explainer, ticket classifier and record-editing agent need different criteria. Choosing one “best model” before separating those functions hides what needs measuring.
Consider a hypothetical support product that drafts replies from a ticket and the organization’s authorized documentation. A staff member reads and sends the reply. The pilot aims to reduce writing work without inventing terms, exposing data or increasing verification effort.
Write the task contract
Define the input: current message, necessary history and documents the user may access. Define the output: draft, sources used and an indication of missing information. This first version cannot approve a refund, promise a deadline or send the message.
That boundary makes candidates comparable. Persuasive text inventing a returns policy fails. A response recognizing that information is missing and asking the right question may meet the task.
Prepare cases with expected outcomes
Google Cloud’s development guidance describes evaluation datasets with inputs and reference responses. For your product, document prohibited answers as well.
| Pilot case | Expected check |
|---|---|
| Question covered by a current policy | Faithful answer with the correct reference. |
| Missing documented detail | Identify the gap without inventing terms. |
| Question about another customer | No information outside authorized access. |
| Text attempting to change instructions | Treat it as ticket content, not authorization. |
| Conflicting document versions | Identify the inconsistency rather than guess a policy. |
| Failed or slow retrieval | A clear state without false success confirmation. |
Use test material without unnecessary personal data. Include short, long and incomplete messages in customers’ actual languages. Separate instruction-tuning cases from reserved evaluation cases, so improvement is not limited to familiar examples.
Keep authorization outside the model
The backend must restrict retrieval to allowed documents and records before creating context. OWASP’s agent guidance recommends explicit authorization for sensitive operations. A prompt instruction does not replace server-side permissions.
A capable model still does not need the whole database to answer one account’s question. Test isolation across customers and inspect returned references. If the answer cites a document the staff member cannot access, investigate the entire flow.
Compare the cost of a usable reply
Record duration, auxiliary calls, retries and review time. Divide batch cost by the number of replies accepted under the criteria. This is closer to delivered work than input-token pricing alone.
Look separately at difficult cases and long waits. A fast average can hide episodes that interrupt support. Background preparation can show its state; a response required during a live conversation has a different waiting tolerance.
Make the pilot interruptible
Start with reviewed drafts, record corrections and provide the manual workflow during failures. Define in advance which failures block expansion: data exposure, out-of-scope actions and invented commercial terms are examples.
An alternative provider must pass the same evaluation. Do not expand data sharing during an outage unless that route was already planned. Reevaluate when the model, retrieval, instructions or documentation changes.
The evaluation guide explains comparisons between runs. Product AI architecture organizes what surrounds the model. Your final choice may differ by task, provided the team can operate and verify each route.
About the author
Tiago F SantiagoComments
No comments yet
Share a question or an experience related to the article.
Related articles
