Most building societies won't commit to an AI tool on the strength of a demo, and they shouldn't. The sensible next step is a pilot: a short, real trial on the society's own material, with a clear decision at the end.
The risk is that pilots drift. They start without agreed measures, run longer than planned, and end with a vague “let's keep going” that helps nobody. Here is how I think a good four-week pilot should run instead.
Before week 1: agree the scope and the success criteria
This is the most important step, and it happens before anything is built.
- Scope. One clear job, such as answering broker questions about residential lending criteria. Not five jobs at once.
- Success criteria. How you will judge the result, agreed and written down before anyone sees an answer.
- People. A named sponsor, an underwriter who can give some time to testing, and a small group of users.
Illustrative success criteria might include:
- The share of test answers your underwriters judge correct against the lending criteria
- How reliably the tool says it doesn't know when the answer isn't in your documents
- Whether users find it quicker than searching the PDF or picking up the phone
The exact measures and thresholds are for each society to set. What matters is that they are agreed up front.
Week 1: set it up on your own documents
The tool is set up on your own published criteria, for the one job you agreed. At the same time, your team writes down a list of real questions they get asked. Real questions are far more revealing than invented ones.
Week 2: underwriters test it
Your underwriters work through the question list and check every answer against the lending criteria. This is where you find out whether the tool understands your criteria, including the awkward parts: conditional rules, cross-references and thresholds that change by product or loan-to-value.
Week 3: a small group uses it for real
A small group of staff uses it for real questions as they come in. Every answer is still checked by a person. This shows how the tool copes with the questions people actually ask, worded the way they actually ask them.
Week 4: review and decide
Look back against the success criteria you agreed at the start:
- What did it get right?
- What did it get wrong, and why?
- When it couldn't find an answer, did it say so?
Then make the decision: roll out, extend with a clear reason, or stop. Because the measures were agreed before anyone saw results, the decision is about evidence, not enthusiasm.
What makes the difference
- One job, not five
- Your own material, not a vendor's demo
- People checking the answers throughout
- Success criteria agreed before anyone sees results
- A clear decision at the end, not a pilot that drifts on
A pilot that ends with a clear answer is worth more to everyone than one that never quite finishes. It's also how we intend to run pilots at AdAstr.
If you're thinking about how to test AI at your society, I'd be glad to talk it through: hello@adastr.co.uk