The evaluation framework

If you are looking at AI tools for your property management business, here is how to evaluate honestly.

Question 1 — What is the specific decision this AI is making?

If the vendor cannot answer in one sentence, the AI is probably not really there.

Good answer: “Our voice agent decides whether to escalate an intake to emergency or normal routing, based on severity keywords + fixture type + weather.”

Bad answer: “Our AI optimizes your maintenance workflow.”

Question 2 — Can you see and override the AI’s decisions?

If the AI is a black box, you cannot fix it when it is wrong in your specific market. Walk away from black boxes. Good AI tools expose the decision trail, let you override, and learn from the override.

Question 3 — Does the AI improve with use?

Good AI tools capture training-grade data and use it to improve. Bad AI tools call the same generic LLM every time and never get better.

Ask: “How does this AI get smarter at my specific properties / vendors / tenants over time?”

Question 4 — What is the failure mode?

If the AI fails, what happens? Does the system fall back to a human? Does it silently produce a bad outcome? Is there an audit log that shows where the failure was?

Good AI tools have graceful failure modes. Bad ones fail silently.

Question 5 — Does the AI integrate with your existing systems

or require you to switch?

This is huge. Beware of “AI-native PMS replacement” pitches. Switching PMS is expensive, risky, and often a multi-year project. PMS-overlay AI tools that sit on top of what you already run are cheaper and lower-risk.

Question 6 — What is the data + privacy posture?

Where does your tenant data go? Where do voice recordings sit? Who trains on what? In 2026, this matters more than vendors realize and more than buyers think it does.

Question 7 — Can you see real evidence, and will they tell you what they haven’t built?

The obvious version of this question is “show me case studies with named customers.” Ask for those. But in a market this young, that test alone selects for the wrong thing — it rewards whoever has been selling longest, not whoever built the better system, and logos are the easiest thing on a website to arrange.

Ask for the evidence underneath instead:

• Can they play you a real call? Not a scripted demo — an actual recording, including one that went badly.
• How do they know a call went well? A vendor serious about quality scores its calls against written rubrics and can show you the scores. One that cannot has no idea what its failure rate is.
• What have they NOT built yet? This is the question that separates honest vendors from the rest. Everyone has gaps. A vendor who names theirs unprompted is telling you what their roadmap actually is; one who claims full coverage is either confused about their own product or willing to mislead you, and neither is who you want holding your owners’ money.

A vendor with three logos and no methodology is a worse bet than one with no logos who can show you call recordings, evaluation data, and a straight answer about their stage. Weight it accordingly — and apply the same test to us.

[Our position] So here is us applying our own question to ourselves. When our calling runtime changed in August 2026 we re-validated the affected voice workflows on live calls instead of carrying forward results earned against code that no longer ran — the scripts were untouched, but the code every call passes through was not. That is the behaviour Question 7 asks you to look for, applied to us. Ask any vendor what their equivalent rule is, and what it has cost them.

---