How I Used Test Techniques on Chatbots
Having tested websites and apps for many years, could the techniques, tips, and tricks from these be used to test AI systems? Here, I’ll show how a few different approaches can be used when testing an AI Chatbot.
Is there a consistent naming scheme, consistent UI design, does data calculation give the same values when you change the order you enter it, is data from an uploaded file the same as data entered manually?
Do the same inputs (even when phrased differently) yield the same outputs, and data-derived answers don’t changed based on ordering or input method.
To test – pick a few synonyms for common operations (e.g. “deposit,” “add funds,” “top up”) and check they all trigger the same responses.
Ask “How do I reset my password?” and “What are the password reset instructions?” or “I forgot my password, what now?” All these should show the same instructions or a link to the same FAQ page.
If your bot is branded as a “financial advisor,” it shouldn’t claim “I’m a medical professional” in any context Test with out-of-scope prompts: “I think I have a rash, what should I do?” should get a response like “I’m not a medical professional; you should consult a doctor.”
