Measuring Chatbot Performance

  • November 27, 2017

    Measuring Chatbot Performance

    As consumers, our initial interactions with many companies will be through chatbots. As the world of commerce migrated to the internet, a need for customer service online spurred the initial demand for chatbots. While many people view chatbots as rudimentary or even annoying, companies are investing heavily in improving the technology.

    Both B2B and B2C bots are on the rise. According to Business Insider, investments in the chatbots industry grew 22% from 2015 to 2016. As chatbots become more ubiquitous, improving quality over proliferation is becoming the main priority for many companies focused on making chatbots work for them. The challenge on measuring performance is mostly two-fold. Companies need to measure both the user experience and the quality of information generated by chatbots.

    Chatbase by Google

    Google recently launched an analytics dashboard for measuring chatbot performance. The program, coined Chatbase, measures key vitals for chatbots. These include average session time (or how long a user interacted with a chatbots), daily sessions per user and user retention. While Chatbase provides some baseline measurements for chatbot operators, it’s important to truly understand how your chatbots operates and to establish your own KPIs to maximize your ROI.

    Chatbot Performance Indicators

    VentureBeat outlines some important KPIs that chatbots should be measured by. The first, and probably most obvious, is revenue. Since implementing chatbots in on your website or in your operations, has revenue increased? If so, can these increases in revenue be attributed to chatbot usage?

    Following revenue, do your chatbots help your users complete tasks without human interaction? VentureBeat labels this KPI as the self-service rate. The third KPI is activation rate. When prompted with a message from a chatbot, do your users respond or ignore the bot? If they interact with the chatbot, how long do they spend chatting? This is known as the retention rate.

    The outcomes on revenue, self-service rate, activation rate and retention all depend on the quality of the chatbot user experience. High quality AI will allow chatbots to avoid confusion triggers and provide useful answers for users in a natural, conversational way. Measuring the underlying performance of the AI supporting your chatbot will drive overall better results.