An Approach for Ex-Post-Facto Analysis of Knowledge Graph-Driven Chatbots – the DBpedia Chatbot

4 years ago by

Over recent years, chatbots have emerged in various sectors like healthcare, retail, banking, etc. and their application is only expected to increase in the future. A majority of these chatbots are powered by some kind of database - relational/non-relational database or knowledge graphs. In this work, we focus on knowledge graph-driven chatbots, which provide users access to information potentially collected from a wide variety of heterogeneous data.

In particular, we examine the chat logs of an existing system and present a generalizable approach to examine how users interact with knowledge-driven chatbots and their expectations with these agents. The objective is (i) to understand the nature of user-requests: query-patterns and user-intentions; (ii) examine whether the chatbot can serve its purpose and satisfy user-requests; (iii) get insights about the conversation flow to improve the chatbot’s architecture. Therefore, we propose three general analytical streams for investigating knowledge-driven chatbots:

1) Request Analysis- to analyze the intents and complexity within user-utterances and to examine whether the users conform to the limitations of the chatbot.

2) Response Analysis- to characterize the common errors made by the chatbot as well as the reasons behind it.

3) Conversation Analysis- to uncover common topics of conversation and inspect the use of anaphora in them.

We apply the proposed approach to the chat logs of the DBpedia Chatbot, which has been running since August 2017. The chatbot is not only capable of answering domain-agnostic factual questions (using DBpedia Knowledge Graph) but also domain-specific questions about DBpedia services and interaction within the community. This makes it an interesting case study. We believe that the findings from our analysis and the suggested solutions to enhance the user experience can benefit all those genres of conversational interfaces that either engage in customer-service or empower data-driven applications or both.

To know more, check out the slides that were presented in the CONVERSATIONS 2019 Workshop or the preprint of our paper. For the source code of our analysis of the DBpedia Chatbot, hop on to https://github.com/dice-group/DBpedia-Chatlog-Analysis.