UC Santa Cruz

Research on open-domain social dialogue systems has exploded over the last few years with the advent of large language models (LLMs) that can chat about any topic. Unlike traditional dialogue systems, open-domain dialogue systems cannot assume any specific information need or domain restrictions - the only inherent goal is to converse socially. While modern systems have access to more information and better tools, foundational components of natural human-human conversation remain elusive, i.e., intimacy and agency. In this thesis, we hypothesize that personalization is pivotal in fostering this genuine connection between users and open-domain dialogue systems.

Our first hypothesis is that personalizing the conversation to specific user interests will build a sense of understanding, rapport, and agency. To investigate this, we heuristically combine the results of an extensive natural language understanding pipeline with handcrafted rules to build a user modeling mechanism; this user model then personalizes the experience through response adaptation and topic-promotion strategies, resulting in a statistically significant positive impact on perceived conversation quality and length when evaluated at scale with a testbed open-domain dialogue system, that real Amazon Echo users access. Analyzing the user models unveils nuanced insights into user preferences, emphasizing a desire for more personalized experiences and receptiveness toward personal questions. This leads to our second hypothesis - asking appropriate personalized follow-up questions (PQs) helps to create a more engaged user experience that increases user satisfaction. Our initial test of this hypothesis uses a crowdsourced corpus of PQs (Would You Rather and Hypothetical) in the testbed system's dialogue policy. Our evaluation of the policy shows that it results in extended topical depth, leading to statistically significant longer, more highly rated conversations.

However, crowdsourcing PQs for every user interest does not scale. Question Generation tasks generally focus on factual questions from textual excerpts. Instead, we create a specialized training dataset of PQs more suitable for the novel task of Personal Question Generation. We first identify over 400 common user interests by sampling ~39,000 user models collected during user interactions with our testbed system. Then, we translate these into prompts and use the LLM GPT-3.5 to generate ~19,000 PQs and associated system answers. Evaluating the impact of this pre-generated data when used in our testbed system's dialogue policy results in statistically significant positive effects on perceived conversation quality. Statistically significant results also suggest that deep, user-centric PQs are the most effective means of increasing intimacy and engagement.

We then use the corpus of ~19,000 PQs to fine-tune a RedPajama 3B prompt-based PQ generator, which further shows the positive impact of producing highly tailored questions when evaluated in our testbed system. To evaluate our hypothesis independently from our testbed system, we synthetically generate a corpus of 2,000 long synthetic social dialogues that strongly aim to resemble real user conversations. We use these social dialogues to compare our fine-tuned PQ generator against 5 other state-of-the-art LLMs. Positive results affirm the importance of PQs in social conversation while also validating our model as a strong baseline for the task of Personalized Question Generation.