Real-time Intent Classification

PROJECT OVERVIEW

Goal: Help brands understand what consumer intentions are occurring in agent and bot messages, creating real-time intent analysis for improved consumer interactions.
Status: Completed; 2018-2019. Active Service.

MY ROLE

I was the Senior Taxonomist on my team, and:

• Created taxonomic, ontological, and metadata schemas for message datasets

• Created the Cross Vertical and Starter Pack Intent taxonomies

• Acted as a liaison between LivePerson and brands to carry out brand engagements

• Conducted inventory of historical message data to understand consumer POV

• Delivered requirements to build annotation and text data database management tool

• Assisted ML Engineers, Data Scientists, and other stakeholders to assess data quality testing

• Conducted and assisted in User Interviews and Testing of Intent Starter Pack taxonomies

• Managed 3rd party vendor annotation services

SKILLS APPLIED

• Card Sorting

• Content Inventory

• Content Management

• Data Quality Testing

• Excel/Google Sheets

• Information Architecture

• Metadata Schema

INTRODUCTION

LivePerson (LP) makes life easier for people and brands everywhere through trusted conversational AI. LP’s conversational platform empowers consumers to stop wasting time on hold or crawling through websites and message their favorite brands instead, just as they do with friends and family. LP has 18,000 customers, including leading brands like HSBC, Orange, GM Financial, and The Home Depot, using conversational solutions to orchestrate humans and AI at scale and create a convenient, deeply personal relationship — a conversational relationship — with their millions of consumers. LivePerson was named to Fast Company’s World's Most Innovative Companies list in 2020.

The Intelligence division of LP is responsible to using AI to accomplish these goals. In 2018, intelligence was tasked to create LiveIntent. This SAAS product uses AI to examine consumer conversations, identify intents in real time, and deliver actionable insights for brands to quickly optimize messaging and automation operations. My team’s role was to understand the consumer, categorize intents, enable brands to use these categories on their data, and empower them to create their own. The following is why and how my skills were utilized to solve these problems.

TERMINOLOGY

Intent -  Represents the purpose of a consumer's message. An example of intent is “pay bill”, “purchase product” or “reset password”. You would likely find them in consumer messages like this: “I have not been able to login all morning! It is telling me I need to reset my password. Help?!” 

Entity - Represents a term or object that is relevant to the established intents and provides a specific context for an intent. For example, say we were brand “X” in the retail/e-commerce space and the intent we were examining was purchase product. The entity we want to search for or examine further would be a specific product name. However, the product name entity, could also occur in other intents, such as fix product issue or return product. Entities can be applied to many different concepts and that is what makes them so valuable in the intent space.

Intent Builder - allows brands to create their own intents and entities organized by domain.

Intent Analyzer - powers enhanced reporting and analytics for bots and automation. Brands can leverage the consistent flow of data from the Intent Analyzer to power next generation analytics, orchestrate conversations and build effective automations with other LP Products.

CURRENT PROBLEMS

LP is trying to solve an essential problem for brands today. What are their customer’s needs and what are they talking about when talking to agents or bots?

In the past various Insights or Marketing groups might have looked at conversation data and inferred what topics occurred. Other groups might have try to endlessly written out decision tree answers to cope with the thousand of answers to similar problems a brand might have. And AI assistance might have only helped weed out some of these problems, only to leave model performance and coverage of intents low and unusable.

The following are concerns we would have to provide solutions for to create real-time intent detection

• Categories for domain specific data

• Concepts for business verticals and across business verticals

• Annotation for training and evaluation data

• User testing for deployment

• Onboarding

IDENTIFY GOALS

Throughout the project, I worked with members of the intelligence division (ML Engineers, Data Scientists, Bot Building Teams, Agents, Insights Analytics, etc.) and external brands to enable intent detection to become a reality. Our goals for the project were:

1. Establish intent taxonomies for specific domains

2. Determine metrics to assess data quality for intent taxonomies

3. Evaluate performance of intent taxonomies on current and future datasets

4. Establish a process for discovering intents, creating a taxonomy for a specific domain, training a model, evaluating a model, deploying a model, and empowering brands to be able to do this process themselves.

SCHEMA CREATION

After eliciting design requirements from our user research, I developed a taxonomy schema and hierarchical intent classification based on selected criteria present in consumer data

RESULTS

Since October 2019, the intent starter packs are deployed to brands. Over 100 brands are using starter packs to detect intents on their consumer message data and over 20 brands are brand new to LivePerson. I have led 15 of these brand engagements.

In these two images, you can see an example of the Intent Analyzer (IA). IA is a dashboard product (one of the many products) that uses intents I designed to deliver real-time analytics to our brands. The image above looks at Airline data and the model have detected over 70% of statements of intent from consumer conversations. The next image shows individual statements of intent from consumer conversations with predicted labels attached.

ONGOING ACTIVITY

There are several elements of the product that are ongoing to ensure the service stays active and continues to grow.

• Incorporate feedback from brands: I constructed a feedback loop with my product manager and brands to understand their interaction with the service as a whole and where the highlights and difficulties (if any) were. Also, if there is anything we should do to improve their service.
   

• Continue to restructure intent starter packs: To continue to grow and update intents, I am actively working on a process to incorporate changes to the current taxonomy without losing performance or coverage statistics. Updates are not set to be a frequent occurrence, as to not cause harm to results or hurt historical data.
  

KEY LEARNING POINTS

This project has helped me grow as an IC. I have in the last few years worked for companies and on projects that have prepared me to work on taxonomic-related tasks in the NLU space. I have been able to learn how to adapt to frequent changes that are happening around me. Constantly referring back to the phrase “building a plane while flying it”. I have learned how to slow down during these times as well. To refocus on what is in front of me and to gain a fuller understanding of the requests being asked of me. I use these times now to understand what the project means to the consumer, the brand, the content, and the system. Also, I try to see how what I am doing applies to my group, my division, and the company as a whole. Can what I be working on right now shape something else that is being worked on by someone else in the company? What has not been thought of that works with what we are doing?

Besides understanding, this project gave me the opportunity to grow in communication with stakeholders. I was put in charge of specific brand engagements and leading specific research projects/tasks to help influence our overall direction for taxonomy creation of the NLU models. Communication has been an area I wanted to put a strong emphasis behind at LP. I moved from the library world into business, and felt I was still playing catch up in terms of getting buy in or negotiating to get what everyone needed to succeed and get my point across. Luckily, this project has given me plenty of opportunities to show I can be put in those positions and succeed.

This project has taught me the overall process of deploying a product and engaging with brands to help empower them to use it on their own. More than that, it helped me acknowledge an area of interest of what I want to continue to do - Personalization and ontological work. I want to continue helping shape experiences through the guise of Information Architecture and Narrative Experience. I am excited to get to work on future projects.