Customer satisfaction is an essential metric for every company, no matter if you sell directly to customers or businesses. Customer Support is fast becoming the voice of the customer inside the company far away from the old view, that of mainly a cost center. Thanks to feedback from customers we can get reliable data to make informed decisions on customer experience and product.

As such Customer Support teams measure performance by metrics like the First Contact Resolution Time.

Measuring customer satisfaction is becoming one of the main KPIs that competitive customer support departments are pursuing to measure and understand.

I remember when we started, we used to discuss how awesome would be to get that feedback — just waiting for people to click and the numbers to come. We thought that measuring customer satisfaction to such a simple question would lead to almost every customer providing an answer. So one day, out of the blue — it happened.

We got a 90% response rate!

I am just kidding there. Anyone who has tried to implement NPS in a realistic environment knows that this is far from true.

However, how can we go further than that? Let’s dive in some results from our adventure on the matter.

A primer on Net Promoter Score (NPS)

Customer satisfaction surveys come in a few common forms, but we chose to use Net Promoter Score® (NPS) as it is popular in SaaS, simple to start with and powerful enough to see results.

Net Promoter Score is a measure that is used by many companies for calculating and understanding customer satisfaction. The concept behind it is pretty simple, you ask a simple question to the customer – “How likely are you to recommend us to a friend or colleague?” and you use a scale for the response.

Based on that response, customers are assigned one of the following labels:

1. Detractors
2. Passives
3. Promoters

Many people will either forget or avoid responding to your emails, leaving you with gaps in your data and wondering on how to convince them to give you a response.

We quickly learned that trying to convince every customer to respond, is futile. So, we tried to figure out a way to understand our customers’ feelings about our product and company and complement the results we get from the NPS. To do that, we turned into Sentiment Analysis and how it could help us to infer the feeling of our customers when they didn’t respond to our NPS questions.

Customer Sentiment Analysis for NPS?

Sentiment Analysis might sound like something exotic which includes complex models, generated by a data scientist. In reality, you can come up with a Sentiment of a piece of text without involving such complex methodologies.

In this article, we demonstrate how we experiment with calculating the sentiment of customer support related data.

Below, we use only a relational database and a dataset that is already prepared for this purpose. We got inspired to use this methodology from a great article on Periscope Data blog, on doing text analysis on Donald Trump’s tweets.

Sentiment Analysis for NPS and why it is not as hard as it sounds

Natural Language Processing (NLP) is the hottest and most active sub-fields of Artificial Intelligence. NLP-related applications aim to make the computer better understand what humans intend or imply by analyzing spoken or written words.

Sentiment Analysis focuses on text, and it has gained much traction with the current state of the art models achieving accuracy score over 95%.

Under these circumstances, data scientists and machine learning engineers are often tempted to jump directly into complex architectures and sophisticated models for sentiment analysis of their company’s support systems. Such an initiative, requires a considerable amount of data, time and computational resources. Right?

What we did is to start the implementation of a first approach that would generate some useful insights and can produce value very quickly. After all, something is always better than nothing.

In our case, we implemented a bag-of-words (BoW) model. A BoW includes the tokenization of the initial textual data from a customer support platform into unigrams and the assignment of a sentiment score to each word separately.

Finally, by performing some aggregation, e.g. average, one can generate the score for a whole text. For those already wondering how do we know the suitable sentiment score for each word, the answer is easy: various word databases, subproducts of research projects in this area, exist that assign scores to opinion or sentiment words for the English language.

Analyzing Sentiment of NPS data on a Relational DB

As discussed prior, what we need to perform our first NPS sentiment analysis in the context of customer support, is customer conversations and a suitable lexicon, i.e., a word database with sentiment scores assigned to each word included. Our conversations (textual data) are from actual tickets of customers and users in our customer support application with is Zendesk. Of course, you can use data that come from any SaaS tool like Freshdesk or Delighted and more.

We must load all these data into a relational database where the rest of the analysis is going to take place. We chose to use PostgreSQL as, like most of the modern databases, offer many functions that facilitate the preprocessing required. For the simple model, we will work, only a standard knowledge of SQL is sufficient.

Let’s go through the steps and requirements we need to get started:

• Choose a suitable and relatively complete lexicon and load data into PostgreSQL.
• Choose textual data to work with, e.g., ticket subject or body, and load data into PostgreSQL.
• Clean of textual data, e.g., removal of special characters & application of standard NLP procedures.
• Tokenization, i.e., split of text fragments into words.
• Combine our support data with the lexicon.
• Calculate suitable aggregations.
• Visualize.
• Lessons Learned and Next Steps.

Choose a Lexicon and Load Data Into PostgreSQL

Regarding the lexicon choice there quite a few popular options we can choose. At this point, we may need to take into consideration, how these lexicons were built, i.e., using what kind of data. That is particularly crucial since depending on each specific occasion people tend to communicate a bit differently.

For example, some written messages, like emails, are probably more formal, and thus more polite, compared to tweets. Additionally, emails include typical greetings or signatures which usually include words with positive sentiment although they do not always reflect the attitude of the sender towards the receiver.

Alternatively, we can attempt to refine the initial text in a way that removes all the formalities and keeps only the words that reveal the predisposition of its author.

Among the most commonly used lexicons, the following included:

SentiWordNet:  a lexical resource containing 117640 for opinion mining that assigns to each synset of WordNet three sentiment scores: positivity, negativity, and objectivity. The development of the resource is based on the quantitative analysis of the glosses associated to synsets, and on the use of the resulting vectoral term representations for semi-supervised synset classification.

SentiWords: a high coverage resource containing roughly 155.000 English words associated with a sentiment score included between -1 and 1. Words in this resource are in the form lemma#PoS and align with WordNet lists (that include adjectives, nouns, verbs, and adverbs). Scores are learned from SentiWordNet and represent state-of-the-art computation of words’ prior polarities (i.e., polarity for non-disambiguated words) using SWN.

AFINN:  a list of 2477 English words rated for valence with an integer between “minus five” (negative) and “plus five” (positive). Finn Årup Nielsen manually labeled the words in 2009-2011.

SenticNet: provides polarity associated with 50,000 natural language concepts. A polarity is a floating number between -1 and +1. Minus one is extreme negativity, and ”plus one” is extreme positivity.

For our use case we the SentiWordNet lexicon, which is one of the largest available and we thought it would be a better fit with our data from Zendesk.

How we loaded the lexicon data into PostgreSQL

We exported the SentiWordNet and used it as a CSV file. As this is not a file that easily changes we can upload it in our PostgreSQL. Of course, you can code around it but if you want an effortless way you can do what we did.

We uploaded the CSV in Google Sheets and used Blendo to load the Google Sheet to PostgreSQL. That synced the file and created a table into PostgreSQL with all the necessary columns in minutes.

We chose Google Sheets as even if we need to change something in the file, the background sync with Blendo makes the change appear in your PostgreSQL without you working on the code.

Choose Textual Data and Load Data Into PostgreSQL

As we discussed earlier, we chose to work with our Zendesk customer support data. Again to load Zendesk data into PostgreSQL, we used Blendo. Other tools like Freshdesk or Enchant would be equally valuable and straightforward to sync.

How we loaded the Zendesk data into PostgreSQL

You only need to click the Zendesk connector and Blendo will a) connect to Zendesk, b) sync and prepare all your data automatically to PostgreSQL c) your data are ready to use right away.

Working with Zendesk textual data

After having Blendo to sync your Zendesk data here is how we worked with them. You can choose to work either with the subject or with the body of the tickets. Since ticket subjects tend to be more compact and to the point we preferred to work with these. Additionally, the ticket body tends to be more noisy, including greetings, signatures, and URLs that would require a lot of data cleaning.

Also, instead of assigning a sentiment score by taking into consideration both positive and negative words we are going to focus only on measuring the negativity of the messages. After all, what is usually essential in the context of customer support, is the identification of the frustrated or angry customers to take appropriate action.

How to prepare and clean your customer support data for Sentiment Analysis

Having decided to work with tickets subject, now it’s time to focus on preprocessing. As a first step, we are going to remove characters that are not letters and convert to lowercase. At this point is important to note that we also need to keep the id of the ticket, the id of the requester, or both to be able to aggregate results later based on these.

Upon completion of this step the dataset could look like this:

Id	requester_id	created_at	subject
933	370612642232	2018-11-14 10:44:17.000000	i am getting remote server error
10	114319404751	2017-07-19 14:03:22.000000	questions on stripe data

The necessary SQL code for this job is the following:

select id,
       requester_id,
       created_at,
       regexp_replace(regexp_replace(lower(raw_subject), '[^a-zA-Z ]', '', 'g'), '( ){1,}', ' ') as body
from zendesk_tickets

Tokenize

As a next step, we need to tokenize, stem the words and remove stop words. In PostgreSQL, this can be done using the to_tsvector and unnest functions, and the desired result would look like this:

Id	requester_id	created_at	subject
933	370612642232	2018-11-14 10:44:17.000000	get
933	370612642232	2018-11-14 10:44:17.000000	remote
933	370612642232	2018-11-14 10:44:17.000000	server
933	370612642232	2018-11-14 10:44:17.000000	error
10	114319404751	2017-07-19 14:03:22.000000	question
10	114319404751	2017-07-19 14:03:22.000000	stripe
10	114319404751	2017-07-19 14:03:22.000000	data

The SQL code used for this is an expansion of the query we used earlier:

with tickets_clean as (
  select id,
         requester_id,
         created_at,
         regexp_replace(regexp_replace(lower(raw_subject), '[^a-zA-Z ]', '', 'g'), '( ){1,}', ' ') as body
  from zendesk_tickets
)

select *,
       replace(unnest(string_to_array(concat(strip(to_tsvector('english', body))), ' ')), '''', '') as word
from tickets_clean

We follow a similar process with the SentiWordNet table. Below is the initial format as downloaded. After tokenization and stemming the table is ready to get joined with the one produced from the Zendesk tickets.

Pos	id	posscore	negscore	synsettermtext	gloss
a	5107	0.5	0	uncut#7 full-length#2	complete; “the full-length play”
a	196233	0	0.25	petrifying#1	paralyzing with terror

word	score
abstin	-0.625
acid	-0.375

The necessary SQL code is the following:

select replace(unnest(string_to_array(
    concat(strip(to_tsvector('english', split_part(unnest(string_to_array(synsettermstext, ' ')), '#', 1)))), ' ')),
               '''', '') as word,
       -negscore         as score
from sentiwordnet

By combining all the previously presented queries and by extending it appropriately so that the generated tables get joined together the final result looks like this:

raw_subject	id	requester_id	avg_sentiment
More databases	858	365993235031	0
Redshift WLM for Blendo	785	114339332791	0
I am getting: Error something went wrong	953	370564069212	-0.2461833707089874
Having trouble connecting to SQL Sever 2016	163	115857476632	-0.2220638401559454

So, now we have calculated a Sentiment but what can we do with it?

The previously generated data can provide any company with precious information, provided that the can be easily accessed and analyzed.

The following graphs are part of a Sentiment Analysis Dashboard constructed using Periscope. This dashboard enables the discovery of product appreciation and customer satisfaction.

By monitoring the average sentiment month over month, brand managers can keep track of how their customers feel about their product and whether the product development is moving in the right direction.

We can obtain a more granular view by taking into consideration the tickets’ type as well.

As expected tickets marked as a problem generally tend to reflect a more negative sentiment compared to questions, for example.

It can also be beneficial to focus on the worst detractors, i.e., those people who got the angriest or frustrated by your company’s products and try to understand the reasons behind this specifically. Is it the quality of the customer support, the product’s stability or the lack of documentation?

Also, we can draw useful insights if we take into consideration the number of opened tickets. For example, it seems that the dissatisfied customers give up your products or services quite easily and tend not to open numerous support tickets. On the other hand, those who communicate with the support the most are relatively much more satisfied.

Lastly, to gain some insight regarding the most prominent words in your customers’ ticket subject you can use a word cloud which aims to visualize the word frequency in the text as a weighted list.

Lessons learned and next steps on incorporating Sentiment Analysis in our Customer Support workflow.

Hopefully, the article so far convinced you that implementing sentiment analysis as a complement to your NPS campaigns, is something that can be done quite easily and give fast results.

There are some lessons we learned doing this ourselves that are worth considering if you decide to implement something similar yourself.

First, you can implement directly the simple approach we described in the article in your analytics database. That is great because you don’t need any complex infrastructure to rely upon for your sentiment analysis. Not only it’s easy to implement it, but it can also provide decent results fast.

As you progress with using sentiment analysis to understand the behavior of your customers, you might start considering more advanced methods to implement.

The reason is that the methodology we described has some weaknesses that you should be aware — the most important of them being the omission of the word order which can lead to erroneous results. For example, the phrases “This is nice” and “Is this nice?” will be mapped to the same sentiment score while they shouldn’t.

Finally, the most critical component of the approach described for sentiment analysis is the lexicon used. The best approach would be to build a lexicon specifically from a corpus of customer support tickets and even better from your interaction with your customers.

The best way to do that is to use the NPS scores to tag your tickets and customer responses with an appropriate sentiment and then use that for the sentiment analysis algorithm. By updating your lexicon frequently as you gather new data, you can improve your results and get something closer to what your customers feel when they interact with your company and product.

Want more?

Learn about how to track your Customer Success Metrics can change the customer experience.

See how you can analyze Conversations from Intercom to generate customer support insights.

Net Promoter, Net Promoter Score, NPS, and the NPS-related emoticons are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.