First time right with NLP: zero shot classification

The availability of pretrained language models has increased tremendously in recent years. With the use of the amazing 🤗 Huggingface infrastructure many have finetuned models for downstream classification tasks. The results are often amazing, the accuracy with which these models are able to classify text is impressive. Finetuning a model requires GPU power at your disposal, moreover you need a dataset with labels to train on. However, labels are not always available. If you are working in a company without historical data or well defined labels in your dataset, you are not alone. It is surprisingly common not to have proper labelled data. So chances are that you or your team have not started to extract value out of textual data yet.

In this article I will show you how easy it is to use a language model to classify text without any pre-existing labels. In NLP this is called zero-shot classification: doing it right the first time without any additional training. With zero-shot classification, you use an already trained model to do something the model is not actually trained to do. Because the information contained in the model is so incredibly rich (billions of trained parameters on huge amounts of text), retraining or fine-tuning a model using existing labels is not necessary.

To do this, knowledge from the Natural Language Inference (NLI) domain is used. NLI models are trained on a task called text entailment. In this task two sentences are compared: a premise and a hypothesis. It is then up to the algorithm to assess whether the hypothesis is a logical consequence of the premise, a contradiction or neither (neutral).

Snaphot taken from the SNLI dataset.

Setting up our tools

In this example I will be using a NLI model suited for zero-shot classification trained on a multilingual dataset. In that way you can follow this example even when you have callcenter or emails at your disposal in other languages. For this example I’ve gathered 100 customer complaints taken from the US Consumer Complaints Database. The original dataset has the written complaints, a product label and 20 other features. Here I use only the issued label and the text of the complaint. Keeping the label will enable us to validate how well the zero-shot classifier has done the job.

Since NLI models are trained using a premise and a hypothesis we use a text sequence as the premise and use a set of possible labels as the hypotheses. In the example below we provide a statement (premise) and use a hypothesis template with all possible labels. The pipeline will provide a probability for each label. When multi_label is set to False the sum of all probabilities will be 1.

The HF pipeline outputs a dictionary containing the inputs (premise and hypothesis with labels) and a score for each label. Let’s turn this into a more human readable format.

Using an existing NLI model not trained for this specific task it is able to see that this premise is primarily about work and about health. Since this model was trained on a multiligual corpus we can easily make the same predictions using a French premise.

The model will struggle however if the text (premise, hypothesis, labels) provided does not resemble the corpus it was trained on. Less frequently used tokens will not appear in the tokenizer vocabulary making it difficult to distinguish between labels.

Classifying real data

Now that you’ve seen the basics of using a zero-shot classification model, let’s move on to making predictions on real data. Many companies have textual data at their disposal, for instance call-center logs, surveys, incoming e-mails or an ordersystem with notes. In this example we will work with complaints received and registrered in the Consumer Complaint Database. The original dataset has more than 20.000 complaints, for the purpose of this example I will use 100 records.

Let’s take a look at an example complaint. Personal information, amounts and dates have been masked in this dataset with XX’s. Before we use these examples to predict what the complaint is about we need to clean up the input.

Using a simple function to clean up the text yields a much cleaner input text to work with.

Next we look at the labels that were already provided to the complaints by the bank that handled the complaint. In this selection we have four different types of complaints: 1) there was a complaint about a deposit or withdrawal, 2) an unauthorized transaction took place, 3) a complaint about closing an account and 4) a complaint about the ability to get access to the account.

Below the labels are altered a bit to reflect a readable hypothesis. In that way the text of the complaint is used as the premise and the hypothesis is constructed like ‘This is a customer complaint about closing the account.’ It might be possbile that these complaints have multiple correct labels. A complaint can be on both a withdrawal and an unauthorized transaction. Therefore we will use a multilabel output.

In this blog we have 100 example records in our dataset. It would be great to have all these examples classified in one go. The function below takes care of this and adds a progress indicator as well. If you use a GPU enabled environment and a lot of examples it might be wise to use a Huggingface Dataset to perform the scoring. This will send batch-wise records to the GPU. In our example, with just a few records, feeding the examples one-by-one to either the CPU of GPU will work just fine.

The function below will hold all the label scores in a Pandas dataframe so we can easily compare results afterwards.

Now that we have the results for our 100 examples we can compare them to the origin labels to see how well the zeroshot model has performed. To be able to make this comparison we need to rename the output to the original labels. The result is stored in the column pred_label.

Please note that in this example I use existing labels to verify how well an untrained model is able to categorize the complaints in different buckets. Using a zeroshot pipeline is very much suitable when you want to classify text in a number of self chosen categories. So here I use existing labels but for your use case you can choose any labels that you want. Test to see what works out best for you.

Now we can use a basic classification report from sci-kit learn to get some statistics on the predicted labels. The overall precision 76,8% and the recall 67%, not bad for a model that was not trained for this specific use case.

Using a visualisation of the confusion matrix we can see the category ‘Unauthorized transaction’ was often predicted for other labels as well, which might be the case in reality.

In this blog I’ve showed you how to use an existing language model to perform a task it was not trained to do using the Huggingface zeroshot pipeline. In this case I’ve used a well performing general model to classify the complaints database examples. If you want to try to classify text using a zeroshot approach, please take a look on the Huggingface models tab and look for models that were trained on text that resembles your use case the most. The dataset I used can be downloaded from Gitlab here, the 100 case sample is available as well as a 10.000 case csv file (records use a pipe-separator |).