research-notifications

Project with tools for notification text correction written in Python 3.9.

Context

The purpose of this project is to optimize process of testing correctness of notifications, as comparison of notification with its exact pattern "as is" takes too much time to prepare and support in ongoing test updates. To solve this problem a different approach of testing was proposed:

1. Check correctness of data fields (dates, amounts, names of products, etc.)
2. Search for typos in notification (evaluation errors, typos in words, word duplication, etc.)
3. Check notification for grammatical and syntactical faults

Some examples of typos and grammar faults:

• Evaluation error

  Поступил платёж на сумму ${(amount)|numberformat("0.##")} руб. 
  

• Duplication error

  В ночь 24 октября спишется спишется 550 руб. по тарифу "Всё".
  

• Typo (duplicated letters)

  Ваш баааланс: 600 руб. Спасибо, что пользуетесь услугами.
  

• Grammar or punctuation fault

  Кстати пополнить баланс легк на сайте или в ЛК.
  

In order to implement this approach research was held, in which Classic NLP, Rule-based and Neural Network methods of text processing were analysed and compared by accuracy and performance on notification dataset.

Features

Before testing text correctness methods, a dataset of notifications was created. Although less than 500 notifications with errors were provided, another 2000 examples were generated to balance correct ones. So the final dataset consists of 5000+ unique notifications and labels (1 if notification is incorrect and 0 otherwise). Formally, further research is an attempt to solve classification problem with different NLP approaches.

Full dataset can not be shown, because it is corporate property, but few examples of real training data samples can be found here.

In this project 4 methods of checking text correctness were tested:

• vocab

Simple approach for a baseline. Basically, just a vocabulary of 1M+ Russian words and forms. Predicts notification as error if at least one of its words is outside vocabulary.

• language_tool

Classifier based on well-known rule-based grammar checking tool LanguageTool, which is used as spell-checker in OpenOffice. Predicts notification as error if at least one suggestion to replace anything occurs.

• bert-multilingual

A bert-base-multilingual-uncased model from HuggingFace by Google fine-tuned on notifications dataset.

• rubert

Another transformers model by DeepPavlov fine-tuned on notifications. RuBERT is based on BERT-multilingual, but it was additionally trained on Russian texts.

Performance

As a result of methods accuracy & performance comparison on notifications dataset here is a table a metrics:

	precision	recall	accuracy	MCC	avg check time (sec)
vocabulary	0.80	0.28	0.61	-	0.0001
language_tool	0.95	0.62	0.79	-	0.17
bert-multilingual	0.99	0.67	0.83	0.70	0.07
rubert	0.99	0.76	0.90	0.84	0.06

where MCC stands for Matthew's Correlation Coefficient.

Note: value of precision metric, which is nearly 1, stands that almost none false positive samples. It means that almost none correct notifications were classified as in-correct.

Requirements

See here.

Best way to get all required packages at once is running the following line:

pip install -r requirements.txt

After installation of nltk you might also need to execute the following in python:

import nltk

nltk.download('punkt')
nltk.download('stopwords')

References

• "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation" by N. Reimers, I. Gurevych (source)
• "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by J. Devlin, M.-W. Chang, K. Lee, K. Toutanova (source)

License

MIT License