Banking news-events representation and classification with a novel hybrid model using DistilBERT and rule-based features

Abstract

This paper discusses a novel hybrid approach to text classification that integrates a machine learning algorithm along with DistilBERT, a pre-trained deep learning framework for natural language processing, offers a base model fine-tuned on Indian Banking News-Events with a rule-based method that is used by filtering false positives and dealing with false negatives to enhance the results given by the previous classifier. The major benefit is that by incorporating unique rules for certain chaotic or overlapping categories that have not been effectively trained, the system can be quickly fine-tuned. This research also compares the effectiveness of state-of-art deep contextual language representation DistilBERT used in our proposed hybrid model with the most preferred context independent language representation TFIDF, on supervised learning of the classification of multiclass Banking News-Events. Both representations are fed into the machine learning classifiers, Logistic Regression, Linear SVC, Decision Tree, and Random Forest. The findings indicate that DistilBERT can better transfer generic domain knowledge to other domains as compared to the baseline TFIDF and results into higher accuracy with Random Forest. The result is further fed into the Rules based method as per our proposed model. And, our proposed hybrid model resulted into improved accuracy.