Utilizing standard deviation in text classification weighting schemes

The term frequency – inverse document frequency (TF-IDF) weighting scheme is widely used in text classification for weighting the features of the vector space model (VSM). It aims at enhancing words’ discriminating capabilities by weighing up the less frequently used words and, at the same time, weighing down the high frequency words (i.e., the common words such as prepositions). This paper attempts to provide an enhanced variant of the well-known TF-IDF method. The TF-IDF is a statistical estimation that computes the weight of each word based on the frequency of the word in both the document and the entire data collection. In this work, we propose considering the word’s standard deviation as another factor when computing the word’s weight. That is, the common words tend to have larger standard deviations more than the uncommon words. In other words, the more the word appears in documents, the greater the standard deviation is. To investigate the proposed TF-IDF based model, we conducted some experiments for Arabic text classification. We used a training textual data collection that contains 1,750 documents of five categories (250 documents for testing). The experimental results show that the proposed approach is superior to the standard TF-IDF term weighting scheme.