How To Create A Vocabulary Builder For NLP Tasks?
The vocabulary helps in pre-processing of corpus text which acts as a classification and also a storage location for the processed corpus text. Once a text has been processed, any relevant metadata can be collected and stored. In this article, we will discuss the implementation of vocabulary builder in python for storing processed text data that can be used in future for NLP tasks. About the dataset The dataset is downloaded from the following link. It contains the tweets of youtube users. The tweets are used for sentiment analysis. Code Implementation Import all the libraries required for this project. import pandas as pd import numpy as np data = pd.read_csv("youtube.csv") data.shape data.head() Normalize the text The next step is to perform the normalization of our text data. We need to convert the text data to lower case. data["comment_text"].str.lower() Making a dictionary for expanding the English language For data pre-processing, expand the data using a contraction function. For example, words like ‘can’t’ are expanded to ‘cannot’. contractions_dict = { "ain't": "am not / are not / is not / has not / have not", "aren't": "are not / am not", "can't": "cannot", "can't've": "cannot have", "'cause": "because", "could've": "could have", "couldn't": "could not", "couldn't've": "could not have", "didn't": "did not", "doesn't": "does not", "don't": "do not", "hadn't": "had not", "hadn't've": "had not have", "hasn't": "has not", "haven't": "have not", "he'd": "he had / he would", "he'd've": "he would have", "he'll": "he shall / he will", "he'll've": "he shall have / he will have", "he's": "he has / he is", "how'd": "how did", "how'd'y": "how do you", "how'll": "how will", "how's": "how has / how is / how does", "I'd": "I had / I would", "I'd've": "I would have", "I'll": "I shall / I will", "I'll've": "I shall have / I will have", "I'm": "I am", "I've": "I have", "isn't": "is not", "it'd": "it had / it would", "it'd've": "it would have", "it'll": "it shall / it will", "it'll've": "it shall have / it will have", "it's": "it has / it is", "let's": "let us", "ma'am": "madam", "mayn't": "may not", "might've": "might have", "mightn't": "might not", "mightn't've": "might not have", "must've": "must have", "mustn't": "must not", "mustn't've": "must not have", "needn't": "need not", "needn't've": "need not have", "o'clock": "of the clock", "oughtn't": "ought not", "oughtn't've": "ought not have", "shan't": "shall not", "sha'n't": "shall not", "shan't've": "shall not have", "she'd": "she had / she would", "she'd've": "she would have", "she'll": "she shall / she will", "she'll've": "she shall have / she will have", "she's": "she has / she is", "should've": "should have", "shouldn't": "should not", "shouldn't've": "should not have", "so've": "so have", "so's": "so as / so is", "that'd": "that would / that had", "that'd've": "that would have", "that's": "that has / that is", "there'd": "there had / there would", "there'd've": "there would have", "there's": "there has / there is", "they'd": "they had / they would", "they'd've": "they would have", "they'll": "they shall / they will", "they'll've": "they shall have / they will have", "they're": "they are", "they've": "they have", "to've": "to have", "wasn't": "was not", "we'd": "we had / we would", "we'd've": "we would have", "we'll": "we will", "we'll've": "we will have", "we're": "we are", "we've": "we have", "weren't": "were not", "what'll": "what shall / what will", "what'll've": "what shall have / what will have", "what're": "what are", "what's": "what has / what is", "what've": "what have", "when's": "when has / when is", "when've": "when have", "where'd": "where did", "where's": "where has / where is", "where've": "where have", "who'll": "who shall / who will", "who'll've": "who shall have / who will have", "who's": "who has / who is", "who've": "who have", "why's": "why has / why is", "why've": "why have", "will've": "will have", "won't": "will not", "won't've": "will not have", "would've": "would have", "wouldn't": "would not", "wouldn't've": "would not have", "y'all": "you all", "y'all'd": "you all would", "y'all'd've": "you all would have", "y'all're": "you all are", "y'all've": "you all have", "you'd": "you had / you would", "you'd've": "you would have", "you'll": "you shall / you will", "you'll've": "you shall have / you will have", "you're": "you are", "you've": "you have" } Contraction Function for expanding english language import regex as re contractions_re = re.compile('(%s)' % '|'.join(contractions_dict.keys())) def expand_contractions(s, contractions_dict=contractions_dict): def replace(match): return contractions_dict[match.group(0)] return contractions_re.sub(replace, s) data["comment_text"] = [expand_contractions(i) for i in data["comment_text"]] Remove patterns using regex(Keep a-zA-Z0-9) We have to keep the letters and numbers in text data and remove all the punctuation data. This can be achieved by using the regex function. import re data["comment_text"].replace( { r'[^a-zA-Z0-9, ]' : '' }, inplace= True, regex = True) data["comment_text"] #Import Libraries import nltk nltk.download('punkt') Tokenize words The main implementation of the code starts by tokenizing the words present in the text data. from nltk.tokenize import word_tokenize tokenized_sents = [word_tokenize(i) for i in data["comment_text"]] print(tokenized_sents) Add words to the list Let’s create a list and add all the tokenized words to the list. flattened = [] for sublist in tokenized_sents: for val in sublist: flattened.append(val) print(flattened) Add words which are not in the list The result obtained includes a lot of duplicate words. So, another list is used to store the words that are not present in the list. Vocab=[] for item in flattened: if not item in Vocab: Vocab.append(item) print(Vocab) Conclusion In this article, we have learned the implementation of the Vocabulary builder that can be used for NLP tasks. Further, we can extend our research by removing the words which are not present in the English dictionary. Hopefully, this article will be useful to you.The complete code of the above implementation is available at the AIM’s GitHub repository. Please visit this link to find the code and this link to find the dataset.
The vocabulary helps in pre-processing of corpus text which acts as a classification and also a storage location for the processed corpus text. Once a text has been processed, any relevant metadata can be collected and stored.In this article, we will discuss the implementation of vocabulary builder in python for storing processed text data that can be used in future for NLP tasks.
We will discuss the implementation of vocabulary builder in python for storing processed text data that can be used in future for NLP tasks
["data preprocessing", "Natural Language Processing", "Python Libraries", "Sentiment Analysis", "sentiment analysis NLP"]
["NumPy", "Sentiment Analysis", "data preprocessing", "Python Libraries", "Natural Language Processing", "AI", "sentiment analysis NLP", "sentiment analysis", "RAG", "NLP", "Python", "Aim", "NLTK", "R", "Pandas"]
["AI", "NLP", "Aim", "NLTK", "Pandas", "NumPy", "RAG", "sentiment analysis", "Python", "R"]
A data analyst with expertise in statistical analysis, data visualization ready to serve the industry using various analytical platforms. I look forward to having in-depth knowledge of machine learning and data science. Outside work, you can find me as a fun-loving person with hobbies such as sports and music.