FlashText – A library faster than Regular Expressions for NLP tasks
People like me working in the field of Natural Language Processing almost always come across the task of replacing words in a text. The reasons behind replacing the words may be different. Some of them are.
- “would’ve” and “would have” represent the same thing. So changing all the occurrences of “would’ve” to “would have” is one such task.
- Changing all Case Variations to a single form i.e Python, pytHon, pYthon, pythoN etc. to python
- Changing all the synonyms of a word to a common word i.e happy, joyous, delightful etc to happy
Now, if the number of words to replace and the corpus of text is not huge i.e within thousands, then Regular Expressions have always been my solution. But as I started working on bigger and bigger datasets with tens of thousands of documents and sometimes millions, I noticed that performing the above tasks started taking days. In today’s fast-moving world, this is not the amount of time one would want to invest in a very simple but important task. So earlier, it