May 27, 2021 NLP, tokenisation

Issue #132 – Tokenization strategies for Korean MT tasks

27 May21 Issue #132 – Tokenization strategies for Korean MT tasks in Model improvement, The Neural MT Weekly Author: Dr. Jingyi Han, Machine Translation Scientist @ Iconic Introduction Asian languages have always been challenging for machine translation (MT) tasks due to their completely different grammar and writing system. As we know, there are specific segmenters for Chinese and Japanese as there is no space between words in these languages. With regards to Korean, even though the words are separated by […]

March 23, 2021 Advanced, NLP, Unsupervised, Videos

Tokenization and Text Normalization

Objective Text data is a type of unstructured data used in natural language processing. Understand how to preprocess the text data before feeding it to the machine learning algorithms. Introduction Text data is a form of unstructured data. The most prominent examples of text data available on the internet are social media data like tweets, posts, comments, or the Conversation data such as messages, emails, Chats. Also, it can be article data like news articles, blogs, etc. Note: If you […]

November 9, 2020 Intermediate, NLP, Python, Technique, Text, Unstructured Data Leave a comment

How to Get Started with NLP – 6 Unique Methods to Perform Tokenization

Overview Looking to get started with Natural Language Processing (NLP)? Here’s the perfect first step Learn how to perform tokenization – a key aspect to preparing your data for building NLP models We present 6 different ways to perform tokenization on text data Introduction Are you fascinated by the amount of text data available on the internet? Are you looking for ways to work with this text data but aren’t sure where to begin? Machines, after all, recognize numbers, […]

November 7, 2020 Beginner, Libraries, NLP, Python, Text, Unstructured Data Leave a comment

Hugging Face Releases New NLP ‘Tokenizers’ Library Version (v0.8.0)

Hugging Face is at the forefront of a lot of updates in the NLP space. They have released one groundbreaking NLP library after another in the last few years. Honestly, I have learned and improved my own NLP skills a lot thanks to the work open-sourced by Hugging Face. And today, they’ve released another big update – a brand new version of their popular Tokenizer library. A Quick Introduction to Tokenization So, what is tokenization? Tokenization is a crucial […]

November 6, 2020 Advanced, Algorithm, NLP, Python, Text, Unstructured Data Leave a comment

What is Tokenization in NLP? Here’s All You Need To Know

Highlights Tokenization is a key (and mandatory) aspect of working with text data We’ll discuss the various nuances of tokenization, including how to handle Out-of-Vocabulary words (OOV) Introduction Language is a thing of beauty. But mastering a new language from scratch is quite a daunting prospect. If you’ve ever picked up a language that wasn’t your mother tongue, you’ll relate to this! There are so many layers to peel off and syntaxes to consider – it’s quite a challenge. […]

October 13, 2020 Python

Simple NLP in Python With TextBlob: Tokenization

Introduction The amount of textual data on the Internet has significantly increased in the past decades. There’s no doubt that the processing of this amount of information must be automated, and the TextBlob package is one of the fairly simple ways to perform NLP – Natural Language Processing. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, tokenization, sentiment analysis, classification, translation, and more. No special technical prerequisites […]