Natural Language Processing - the term is everywhere, but a true NLP app is hard to find

Profile picture for user Neil Raden By Neil Raden February 26, 2020
Just about every vendor claims they have NLP capabilities of some kind. But not all apps tagged with the "Natural Language Processing" label are created equal.


Have you noticed that you can't swing the proverbial dead cat by the tail without hitting a software product that claims to use Natural Language Processing (NLP)?

I've heard the term so frequently, I thought I'd try to create a sort of taxonomy of the different types and functions of NLP. The broadest definition of NLP is a method of communicating with intelligent systems using natural language.

At it’s the highest level, NLP is used to understand text (or voice). That's the hard part. At the lowest level, it's just a sort of user interface that parses and interprets text (or voice).

NLP can be utilized for User Interfaces. This is quite common, and everyone experiences this when they type a search string into Google, such as "Why did the space shuttle blow up?" I sometimes wonder if Google is really parsing the statement or just launching on keywords. In this case, I'd get a lot of hits on the Challenger disaster, but maybe not hits that answer my question, "Why?"

Where this gets interesting is when products allow you to type or speak an instruction. It can be straightforward, like launching into warp-drive when you say “Engage!” Or it can be more subtle, like when you say, “Which of the pricing models I’ve developed is showing the best likelihood of profitability?” What happens with a question like this? The NLP has to comprehend the semantic meaning. But then what? Does the NLP also possess the smarts to craft a complex query, or does it just use "pricing models," "best likelihood," or "profitability," and hope that is enough for the engine to give the right answer?

In some cases I’ve seen, the queries are limited to certain words that map directly to the underlying system. This isn’t very sophisticated. At the next level of sophistication, something takes the parsed question and translates it to some sort of internal request, by understanding the relationship between the parsed query and language needed to satisfy it. The question: is that NLP, or is that some internal process? I'm not sure, but one thing I can say for sure is that if the engine can't process what the NLP presents, it's not very valuable.

I've seen cases where these queries are iterative, where the user actually conducts a dialogue. After being presented with the system's best suggestion for a pricing plan, rather than starting over and changing the query from scratch, the user simply asks, "How about the top three?" Now the NLP has to do some thinking before framing the question to the engine. Consider a dialog using some simple statistical modeling.

-“I want to model with the variable having the highest pair-wise R2with dependent variable xi. Then induce xi.2. Is R2increasing? “


“Remove xi.2.”

- Or, If yes, “Accept. xi.2.and induce all higher orders of until the rth order based on improvement of R2.”


“Induce xj”

And so forth. This is fairly simple multi-step analysis, and there are plenty of statistical GUI packages with which one could do the same thing. The measure of an NLP for analytics is: can it follow these instructions without losing the topical thread?

As for the use of NLP for understanding language at a broader level, there I've seen not only some impressive results, but applications we use (or we are used by) every day: Keyword Search, Spell and Grammar Checking, Sentiment Analysis, Speech Recognition, Machine Translation).

Chatbots are an interesting innovation. In a way, they are a more capable technology than the NLP multi-query example above. A well-designed chatbot not only engages in two-way dialogue, but learns very quickly and can at some level be indistinguishable from a human, but only until it gaffs and says something inappropriate or irrelevant. Nevertheless, a chatbot with NLP for speech generation and speech recognition is something we will see more of, but there is other technology behind it that drives the interaction.

When you get down to bits and bytes, these smarter NLP's use actual AI techniques in the form of Recurrent Neural Networks and Attention Neural Networks, which allow for temporal (time) dynamic behavior. Unlike feed-forward neural networks, RNNs can use their internal state (memory) to process sequences of inputs, like the iterative query above. This makes them applicable to tasks such as unsegmented, connected handwriting recognition, or speech recognition.

One application that is getting a lot of notice in the BI/Visualization space is Narrative Science. Tableau, Sisense, and Qlik have all partnered with them to narrate data visualizations with text. Data visualization narration provides storytelling to the analytics, while reducing the likelihood of misinterpretation. It facilitates a collective understanding of data visualization. In the absence of that, people may disagree about what a chart or graph means.

My take

Tread carefully about NLP claims. If the engine can't execute what the NLP presents, that's a non-starter. In cases where NLP is the user interface, understand if it is just a thin layer to do what you could yourself without it. Look closely. But if it genuinely has semantic understanding, remember, the simplest things to say are often the most difficult for a machine to understand (“summarize that”). NLP apps like sentiment analysis, chatbots, etc. stand on their own as actual applications of NLP technology. But they should still be put to the test before NLP claims are made - not all bots are created equal.

Image credit - Where,who,when,how,why,what,questions and researching concept. © Sondem -