In response to the question, “how does machine learning work?” there are a lot of myths: that it learns automatically, that it doesn‘t require customization or any modification from the organization using it, that it fits any business case and can solve any problem. But which of those response are true?

Like any other technology, machine learning (ML) has great potential for business value in some cases, but not always. Despite the hype generated by the big tech marketing Machine it’s often not the best solution for analyzing unstructured information. Despite what you may have heard, IBM’s Watson does not think or reason. It will not do so today nor in the near future. This is the truth that (almost) no one wants to talk about.

As explained in this blog post, machine learning is “a different way of programming a computer to execute a task;” so, before asking “how does machine learning work?” it’s very important to set the right expectations about the potential of machine learning for business while keeping in mind that there is nothing magic or new. And, although it’s easy to get the (mistaken) impression that computers could quickly become very intelligent (on or beyond the level of humans, thanks to artificial intelligence), machine learning has little to do with human intelligence. It’s just a technology that “learns” through training and processes specific inputs in order to apply text analysis approaches.

What characterizes machine learning techniques (and their limitations)

Although there are several machine learning techniques, they all share a similar core composed of statistics and co-occurrences. In layman’s terms, ML doesn’t have embedded knowledge but requires a set of documents for training (the larger the set of documents, the better). This training set must be manually pre-tagged by people so that the algorithm can record what’s in the document and identify examples of how specific content is paired with specific tags. This is why ML can’t do everything itself… A lot of manual work will be required in any scenario!

To “learn” and recognize documents related to a topic, machine learning techniques have to ingest a large quantity of documents related to the topic. It also requires manual tagging of a large amount of documents that cover this topic and a similar amount that do not cover the topic. Only after processing many documents by looking at co-occurrencies and keyword frequency the system will be able to recognize the topic of the document.

The level of accuracy of a trained system will vary based on the number of documents used during the training phase and the coverage of the specific jargon in those documents. The system must also be retrained frequently to maintain the same level of quality. Information overload will slow down the whole system, while overfitting it (too many documents of the same “genre”) will make the system less accurate; in other words, the wrong selection of documents will actually cause a decrease in quality.

In a famous article, we read that “with machine learning, the engineer never knows precisely how the computer accomplishes its tasks. The neural network’s operations are largely opaque and inscrutable. It is, in other words, a black box.” This means that there is a limit to the level of improvement possible, and it is often difficult to understand why the system has improved or how you can improve it further. For machine learning systems, there simply are no tools with which to refine the algorithm. With pure ML, the only option you have is to feed more examples to the algorithm. Unfortunately, this doesn’t always guarantee that you’ll improve the results or reach the level of accuracy required.

If any mistakes are discovered or the trained system needs to be modified for any reason, the entire process must go back to square one.

From machine learning to Cognitive AI

Going back to our first question, “how does machine learning work?” we can say that machine learning easily supports the organization when we have a significant number of sample documents to train the ML algorithms and we have to face a simple scenario. The text analysis project that is a true candidate for pure ML is exactly this: a low complexity case and a large training set with a balanced distribution of all the possible output.

Instead, when we have scenarios that involve a small and not uniformly distributed set of samples and high complexity, ML is not enough. For these use cases, you need a linguistic engine that is sophisticated enough to ensure a deep understanding of the content and a set of tools that are powerful enough to ensure the development and effective application of advanced linguistic rules.

The Cogito cognitive technology stack provides a unique combination of rule-based cognitive technology capabilities and ML-based algorithms to address the most common use cases for analyzing unstructured information. Combining the most advanced AI techniques such as machine learning with proprietary semantic and natural language processing algorithms, Cogito ensures a higher level of performance in each situation.

Watch this video to understand the difference between Cogito and other technologies based only on machine learning and how Cogito can create value for your business.

The post How does machine learning work? Myth and reality appeared first on .