By Patrick McDonald,
Chief Architect, Advanced Analytics at Clarity Insights. Patrick brings 23 years of experience to over 50 data science and advanced analytics projects and has delivered $4.4 billion to client bottom lines.

You have to have a well-honed sense of irony if you work in the world of data analytics and Artificial Intelligence (AI). So many situations evolve counter to the way they’re supposed to. Take the unlikely role that commercial AI vendors are now playing in the democratization of AI and Machine Learning (ML). See, open source was supposed to make advanced technologies available without the interference of those money-grubbing commercial AI vendors. “Power to the (tech) people!” Well, not so much, it turns out.

Commercial AI vendors are the ones actually making AI and ML available to regular business users on a sensible and affordable basis. Gartner has caught this trend and discussed it in their recent report, “Top 10 Data and Analytics Technology Trends That Will Change Your Business.” In their view, “Commercial AI/ML Will Dominate the Market Over Open Source.”


Limitations of Open Source AI Platforms

We’re in an inevitable market and innovation cycle where open source is revealing some weaknesses in the broader market. AI and ML are dominated by open-source platforms like Python, R, Apache Spark,, Anaconda and TensorFlow. Their dominance is well-earned. As Gartner notes, “Continued fast-track innovation in both algorithms and development environments over the past five years has mostly occurred in open-source options.”

However, as Gartner also notes, the prevalence of these platforms is a sign of market immaturity. The commercial vendors have not stepped up so far. That is about to change. To understand why this matters, consider the following example. Imagine that you want to set up a data analytics solution for a healthcare provider.

Let’s say you want to determine if there is a causal effect between prescription fulfillment and re-admission to the hospital. You might have a hunch that hospital patients who do not fill prescriptions for antibiotics tend to come back for an unexpected and preventable second hospital stay due to infections. If you can prove a connection, you could save the business millions of dollars by implementing changes to the prescription fulfillment and patient follow-up process.

Let’s further imagine that, due to the lack of a suitable commercial AI solution, you are going to use the Apache Spark platform for your analytics project. Spark can do everything you want. But, as is the case with open source, you’re completely on your own. You have to stand up the Spark Core on machines that you configure by yourself. Then, you have to stand up the MLLib Machine Learning Library that runs on the Spark Core. From there, you have to make the ML tools work the way you need them to for your prescription analysis use case.

This is all fine if you know what you’re doing, but that’s a huge assumption. The reality is that many, if not most, businesses lack the skill-sets required for this project. Or, if you have the people who know how to use Spark, there might be a long line of projects ahead of you waiting for them.

The irony here, of course, is that the open source solution, which was meant to make technology available to the many, can only be accessed by the few—the few with the knowledge to use open source AI and ML. Commercial AI platforms are now catching up.


Navigating the Landscape of Disruption, Innovation and Transformation - A Gartner Report

Get the Gartner Report



The Gartner Prediction about Commercial AI

Gartner predicts that 75% of new AI and ML end-user solutions will be built with commercial, rather than open-source, platforms by 2022. This is happening because commercial AI vendors, who were originally slow to adapt, are now building connectors to the open source ecosystem.
And, commercial providers bring with them the enterprise-class features necessary for scaling AI and ML, e.g. project and model management, reuse, transparency, data lineage, platform cohesiveness and integration. Most open source platforms lack such features, according to Gartner.

Democratization of AI is the effect forecast by Gartner due to this trend. As they note, “Commercial providers will increasingly “smooth” the rough edges often associated with open-source projects by orchestrating the user experience and “connecting the dots.” In the healthcare example, a commercial AI platform makes it possible to stand up the analytics solution and establish the analytical models, data integrations and so forth without relying on hard-to-find open source specialists. According to Gartner, “Cutting-edge AI/ML development becomes much more available to the broader skill set found in most enterprises.”

The major tech vendors are moving in. Amazon has SageMaker. Google is likely to put most of its ML innovations into the Kubernetes framework. IBM, SAS, SAP and Oracle are similarly revamping and enhancing their AI and ML offerings. The Gartner report suggests that, by 2022, “Cloud-based ML services from the hyperscale cloud providers (Amazon, Google and Microsoft) will achieve the digital tipping point of 20% share in the data science platform market.”


Business Impact of the Democratization of AI and ML

It’s worth noting that Gartner is not predicting that open source AI and ML will disappear. Rather, they see that improvements in integration between open source and commercial models will enable better scaling and enterprise adoption. Commercial AI platforms offer a range of advantages over open source, beyond limiting the need for highly-specialized staff resources. While you have to pay for these benefits, you get a much more rigorous and disciplined approach and set of capabilities.

For instance, commercial platforms should enable rapid innovation in big data. Gartner finds that data science and AI have started to “cool off.” However, as commercial vendors build connectors into open-source ecosystems, users can combine the innovation found in open-source components and the enterprise-ready tools found in commercial ML/AI platforms.

Other benefits of commercial AI platforms include:

  • Improvements in data analytics team productivity—Given how much skill and manual labor has been required for the assembly and retrofitting of diverse OS tools, commercial platforms will enable people to get more analytics work done in less time. This should offer a clearer, faster path to business value, which is now a core expectation of an analytics project. In the healthcare case, commercial AI would enable the project team to quickly show a demonstrable financial value for their work, e.g. a reduction in costly patient re-admissions.
  • Better planning and roadmaps for AI—Robust planning and project management tools in commercial AI/ML platforms promise to reduce uncertainty and conflict in organizational AI roadmaps, e.g. after calculating the financial impact of reducing patient re-admissions, let’s study how we could cut costs on caring for those patients who were re-admitted.
  • More collaboration—Commercial platforms increase the probability of collaboration between AI/ML specialists with varying skill levels. This is good on its own, and given the kind of organizational churn we see due to M&A, to name just one driver of this phenomenon, the ability to have people work together in AI will accrue to the benefit of the business.


Looking Ahead

The predictions about commercial platforms’ democratization of AI and ML are simply predictions. It’s impossible to know if they will turn out to be true. Usually, what actually happens is different from what’s expected, though the general outlines of the trend are accurate. It does seem likely that commercial AI and ML platforms will expand the field in coming years, offering a range of use cases to users of varying skill levels.

AI and ML still take time, focus and skill. No platform can do it all for you. We can help. Clarity Insights can advise your organization on how to make the most of the coming advances in commercial AI. 

You may also like:

Machine Learning Business Intelligence Artificial Intelligence

The Rise of Continuous Intelligence and How It’s Changing Analytics and BI

Continuous intelligence, according to the Gartner report, Top 10 Data and Analytics Technology Trends That Will Change Y...

Machine Learning Data Management Artificial Intelligence

What Will Augmented Data Management Mean for Your Organization?

To understand augmented data management, it’s first necessary to know what is involved in basic data management. “Data m...

Machine Learning

Can Machine Learning Improve Content Marketing Effectiveness?

The idea of a thinking machine may sound like a far-off, heady concept fit for Sci-Fi stories, but analytics firms have ...