ZohoDay 2024 - how Zoho plans to integrate LLMs and smaller language models to get a better (and cheaper) AI result

Jon Reed Profile picture for user jreed February 26, 2024
ZohoDay 2024 surfaced fascinating details on Zoho's generative AI strategy - by integrating a range of language models in the same workflows. What does this mean to customers? And is this approach consistent with Zoho's tech philosophy? Grab your beverage of choice and let's dig in.

 Zoho's Raju Vegesna at ZohoDay 2024
(Zoho's Raju Vegesna at ZohoDay 2024)

One of our top goals heading into ZohoDay 2024? Get a noteworthy update on Zoho's generative AI strategy.

Yes, every enterprise software vendor is vigorously pounding their generative AI talking points.

But Zoho has a very Zoho way it goes about technology, whereas generative AI seems to take us headlong into the realm of big data - and big tech. That's not the Zoho way - so doesn't something have to give?

Last May, I delved into that apparent contradiction in Zoholics 2023 - Zoho unveils its generative AI news, but does it co-exist with customer data privacy? Since then, we've documented the AI strategies of a range of enterprise vendors. It boils down to this:

1. Achieving a good generative AI result in the enterprise almost always comes down to data quality, data governance, and the existing state of workflow automation.

2. Enterprise vendors employ a range of gen AI architectures, but all of them employ techniques such as RAG (Retrieval Augmented Generation) to improve upon the less-than-ideal output of consumer LLMs - and infuse them with customer and industry-specific data for better/more relevant output.

3. Few vendors are building their own LLMs. Most are sourcing multiple external LLMs for different use cases, but without allowing the customer's data to intermingle or become part of the LLM's training data. (However, a number of vendors are building their own foundational models to integrate with external LLMs).

4. Most enterprise vendors that have a credible gen AI strategy have been embedding AI in their products for years, and are not new to the pros/cons of deep learning.

Zoho's response to generative AI involves all four of these points. In particular, we learned about how Zoho is essentially playing one LLM against another for better output - and taking full advantage of Small(er) Language Models (SLMs) to control AI operating costs, and further improve outputs.

"The best AI is the AI customers don't notice"

During ZohoDay, the most memorable AI demos were 'classic' AI features already live in Zoho's applications. In his day one keynote, Zoho Chief Evangelist Raju Vegesna spoke to the AI marketing fatigue in the room:

You've probably heard enough of AI from various vendors, and you want to peer through the BS layers.

Indeed. Vegesna continued: for Zoho, the best AI is the AI customers don't even notice:

What is the best AI implementation? The best AI implementation, our view, is when customers don't know that they're using AI, and they're just reaping the benefits.

Instead of detailing new gen AI features, Vegesna honed in on Zoho's AI philosophy: contextual intelligence.

We have been rolling out a lot of AI capabilities; I'm not going to talk about any of them, but I want to directionally provide where we are going...  Our focus is on what we call contextual intelligence. What is contextual intelligence? Well, if you fuse business context with Artificial Intelligence, today, that context is something that has been missing in a lot of cases.

Vegesna grilled the consumer LLMs for falling short on this:

If you look at today's consumer [LLMs], they totally lack business context. That's something the customer needs - what industry they're in, more importantly, their data, their domain; their context is completely missing from these consumer LLMs.

But what does "contextual intelligence" look like for Zoho users? Vegesna cited the treacherous domain of enterprise search, where users still squander way too much time. But Vegesna said with Zia, Zoho's "AI powered digital assistant for business," queries like "Show me all the documents shared to my team this month" can deliver a quick result.

As Vegesna pointed out, it's a simple question - but only on the surface. If your AI isn't aware of the role/team/context, it can't serve up the relevant documents. Role-based access and security is part of this equation too. Zoho believes it has an edge here, via its "unified" Zoho One platform.

Vegesna takes this further: what if we can move past search, and into our AI assistant prompting us each day?

Imagine logging into the system. And the system says, 'You have four tickets that needs to be required, and you have five high priority tasks planned this week, and three escalations from the customer.

On a per-user basis, this is not easy to serve up: "All of this needs context," says Vegesna. Agreed - but it also needs a menu of pre-built automations the AI can invoke.

Zoho AI live demo - analytics and next best actions

Vegesna showed us a Zoho Analytics demo of a profitability dashboard, where recommended advertising adjustments are made on the right-hand (mouseclick) menu, via the "Insights: Diagnosis" section:

Zoho - BI profitability screen shot
(Screen shot from Zoho Analytics, profit analysis and next best actions) (Screen shot from Zoho Analytics, profit analysis and next best actions)

This is already live for Zoho Analytics customers - but what about gen AI? In terms of Zoho's upcoming generative AI features, I won't detail the plans Zoho has already shared with us, but a sampling of new/upcoming gen AI features includes:

Zoho Analytics with Generative AI - Get suggestions and import public datasets into Zoho Analytics. Blend public data with business data to gather insights; Define formulas for KPI metrics.
Zoho Desk with Generative AI - Automatically summarize incoming and outgoing tickets; Analyze mood of customer based on tone of request.
Zoho Writer with Generative AI - Suggest headlines, titles, and better word replacements; Ask questions within Writer and integrate answers into document.

In sum, Zoho will be embedding generative AI across Zoho One. The big news of ZohoDay wasn't the features, but the architectural reveal. In my May 2023 piece on Zoho's AI direction, I shared Zoho's phased approach to gen AI, which gives customers opt-in access to third party LLMs like OpenAI, transitioning to open source tooling that brings all customer data back in-house, with all AI operations pulling from Zoho's globally distributed data centers.

These phases still hold up, but I see changes. Zoho has found a way to integrate its own SLMs (and Medium Language Models) into the mix.

Zoho's gen AI architecture - integrating LLMs and SLMs into the same workflows

During his keynote, Vegesna showed how different language models can be applied to achieve a workflow result. One example: an "intelligent" expense approval scenario. Moving from the initial receipt photo to the automated expense report involved OCR (Narrow Language Model), Text Extraction (SLM), Inference (SLM), Anomaly Detection (MLM) and Deviations (LLM). The only LLM required in the process is for policy deviations. Notably, the "inference" of category, currency conversion etc. is simply an SLM.

The potential gains?

1. Different language models excel at different things. Utilizing them in the same workflow increases accuracy/benefit.

2. Reduced dependency on LLMs is a potential cost/energy consumption win. Controlling gen AI costs is an underrated factor in how much of this functionality is truly accessible to customers, particularly in the SMB market. Also, this reduces the external language models Zoho needs to incorporate.

And yes, foundational models have a role to play. For enterprise AI, I think of foundational models as a way of further refining what's possible with LLMs - by providing more industry or process context. This can be a way of tuning model output. During our diginomica team sitdown with Ramprakash Ramamoorthy, AI Research Lead at Zoho, he told us:

Now talking about the small, medium and large language models, how we have worked it out is we have built foundational models that are, let's say, a 3 billion [parameter] model, a 5 billion model, a 7 billion model and a 20 billion parameter model. Then what we do is take this foundational model, and fine tune it to either a domain or a task.

So for example, we fine tune it to the finance domain, or we fine tune it to a legal domain, or we fine tune it to something like question answering, or we fine tune it to something like document similarity prediction. So it's either fine tuned on a domain or a task.

Ramamoorthy detailed the tools in use:

These foundational models are developed as proprietary Zoho models based on Tensorflow or Pytorch frameworks, and trained using open source information or third-party data sources. Zoho then fine-tunes the models using its own curated data sets.

My take - Zoho wants to advance enterprise AI without the trap of big tech AI costs

Zoho is by no means the only enterprise vendor that believes they can mitigate the downsides of LLMs. But Zoho's approach to playing LLMs off of each other in the same process is not as common, at least not yet. Bringing smaller language models into the mix, and determining just the right amount of parameters - and no more - will probably become commonplace. But it isn't today.

This speaks to Zoho's determination to achieve AI results - without passing "big tech AI" operational costs to their customers. Ramprakash told us that with "proper context," a 50 billion parameter model (which I would consider a Medium Language Model) is large enough for Zoho Finance - because finance users "don't need to ask the model how to bake a cake."

I'm out of space for deep dives, but it's worth noting that when Zoho productionizes smaller language models, they use CPUs for inference. Ramprakash says the cost savings are substantial:

We don't use the GPU for the less-than-5-billion model parameter inference. For [model training], we have to do it on the GPUs. But when we do the inference on CPUs, the cost really comes down.

You could make the argument that some of these "small language models" were already in existence, and just have a shiny new name. Perhaps - but pulling different models into the same process is where Zoho finds the real action. However, smaller language models have a narrower purview. They lack both the wackier (and more brilliant) outputs we see from the larger consumer-facing models. Ramprakash told Wainewright and I that in Zoho's AI pursuits, once language models get to about 2.5 billion parameters, you see some of the output behaviors that have made GPT type interfaces so popular - and controversial.

But Zoho is not in the AI-goes-viral game. They intend to stay out of such controversy by avoiding under and overfitting; Zoho will stick to their "best AI is no AI" mantra. And yes, Ramprakash says, RAG (Retrieval Augmented Generation) is also applied in the final stages of the output generation, to (privately) infuse customer-specific information into the results. When larger models are involved, reinforcement learning comes into play, to further tune the output.

I may have buried the lead. During the event, Zoho CEO Sridhar Vembu said he was spending 70 percent of his time pursuing the impact of AI on Zoho's own development process. We asked Vembu about the impact:

A lot of it is raising programmer productivity by removing a lot of the friction in software development today. If you come in to see how the sausage is made, the average programmer is spending 80 percent of their time doing mundane stuff. In other words, it's a lot of tedious stuff that doesn't really add value, but that's where they're stuck. It's long been long known, but the more we have we have tried to solve it, the more and more layers of complexity are thrown at it.

Vembu believes AI has a big role to play here - along with other low-code tooling. Generative AI seems to boost developers most in sandbox settings, where coding errors have less impact than in production. The use of gen AI to help business users generate their own workflow apps is largely still uncharted territory - and will surely involve developers either way for the "last mile" of code validation.

Vembu has seen indications of productivity boosts. He gave us an example local to McAllen, Texas, where ZohoDay was held. Rural Zoho employees in McAllen were brought up to speed on low-code programming tools. They turned around a deliverable to a customer in Mumbai in just three weeks.

Regardless of how AI changes development at Zoho, Vembu says we are dealing with a huge technology shift - but one that brings a major caution. Perhaps we should take this as a short-term positive: despite the hype onslaught, LLMs haven't led to mass unemployment. As Vembu told Wainewright and I:

Clearly we are in the middle of some huge technology wave that's happening right now. But it's also clear now, compared to the hype of, say, a year ago, LLMs haven't completely altered the world or anything. It's not like they've destroyed 10 million jobs right now. It's not happening. Doesn't mean it won't happen. But it just means that we have to bring other technologies into the mix.

Particularly, it's a big no-no for any mission-critical applications to have hallucinations. All of today's Large Language Models have this problem - and the problem doesn't go away easily. It's very fundamental to the way they operate, because they are not producing truth. They are producing truthy statement - truthiness. That's how we should understand it.

They produce plausible-sounding stuff, which could be wrong... An LLM once told me that I have a PhD from Stanford. Pretty close. I have a PhD from Princeton. But you know, these details matter - if I put that in my resume, that I have a PhD from Stanford. You don't want to make such mistakes.

Vembu is of two minds on what needs to happen: either we require a major tech breakthrough, or gen Ai's limitations will be overcome in smaller steps. Readers probably know my take by now, but that's a debate for another day.

Zoho customers told us they are counting on Zoho to come through for them on AI, because they aren't about to build their own data science teams; they are wary of facing down gen AI risks on their own. Zoho even wants to spare customers any role in so-called prompt engineering. (For more on that, see Raju Vegesna's diginomica piece, Prompt Engineer, the hottest new job in tech, is already almost obsolete).

Some might argue that Zoho's AI plans aren't uniquely Zoho. After all, external LLMs are involved, and Zoho is historically fanatical about building their own tools ("Privacy is not a feature" - how Zoho's approach to workplace privacy impacts AI development, and more). They even issued their own "privacy-centric" web browser. At ZohoDay, Zoho's leadership confirmed they still intend to build all their own language models in the long run. I don't see that as a deal breaker for customers. They care more about AI results, while preserving data privacy and minimizing risk. No one questions Zoho's commitment to the latter; ROI will have to be evaluated down the line.

Author's note: my colleague Phil Wainewright reviewed and contributed to this piece. Also see Phil's ZohoDay review, ZohoDay 2024 - Zoho, the tech vendor aiming to put capitalism on a different path.

A grey colored placeholder image