Unlock the power of Local LLMs in Knime workflows and take analytics to the next level!

As pointed out in Databricks' Generative AI Fundamentals course, which I strongly recommend (it's free and really good), AI “is a necessary condition to compete, but it is not a sufficient condition to differentiate yourself in the market.” AI tools and LLMs are widely available and accessible, including local LLMs.

So, how can LLMs be leveraged in a way that is practical and contributes to success?

From what I have seen and experimented with so far, one winning formula seems to be the following:

Local LLMs + Proprietary Data = Comparative and Competitive Advantage
Getting started with running local LLMs is as easy as navigating to Ollama and running the following commands (Linux example, of course):

Download Ollama:

curl -fsSL https://ollama.com/install.sh | sh
Run llama 3.2:1b (example):

ollama run llama3.2:1b
Check the list of downloaded models:

ollama list
Verify Ollama is running locally (that's right, you don't need an internet connection). 

http://localhost:11434/

Here are some advantages of running local LLMs as opposed to relying on proprietary models:
Free
No Contracts
Control
Flexibility
Choices
Security
Anonymity

However, appropriate hardware and know-how is needed for successful executions.

As we are told, there are no perfect models; only trade-offs. So, how do you choose and evaluate various options across family of models such as llama, hermes, mistral, etc. and select the appropriate size within a model (1.5b, 3b, 70b)? I call this horizontal vs vertical comparison, respectively.

Obviously, use cases and hardware requirements need to be considered, but beyond that, in addition to running prompts ad-hoc in the terminal or using the Open Web UI, a systematic way to compare models is needed for analytics purposes, especially if we want to follow the winning formula and take a full advantage of our data.

Running local LLMs in Knime workflows for model comparison for analytics use cases, holding prompt(s) and hardware constant, and systematically capturing outputs has proven incredibly empowering, especially when the workflow is already integrated with proprietary data sources. Needless to mention, data wrangling is often needed prior to running analytics, as well as LLMs, and Knime workflows are great at that.

In this workflow, as a POC, I compare llama3.2:1b vs llama3.2:3b vs deepseek-r1:1.5b for sentiment analysis and topic modeling. The prompt as well as the responses of each of the three models is captured and can then be easily compared and analyzed: 

When new LLMs are released, the workflow provides a flexible way to incorporate and test them. Such framework provides a systematic pipeline to experiment, evaluate, productionalize, and monitor models' outputs. Great for POCs!

Once the appropriate model is selected, the workflow can run local LLMs for analytics use cases such as sentiment analysis, topic modeling, etc. And to visualize the results in Metabase (coming soon as another FOSS Analytics segment) for a seamless FOSS Analytics experience is priceless (literally).

It is all about utilizing tools and building workflows in an integrating and systematic manner. 

This end-to-end FOSS Analytics process flow is running on Linux on Framework hardware - the perfect combination for optimal performance and flexibility.

Join me in democratizing access to analytics capabilities and empowering people and organizations with data-driven insights! 