The art of simplicity

GitHub Copilot CLI is my 'go-to' coding agent when I work directly from your terminal. It understands my codebase, proposes edits, runs commands, and helps me move faster without leaving the command line. As I care about privacy, offline workflows, or custom model experimentation, I decided to try Copilot CLI entirely on local LLMs using Ollama. No cloud dependency. No API keys. Just my machine, a local model and my workflow. In this post, I’ll walk through how to set it up, and how to use it effectively. Why combine Copilot CLI with Ollama? Copilot CLI gives you a powerful agentic interface for your codebase. Ollama gives you a fast, local model runtime with support for dozens of open models. Together, you get: Local-first AI coding:  keep your code and prompts on your machine Predictable performance:  no rate limits or network delays Model flexibility : swap between Qwen, Llama, Mistral, Gemma, and more Agentic workflows:  Copilot CLI can edit...

Ollama remains my go to tool to run LLM’s locally. With the latest release the Ollama team introduced a user interface. This means you no longer need to use the command line or tools like OpenWebUI to interact with the available language models. After installing the latest release, you are welcomed by a new chat window similar to ChatGPT: Interacting with the model can be done directly through the UI:   A history of earlier conversations is stored and available:   You can easily switch between models by clicking on the model dropdown. If a model is not yet available locally, you can download it immediately by clicking on the Download icon: If you need a bigger context window, you can now change this directly from the settings: Some other feature worth mentioning are the file support (simply drag and drop a file to the chat window) and multimodal support: All this makes the new Ollama app a good starting point to try and interact with the available LLM's local...

I'm on a journey discovering what is possible with the Microsoft.Extensions.AI library and you are free to join. Yesterday I looked at how to integrate the library in an ASP.NET Core application. Today I want to dive into a specific feature; tool calling. This post is part of a blog series. Other posts so far: Part I – An introduction to Microsoft.Extensions.AI Part II – ASP.NET Core integration Part III –Tool calling (this post) What is tool calling? With tool calling you are providing your LLM with a set of tools (typically .NET methods) that it can call. This allows your LLM to interact with the outside world in a controlled way. In Semantic Kernel these tools were called ‘plugins’ but the concept is the same. To be 100% correct it is not the LLM itself that is calling these tools but the model can request to invoke a tool with specific arguments (for example a weather tool with the location as a parameter). It is up to the client to invoke the tool and pa...

Yesterday I explained how we can create and upload our own language models in Ollama through the usage of a modelfile. I explained the modelfile format and the different building blocks that can be used to define and configure a model. Today I want to continue on my previous post by explaining how to use OpenWebUI instead of doing everything by hand. Start by opening OpenWebUI (checkout my previous post on how to get it up and running): Click on the Workspace section on the left:   Click on the + button in the Models section on the right: Start editing your modelfile:   Hit Save & Create at the bottom:   After saving the new model, you can immediately test it: More information Explore and test local modals using Ollama and OpenWebUI Models | Open WebUI

If you want to create and share your own model through Ollama or tweak an existing model, you need to understand the Ollama Model file. The model file is the blueprint to create and share models with Ollama. Understanding the Ollama model file Let us first  have a look an existing model file to give you an example. Therefore you can use the following command: ollama show <modelname> --modelfile Let’s give it a try: ollama show phi4:latest --modelfile # Modelfile generated by "ollama show" # To build a new Modelfile based on this, replace FROM with: # FROM phi4:latest FROM C:\Users\bawu\.ollama\models\blobs\sha256-fd7b6731c33c57f61767612f56517460ec2d1e2e5a3f0163e0eb3d8d8cb5df20 TEMPLATE """{{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} < |im_start|>{{ .Role }}<|im_sep|> {{ .Content }}{{ if not $last }}<|im_end|> {{ end }} {{- if and...

The introduction of the DeepSeek R1 model has sent shockwaves through the AI industry, challenging established norms and redefining the economics of AI development. This model has demonstrated that we can train AI models more cost-effectively and in an environmentally friendly manner, without sacrificing performance. By leveraging innovative techniques, DeepSeek has shown that it's possible to achieve remarkable results without the exorbitant costs and environmental impact typically associated with AI training. I think this is good news for all of us. As a big believer in the advantages that LLMs has to offer, I always feel somewhat uncomfortable knowing the environmental impact that these models have both during training and execution. DeepSeek has shown us that a different path is possible, providing a better balance between productivity and (environmental) cost. My hope is that other AI players will now re-evaluate on how to move forward and start applying the same techniques ...

A  few weeks ago I got contacted by someone(Hi Chris!) who was trying to get my Semantic Kernel demo's up and running on his machine. Chris tried to get the application up and running but got some error messages. I used his input to improve the readme file and updated the main branch to simplify the getting started experience. However there was one specific error he shared that I want to talk about a little more. Here is the screenshot he shared with me:   The reason that he got this error is because ‘auto function calling’ was enabled in the code and the model he was using didn’t support this feature. Remark: I updated the code to disable auto function calling after I got his email. A good excuse to talk a little more about this feature… What is (auto) function calling in Semantic Kernel? With function calling, you give the LLM the option to interact with your existing code. You can do this quite explicit as I explained in my OllamaSharp post but with Semanti...

After announcing Phi-4 last month, it is now also available on Ollama to use locally. Phi-4 is a "small" language model that performs at the same level or even better than some of the larger language models. The latest incarnation in Microsoft's Phi series, took a big step forward in tasks requiring complex reasoning and problem solving, making it a good candidate to use in a (multi-)agent solution. One area where the model is particularly good is in math.It outperformed larger models in the November 2024 AMC competitions, proving its real-world application potential.   Time to download the model through Ollama and give it a try: ollama pull phi4 More information microsoft/phi-4 Ollama microsoft/phi-4 · Hugging Face microsoft/phi-4 Azure AI Foundry Introducing Phi-4: Microsoft’s Newest Small Language Model Specializing in Complex Reasoning | Microsoft Community Hub

Retrieval-augmented regeneration, also known as RAG, is an NLP technique that can help improve the quality of large language models (LLMs). It allows your AI agent to retrieve data from external sources to generate grounded responses. This helps avoid that your agent hallucinates and returns incorrect information. There are multiple ways to retrieve this external data. In this post I want to show you how to generate vector embeddings that can be stored and retrieved from a vector database. This means we first need to decide which database to use. The list of options keeps growing (even the new SQL server version will support vector embeddings out of the box). As we want to demonstrate this feature using Semantic Kernel, we need to take a look at one of the available connectors . Qdrant Vector Database I decided to use Qdrant for this blog post. Qdrant is an AI-native vector database and a semantic search engine. You can use it to extract meaningful information from unstructure...

I talked about Structured Output before in the context of Semantic Kernel. Structured Outputs is a feature that ensures the model will always generate responses that follow a supplied JSON Schema, so you can process the responses in automated fashion without worrying about getting invalid JSON back. Recently support for Structured Output was announced by Ollama . In this post I want to show you how you can use this in combination with OllamaSharp , the C# client for Ollama. Using Structured Ouput in OllamaSharp Remark: Make sure you have latest Ollama version running on your local machine before you continue. Add the OllamaSharp client to your project: dotnet add package OllamaSharp Now let’s first define our response model: Afterwards we need to initiate a new OllamaSharp client instance: We create a new request object. Notice that we specify a JSON schema object based on the Recipe model we created earlier If we now invoke the appl...

I talked about Ollama before as a way to run a Large-Language-Model(LLM) locally. This opens the door to try out multiple modals at a low(er) cost (although also a lower performance) and could be interesting if you are not allowed to share any data with an AI provider. For example you are a developer but your employer doesn’t allow you to use AI tools for that reason. If this is a use case that is relevant for you, than I have some good news for you. With the latest version of JetBrains AI Assistent(available in JetBrains Rider but also other IDE’s) you can now use Let me show you how to use this: Open JetBrains Rider(or any other IDE that integrates the JetBrains AI Assistent) Hit Ctrl-Alt-S or go to the Settings through the Settings icon at the top right   Go to the AI Assistant section under Tools Check the Enable Ollama checkbox. A warning message appears about Data Sharing with Third-Party AI Service Providers. Click OK to con...

I'm a big fan of Ollama as a way to try and run a large language model locally. Today I got into trouble when I tried to connect to Ollama. When I tried to run Ollama through ollama serve I got the following error message: time=2024-12-02T21:15:55.398+01:00 level=ERROR source=common.go:279 msg="empty runner dir" Error: unable to initialize llm runners unable to locate runners in any search path I was able to fix the issue by going the AppData\Local\Ollama folder. There inside updates I found a new(er) version that I installed manually by executing the OllamaSetup.exe. After the setup completed, Ollama was running again as expected. More information Ollama

As Semantic Kernel could work with any OpenAI compatible endpoint, and Ollama exposes it language models through an OpenAI compatible API, combining the 2 was always possible. However not all features of Ollama were accessible through Semantic Kernel. With the recent release of a dedicated Ollama connector for Semantic Kernel, we can start using some of the more advanced Semantic Kernel features directly targetting Ollama deployed models. The new connector is using Ollama Sharp(I talked about it in this post ) so you can directly access the library if needed. Giving the new connector a try… Create a new Console application and add the Microsoft.SemanticKernel.Connectors.Ollama NuGet package: dotnet add package Microsoft.SemanticKernel.Connectors.Ollama --version 1.21.1-alpha Now instead of creating a Semantic Kernel instance, we can directly create an OllamaChatCompletionService instance: The remaining part of the code remains the same as with the default ...

Yesterday I talked about OllamaSharp as an alternative (to Semantic Kernel) to talk to your Ollama endpoint using C#. The reason I wanted to directly use Ollama and not use Semantic Kernel was because I wanted to give the recently announced Ollama Tool support a try. And that is exactly what we are going to explore in this post. Keep reading... Tool support Tool support allows a model to answer a given prompt using tools it knows about, making it possible to interact with the outside world and do things like for example calling an API. It makes your model a lot smarter as it can start using information that was not part of the originally trained model and do more things than just returning a response. Remark: A similar feature exists in Semantic Kernel through the concept of Plugins but as far as I’m aware the Plugins are not using the Ollama tools support (yet).  When writing this post I noticed that a new Ollama connector was released for Semantic Kernel which uses Ollam...

If you are a C# developer and want to interact with Ollama(which allows you to interact with Large Language Models locally), the easiest solution is to use Semantic Kernel. This is possible because Ollama exposes an OpenAI compatible API. However I wanted to try some Ollama specific features that were not yet exposed through Semantic Kernel. Does this mean that I can no longer use C#? Remark: While writing this post I noticed that an Ollama connector was released for Semantic Kernel that also uses OllamaSharp behind the scenes. The good news is you still can. Thanks to OllamaSharp you get .NET bindings for the Ollama API . Getting started Let’s write a simple demo application to try OllamaSharp: Create a new Console application: dotnet new console -o OllamaSharpDemo Add the OllamaSharp Nuget package: dotnet add package ollamasharp Now we can start writing our code. First create a new OllamaApiClient instance and specify the model we’ll use: Next ...

In a previous posted I introduced you to Continue in combination with Ollama, as a way to run a fully local AI Code Assistant. Remark: This post is part of a bigger series. Here are the other related posts: Part 1 – Introduction Part 2 -  Configuration Part 3 – Editing and Actions Part 4 (this post) -  Learning from your codebase Today I want to continue by having a look at how continue can learn from your codebase and provide suggestions based on that. But before I can show you this feature we first need to download an embedding model. Embedding models are models that are trained specifically to generate vector embeddings : long arrays of numbers that represent semantic meaning for a given sequence of text. These arrays can be stored in a database, and used to search for data that is similar in meaning. We’ll use the nomic-embed-text embeddings, so let’s download that one: ollama pull nomic-embed-text Now we need to update the Continue configura...