If you are new to Semantic Kernel, I would point you to one of my earlier posts. In this post I want to show how you can change the timeout values when using Semantic Kernel.
The power of Semantic Kernel is that it gives you the ability to interact with multiple (large language) models in an uniform way. You interact using C#, Java or Python with the Semantic Kernel SDK and behind the scenes it will do the necessary API calls to OpenAI, Azure OpenAI, Hugging Face or a local OpenAI compatible tool like Ollama.
Of course as we are interacting with an API behind the scenes, it can happen that the API doesn’t return any results in time and that we get a timeout exception.
The operation was cancelled because it exceeded the configured timeout.
Let me share how I fixed it…
Use a custom HttpClient
One option you have is to explicitly pass an HttpClient
instance when creating the Semantic Kernel instance:
HttpClient client = new HttpClient(); | |
client.Timeout = TimeSpan.FromMinutes(2); | |
var semanticKernelBuilder = Kernel.CreateBuilder() | |
.AddOpenAIChatCompletion( // We use Semantic Kernel OpenAI API | |
modelId: "phi3", | |
apiKey: null, | |
endpoint: new Uri("http://localhost:11434"), | |
httpClient: client);// With Ollama OpenAI API endpoint |
Retry when a timeout happens
If the timeout typically happens because the AI platform that you are targeting reacts slow sometimes, it is maybe a better idea to configure a retry policy instead of changing the timeout value. This can be easily done in .NET Core by adding a ResilienceHandler
:
//Found here: https://github.com/microsoft/semantic-kernel/discussions/3412 | |
builder.Services.ConfigureHttpClientDefaults(client => | |
{ | |
client | |
.AddStandardResilienceHandler() | |
.Configure(o => | |
{ | |
// Combine checks for status codes and timeout into one delegate to set as the ShouldHandle policy | |
o.Retry.ShouldHandle = args => | |
{ | |
// Check if the response has a status code that we should handle | |
var isStatusCodeToRetry = args.Outcome.Result?.StatusCode is HttpStatusCode.Unauthorized | |
|| args.Outcome.Result?.StatusCode is HttpStatusCode.BadGateway; | |
// Check if the result was a timeout (typically when the result is null and an exception is a TaskCanceledException) | |
var isTimeout = args.Outcome.Exception?.InnerException is TaskCanceledException; | |
// Combine the checks | |
return ValueTask.FromResult(isStatusCodeToRetry || isTimeout); | |
}; | |
}); | |
}); | |
}); |
More information
Build resilient HTTP apps: Key development patterns - .NET | Microsoft Learn