One of the challenges when integrating a large language model into your backend processes is that the response you get back is non-deterministic. This is not a big problem if you only want to output the response as text, but it can be a challenge to process the response in an automated fashion.
Prompting for JSON
Of course you can use prompt engineering to ask the LLM to return the response as JSON and even provide an example to steer the LLM, but still it can happen that the JSON you get back is not formatted correctly.
Here is a possible prompt:
A trick that also can help as mentioned in the Anthropic documentation is to prefill the response with a part of the JSON message.
JSON mode
Although the techniques above will certainly help, they are not fool proof. A first improvement on this approach was the introduction of JSON mode in the OpenAI API. When JSON mode is turned on, the model's output is ensured to be valid JSON, except for some edge cases that are described in the documentation.
To use this technique you need to update the execution settings of Semantic Kernel:
Structured output
With structured output we can take this approach one step further and specify an exact JSON schema that we want to get back.
We have 2 options when using this approach. Either we specify the schema directly:
Or we let Semantic Kernel automatically generate the model based on a provided type:
Remark: Structured output is only supported in the more recent language models.
More information
Entity extraction with Azure OpenAI Structured Outputs | Microsoft Community Hub
Using JSON Schema for Structured Output in .NET for OpenAI Models | Semantic Kernel
Prefill Claude's response for greater output control – Anthropic