Testing of the LLM vs Testing with the LLM

I don’t need to talk about the importance of AI and Generative AI in the world of today. All of us are on the path of transformation. From our legacy testing methodology to bringing the flavors of AI and Gen AI.

Be it Cloud or an On Premise application, doesn’t matter. AI and now its sub — GenAI is everywhere. But the catch is — you also need to test your AI models. And that’s where the knowledge of AI will play its part.

This article is divided in two sections -

Section 1 : Testing the LLM : In this sections we will talk about the requirements that need to be considered for testing an LLM model and ensuring that it gives the right results.

Section 2: Testing with the LLM : In this section , we will discuss on how LLM models / aka Gen AI is helping us in our daya to day Software testing processes.

Section 1 : Testing of the LLM

When it comes to testing of an LLM model, below are some of the considerations :

1. Prompt responses : Accurate prompt response is a must. Thoughit is very difficult to have 100% verification of this feature as everytime a new response will be generated by the model for the same prompt. But what can be done is the %accuracy of different responses over certain round of responses. Test your prompt responses for hallucinations, chain of thoughts, context relevance responses.
2. Response time : Verification of the performance of the prompt…