We make use of Large Language Models (LLMs) like ChatGPT, Gemini, or CoPilot for daily work. But do you find yourself sometimes questioning the sanity of the responses of these models? You might have meant one thing but their response is something else entirely.

Before you go all out blaming the AI model for giving you incorrect answers, take a look at the prompt or instruction that was given to the model…

One needs to know how to “speak” to the AI model. Only then can we expect it to give us what we want. The art of doing this is called prompt engineering. Let’s learn how to come up with effective prompts that are much more likely to get you the answers that you want.

What is Prompt Engineering?

Imagine you’re having a conversation with a smart assistant, like Siri or ChatGPT. If you ask a vague or unclear question, you might not get the answer you’re hoping for. But if you ask a clear, specific question, you’re more likely to get a useful answer. This is what prompt engineering deals with. It is the practice of carefully crafting the input or “prompt” you give to an AI system to get the best possible response.

Prompt Engineering in QA

In software testing and quality assurance (QA), prompt engineering involves designing the right “questions” or inputs that help AI tools generate useful results.

For example, you can ask AI to test a piece of software. This might give you vague answers. Instead, you can ask it to look for specific types of bugs or generate test cases that cover certain features. The better you design these prompts, the more accurately the AI can help with the testing process.

Types of Prompts

There are different styles or approaches to how you can craft prompts to get the best responses from AI tools. These approaches are designed to make sure the AI understands the task and gives you the most useful results. Here are a few types of prompts you can use:

  • Direct Prompts: This is the most straightforward type of prompt. You give the AI a clear and direct instruction to perform a specific task. Direct prompts are clear, concise, and often start with an action verb. They leave little room for interpretation. They work well when you know exactly what you want the AI to do and can explain it in a few words.
    Examples:
    • “Generate 5 test cases for the login functionality.”
    • “Summarize the key findings of the bug report.”
    • “Translate the following error message into Spanish: [error message].”
    • “List the potential risks associated with this new feature.”
    • “Write a concise definition of unit testing.”
  • Chain of Thought Prompts: In this type of prompt, you guide the AI through a series of steps or reasoning to arrive at a conclusion. You start with one instruction and then break it down into smaller, logical steps. Chain of thought prompts help the AI focus on each part of the task step-by-step. This approach is great when the task requires multiple actions or decisions, as it breaks it down into simpler, manageable parts.
    Examples:
    • “Given the following user story: ‘As a user, I want to be able to reset my password.’ Explain step-by-step how you would approach testing this functionality.”
    • “Analyze the following bug report and explain your thought process in identifying the root cause: [bug report details].”
    • “If a user reports a slow loading time on a specific page, what are the potential reasons? Let’s think step by step.”
  • Few-shot Prompts: In this type of prompt, you give the AI a few examples of what you want it to do, and then ask it to follow the same pattern. This helps the AI model learn from the examples and generate similar responses. This is helpful when the AI needs to understand the format or style of what you’re asking for.
    Examples:
    • Example 1: Input: “Bug: Login button is misaligned.” Output: “Test Case: Verify login button alignment. Expected: Button is aligned correctly. Actual: Button is shifted to the right. Status: Failed.”
    • Example 2: Input: “Feature: Implement dark mode.” Output: “Test Case: Verify dark mode toggle. Expected: Dark mode is enabled when toggled. Actual: Dark mode is not enabled. Status: Failed.”
    • Prompt: Input: “Bug: Image on product page is not loading.” Output: ___________ (You would expect the LLM to generate a similar test case format).

Here’s a link that nicely explains the different ways you can craft prompts – 10 ChatGPT Prompt Engineering Frameworks You Need To Know.

Benefits of Prompt Engineering in QA

Prompt engineering can fastrack your QA endeavors and help you deliver quickly. Here’s how it helps:

  • Faster Testing: By using well-crafted prompts, you can automate many testing tasks that would otherwise take human testers hours or days. This makes the whole testing process faster and speeds up software releases.
  • Improved Test Coverage: With prompt engineering, you can create specific test prompts for different parts of the software, like login pages, shopping carts, or payment gateways. This ensures that AI tests all the important features.
  • Reduced Human Error: By using AI and prompt engineering, you can automate repetitive tasks, which reduces the chances of human error. The AI follows your specific instructions exactly, which helps ensure the tests are done the same way every time.
  • Better Resource Allocation: With prompt engineering, AI can take over many of the routine and time-consuming tasks. This frees up your human testers to focus on more complex tasks, like exploratory testing or addressing the more challenging issues that AI might miss. This results in better use of your team’s time and skills.
  • Faster Feedback: AI can instantly execute tests based on prompts, analyze the results, and provide feedback quickly. This allows development teams to catch and fix issues earlier.
  • Consistency in Testing: When using prompt engineering, the AI follows the exact same instructions every time. This makes your tests repeatable and reliable and reduces any variations between test runs.
  • Scalability of Testing: With prompt engineering, it’s easy to scale up your tests. You can create new prompts for new features, and AI can handle a larger volume of tests without much extra effort. This helps ensure that testing keeps up with the growing complexity of your software.
  • Fewer Resources Needed for Testing: With prompt engineering and AI, you can reduce the number of people needed for repetitive or time-consuming tests. AI tools can take over many of these tasks and reduce the overall resource requirements.

Applications of Prompt Engineering in Software Testing

Within the software testing domain, AI can be a very powerful ally that can do a lot for you.

  • Test Case Generation: With prompt engineering, you can instruct AI tools to generate test cases automatically. This saves time by automatically generating a variety of test cases that might be hard to think of manually, improving test coverage.
  • Bug Detection and Reporting: AI tools can detect bugs quickly and consistently and provide detailed reports to make the fixing process faster. You provide a prompt to instruct the AI to look for specific bugs. For example, you might want to check if clicking a button causes an error. The AI can analyze the software, detect problems, and even generate a bug report with information about what went wrong.
  • Test Data Generation: For testing purposes, you need data that mimics real user interactions. Test data can include things like fake user accounts, product listings, or transaction records. You can use prompt engineering to instruct AI to generate realistic test data for various testing scenarios. This test data can be used to check different parts of the software under various conditions.

Training and Skillset for QA Testers

With AI becoming central to software solutions, the required skill set for QA engineers is also evolving. In the age of prompt engineering, along with an understanding of code structure, QA testers need to have an understanding of:

  • AI behavior
  • Linguistic intricacies
  • Domain knowledge

Training programs are emerging to equip QA professionals with these competencies to make sure they are primed to navigate the challenges AI presents.

Prompt Engineering Best Practices

Here are some simple yet effective techniques to craft good prompts

  • Be Clear and Specific: The more precise your prompt is, the better the AI will understand and respond. Vague or broad prompts can lead to incorrect or irrelevant results. Example: Instead of saying, “Test the login page,” say, “Test the login page by entering a valid username and password, then check if the user is directed to the homepage.”
  • Provide AI Context: Giving the AI enough context ensures it understands the environment or situation in which the task is being performed. Without context, the AI might not know what to focus on. Example: “For an e-commerce site, test the checkout process for users with both a valid and invalid coupon code.”
  • Keep the Prompts Concise: While context is important, you should also aim to make your prompts as short and focused as possible. Long-winded prompts can confuse the AI or make the task unclear. Example: Instead of saying, “Please check the login page thoroughly to ensure that everything is working properly, including the input fields, the error messages, and the submission process,” say, “Test the login page with valid and invalid credentials and check if error messages appear for incorrect inputs.”
  • Test and Refine Your Prompts: The first version of your prompt may not always give perfect results. It’s important to test the AI’s response and refine the prompt as needed. Example: If the AI-generated test cases that missed a key feature, refine your prompt by adding more specific details. For example, if your original prompt was, “Generate test cases for the shopping cart,” but the AI missed the checkout process, update it to “Generate test cases for the shopping cart and checkout process.”
  • Use Structured Formats: Using a structured format helps the AI organize its response in a way that’s easy to understand and follow. Example: “Provide a list of test cases for the contact form with the following details: test case ID, description, expected result.”
  • Ask for Specific Outputs (Measurable Results): You need to be clear about what kind of output or result you want from the AI. Whether it’s a list, a summary, or a detailed report, specifying the output format helps you get the results you need. Example: “Generate a list of performance test cases that check for response time and error rates during high user traffic.”

Conclusion

Prompt engineering in QA is a powerful way to make testing faster, more accurate, and more efficient. By guiding AI tools with clear, specific instructions, you can automate many tasks, reduce human error, and improve the overall quality of the software. Whether you’re looking to save time, increase test coverage, or ensure consistent results, prompt engineering can help achieve all of these goals.