Created on 2024-10-16 21:39
Published on 2024-10-17 15:35
The effectiveness of using AI tools largely depends on how we communicate with it. The problem faced by a lot of people who use AI tools: is an AI tool not capable of accomplishing your task or is your prompt not good enough to get the results you're looking for?
Your prompt is the AI’s guiding light. The quality of your prompt put into the model significantly affects the quality of the output. The adage ‘garbage in, garbage out’ holds true here as much as it does anywhere else in life.
Last week, I published a simple audit summary tool that extracts specific information from pdf audit reports to create a report summary and then gets AI-generated insights and potential audit ideas to perform in the future based on the report and the key issues identified.
Previous Article: From PDF to Insight
Check out the tool, upload a report from OIG or GAO or any audit organization, and let me know what you think.
It's a simple demonstration that uses AI to extract key information from a pdf document in a standardized format, consistently, and reliably. But even a simple tool like this requires a prompt more complex than, "please summarize this audit report."
Audit Analyzer Tool Prompt
Here's the prompt that is being sent to the AI models to create consistent and reliable audit summaries and AI generated audit ideas. It's written in the python programming language and it includes the report language embedded inside the request.
prompt = f"""
Analyze the following audit report content and extract key information:
{content} # content is the text data from the uploaded pdf.
Please extract and provide the following information in a structured JSON format:
1. "report_title": The title of the audit report.
2. "audit_organization": The federal or state organization that conducted the audit and published the report.
3. "audit_objectives": A list of all the objectives of the audit.
4. "overall_conclusion": The overall conclusion or summary of the audit findings.
5. "key_findings": A list of the main findings from the audit.
6. "recommendations": A list of recommendations based on the audit findings.
7. "llm_insight": Your insights or analysis based on the report content.
8. "potential_audit_objectives": A list of useful potential audit objectives for future audits based on this report.
Ensure your response is a valid JSON object with these exact keys and nothing else.
If you're unsure about any field, use "N/A" for string fields or an empty list for list fields.
"""
For the prompt in the Audit Analyzer tool, I kept it simple by outlining the exact fields and descriptions of the information I needed the model to focus on. By being clear and specific, you can reduce the chances of the AI misinterpreting your request or providing irrelevant information.
I also asked the AI model for the output in a structured format called JSON which is useful to organize data and makes it easy to save or transform for further processing or integration into other systems. This is a powerful use of AI models, extraction of unstructured data and transforming the data into an organized structured format.
Mind your words
Each word you add to your prompt teaches the model how to respond and switching one word in your prompt will dramatically change the character of the output. For instance, 'tell me about Java coffee' will generate information about the coffee produced in Java, whereas 'tell me about Java Indonesia' prompts the AI to discuss the island's geography, culture, and history. The model can answer both questions thoughtfully and coherently at any length you want, and all we have to do is substitute one word.
But let's project out this problem to a tool like the Audit Analyzer tool, if you need a summary of a long report or document, should you simply ask the AI tool to summarize the document? Unfortunately, without direction, the tool might provide a paragraph with a general overview without detail on the issues, or it could get distracted with insignificant aspects of the report, or maybe it provides the summary with a bulleted list.
By specifying 'audit_objectives' and 'key_findings' etc., and the output format, we ensure the AI focuses on extracting the most critical information relevant to our analysis and preparing it to input into downstream processes.
The Prompt is Where You Teach Your Problem
The take home message I want to communicate is, think about your prompt as where you teach the model the information needed to tackle your problem or task, treat it as if you were teaching an intern or new hire to accomplish a similar task for the first time, what information do you think they would need to create the best results?
Invest the time to provide the model with very clear instructions for the task and output you need help with, you can describe the audience for the output, provide examples of quality output that you want repeated, explain the context for how this task fits into a larger process, or whatever you think might make it easier for the model to succeed at this task and I bet you will see impressive results.