KHAGAN AYDIN
Writing

How to Write Prompts That Get Reliable Answers

A language model answers the question you typed, not the one you meant. Seven habits close the gap.

· 5 min read

ChatGPT opened to the public two weeks ago, and most people in technology have already tried it. The reactions split into two groups: people who say it is remarkable, and people who say it gives vague or wrong answers. Very often both groups are using the same tool. The difference is the prompt.

A language model does not know what you want. It reads your text and predicts the most likely useful continuation. If the prompt is vague, the most likely continuation is a vague, general answer. If the prompt is specific, the answer becomes specific. Writing a good prompt is not a trick. It is the same discipline as writing a good ticket for an engineer or a good brief for a contractor.

Here are seven habits that make the biggest difference.

1. Say who the answer is for and what it is for

The same question needs a different answer for a developer, an executive and a customer. Tell the model.

Weak:

Explain cross-site scripting.

Strong:

Explain cross-site scripting to a product manager who has no
security background. They need to decide whether a fix should
go into this sprint. Keep it under 150 words and end with one
sentence on business risk.

The second prompt sets the reader, the purpose, the length and the ending. The model no longer has to guess any of them.

2. Give it the material instead of making it guess

A model trained on public text knows nothing about your company, your code or your customer. When you need an answer about a specific document, paste the document and tell the model to use only that.

Below is our password reset procedure. List every step where an
attacker could take over an account. Use only the procedure below.
If the procedure does not say how a step works, write "not
described" rather than assuming.

PROCEDURE:
...

The last instruction matters. Without it, the model fills gaps with what is typical, and typical is not the same as what your system does.

3. Set the limits

State the length, the format and what to leave out. Models follow these instructions well, and they save you a round of editing.

Summarize this incident report in five bullet points.
Each bullet must be one sentence.
Do not include names of employees.

If the answer will be pasted into another system, ask for that system’s format directly: a table, a numbered list, CSV, or JSON with named fields.

4. Show an example of the answer you want

When the task is to sort, label or rewrite many items in the same way, one or two examples work better than a long description. This is often called few-shot prompting.

Classify each support ticket as BILLING, ACCESS or BUG.

Ticket: "I was charged twice this month."
Category: BILLING

Ticket: "The export button does nothing in Firefox."
Category: BUG

Ticket: "I can't log in after resetting my password."
Category:

The examples show the labels, the format and the level of judgment you expect, all at once.

5. Ask for the steps when the task needs reasoning

For anything with logic, calculation or several conditions, ask the model to work through the problem step by step before giving the answer. Research published this year shows that this simple request improves accuracy on reasoning tasks, and it gives you something better than a bare answer: steps you can check.

A user has the roles "editor" and "billing-viewer". Using the
permission table below, can they export invoices? Go through the
table step by step, then give a final yes or no.

Check the steps, not just the conclusion. Models still make arithmetic and logic mistakes, and they make them confidently. Any number that matters should be recalculated by you or by code.

6. Give it permission to say “I don’t know”

By default, a model tries to answer. If the honest answer is “this is not in the text” or “there is not enough information”, say that this answer is acceptable.

If the contract below does not state a termination notice period,
reply "not stated". Do not infer one from typical contracts.

This single line removes a large share of invented answers in document work.

7. Test the prompt before you trust it

A prompt that will run inside a product or a team process is code, and it deserves the same discipline. Collect ten to twenty real inputs, including the awkward ones, write down the answer you expect for each, and run the prompt against all of them. Run the set again every time you change a word.

import openai

PROMPT = """Classify the support ticket as BILLING, ACCESS or BUG.
Reply with the category only.

Ticket: {ticket}
Category:"""

CASES = [
    ("I was charged twice this month.", "BILLING"),
    ("The export button does nothing in Firefox.", "BUG"),
    ("My reset link expired before I could use it.", "ACCESS"),
    ("Why is my invoice in euros? I'm in Texas.", "BILLING"),
]

failures = 0
for ticket, expected in CASES:
    result = openai.Completion.create(
        model="text-davinci-003",
        prompt=PROMPT.format(ticket=ticket),
        temperature=0,
        max_tokens=5,
    )
    answer = result["choices"][0]["text"].strip()
    if answer != expected:
        failures += 1
        print("FAIL:", ticket, "| expected", expected, "| got", answer)

print(failures, "of", len(CASES), "cases failed")

Ten minutes spent on a test set saves you from discovering, in front of a customer, that a small wording change broke a case that used to work.

Habits for working sessions

  • One task per prompt. “Summarize this, translate it and suggest a title” produces three weaker results than three separate prompts.
  • Correct, do not restart. If the first answer is close, tell the model exactly what to change: “Shorter, and remove the second paragraph.”
  • Keep the prompts that work. Save good prompts in a shared document. Your team will reuse them more than you expect.
  • If you use the API, lower the temperature for factual work. A temperature between 0 and 0.3 makes answers more consistent. Higher values help with brainstorming, not with facts.

What a good prompt cannot fix

A better prompt improves the answer you get. It does not make the model a source of truth.

  • It does not know recent events. ChatGPT’s knowledge of the world largely stops in 2021.
  • It can state false things fluently. Names, numbers, quotes and references are the most common errors. Verify anything you will act on.
  • It does not keep your secrets. Do not paste passwords, keys, customer data or confidential code into a public tool.

Treat the answer as a strong first draft from a fast, well-read assistant who has never worked at your company and does not know when it is wrong. Give it clear instructions, the right material and room to say “I don’t know”, and then check its work.

A template to start from

Role: You are helping [who] with [what decision or task].
Material: Use only the text below. If something is not in it, say "not stated".
Task: [one clear task].
Format: [length, structure, fields].
Avoid: [what to leave out].
Example: [one example of a good answer, if the task repeats].

TEXT:
...