Watch how an assistant handles a long document and practise a method that keeps its answers accurate.
Hafiz is a procurement executive at a manufacturing firm. He wants to know whether he can rely on an assistant to read a supplier's annual report for him, so he tests one before he needs it.
The situation is common. A colleague sends you a long document, perhaps a tender document, a supplier's annual report or a long government guide, and asks you to pull out what matters by a deadline. Say it is a hundred-page document and they need it by Friday. The obvious move is to upload it to an assistant and ask it questions. The exercise below shows how far those answers can be trusted. You need about thirty minutes, an assistant that accepts file uploads or long pasted text, and the scoring table provided below the lesson.
Use a public document only. Confidential material stays out of this test, and lesson 8.3 covers what you should never upload to an assistant. The document should be long, ideally fifty pages or more, and one you can read yourself. Good choices are the annual report of an SGX-listed company, downloaded from its investor relations page, or a detailed guide from a government agency such as IRAS, HDB or the Ministry of Manpower.
Before you open the assistant, write five questions and find each answer yourself, noting the page number. Do this first because if you read the assistant's reply before you know the right answer, its version tends to become yours. Spread the questions across the document:
two answered in the first few pages one answered near the end one answered in the middle, ideally in a table or an easy-to-miss paragraph one the document does not answer but plausibly might
That last one is your unanswerable question: a question the document does not answer but that sounds as if it might.
Hafiz picks an 80-page annual report from a listed logistics company that is one of his suppliers. His opening questions ask for total revenue and the number of countries the company operates in. His question near the end asks for the auditor's name. His middle question asks how many staff the company employs in Singapore, which sits in a table on page 41, deep in the workforce section. His unanswerable question asks about plans to open a Vietnam office.
Start a new chat, upload the whole document, and ask the five questions one at a time. Add the same request to every question: quote the exact passage you relied on, give the page number, and say so if the document does not answer the question. Paste each reply into the scoring table straight away.
Lesson 5.2 explained that long material gets uneven attention. This step tests that on a real file, so watch the middle question and the unanswerable question most closely.
Hafiz's two opening questions come back right, with real quotes on the correct pages. The auditor's name, near the end, is also right. For the Singapore headcount, the assistant gives the group's total headcount from the opening pages instead. The quote is real, but it answers a different question. For Vietnam, the assistant says the company "intends to expand selectively in Southeast Asia, including Vietnam" and gives a quote.
For each answer, search the document in your PDF reader for a distinctive phrase from the quote. You will get one of three results. The phrase is there word for word on the page given. A close but altered version turns up, sometimes on a different page. Or the phrase is not in the document at all, which means the model wrote text that only looks like a quote. That last case is a fabricated quote. Record which result you got for each answer.
Then score each answer against the page you found in step 1. Right means correct and complete. Partly right means correct but missing a condition, figure or qualification that changes how you would use the answer. Wrong means wrong, and a confident answer to the question the document does not cover also counts as Wrong.
Two things catch people here. A real quote can still answer a different question from the one you asked. And a model can stretch a real phrase from the document to fit your question.
Hafiz sees both. His Singapore headcount answer has a genuine quote, but it is the group figure, so he scores it Wrong. For Vietnam, his search finds "expand selectively in Southeast Asia" but no mention of Vietnam anywhere. The model extended a real phrase to fit his question, so that answer is Wrong too. His tally is three Right (revenue, countries, auditor) and two Wrong (the middle question and the unanswerable question).
This step tests one idea: does giving the model less to read, but the right part, improve the answer? Open a new chat. Upload or paste only the section that contains the middle question's answer, a few pages at most. Ask the same question with the same request for a quote, a page number and a statement if the document does not answer. Compare the result with the whole-document answer. If you have time, repeat this for any other question you scored Partly right or Wrong.
Hafiz uploads pages 38 to 44 alone. The Singapore headcount comes back right, with an exact quote from the table.
Finish by writing your own rule for using assistants on long files, based on your results. Hafiz's rule is this: for any figure from a long file, give the assistant only the section it is in, and do not use a quote he has not found with search.
Your own rule may come out differently, because it should come from what your five answers show. Score them honestly, including the ones that were nearly right, since partly right is where most of the expensive mistakes hide.
Mark each of the five answers right, partly right or wrong, note whether each quote was real, and write one rule you will follow with long files from now on.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).