Choose a task, a recipient and a limit
Pick an action your team knows well: finding a procedure, summarizing a report, or preparing a draft. The first deliverable can be a table gathering each question, the raw answer, the expected source, and the human verdict.
Define the behavior when information is missing: ask for clarification or flag an absence. Designate a person who knows the domain and will decide on acceptance. If no one can verify the answers, the test cannot lead to a useful conclusion.
Stabilize the documents before choosing the questions
Prepare the necessary and authorized documents. Remove duplicates, identify the versions in effect, and spot contradictions. Give each document an identifier and keep track of where the expected passages are located.
Transformers distinguishes the answer extracted from the context from the answer generated from that context. In both cases, check that the right passage was actually provided. Placing files in a folder does not mean the software read them all or selected them correctly.
A fictional corpus and twenty questions ready to test
Fictional exercise for an equipment lending workshop: PA and PB are the only sources, unrelated to BriefGPU services. Provide them to the assistant and ask each question without giving it the expected answers.
PA — Photo kit. Send the form at least two calendar days before pickup, with first name, pickup date and return date. Louise confirms the booking. The kit contains one device and two batteries. Pickup and return at counter A, Monday to Friday: pickup from 9 AM to 12 PM, return before 5 PM. Staff count the items on return.
PB — Audio kit. Send the form at least one calendar day before pickup, with first name, pickup date and return date. Malik confirms the booking. The kit contains a microphone and a headset. Pickup and return at counter B, Monday to Friday: pickup from 2 PM to 5 PM, return before 5 PM. Staff count the items on return.
Questions 1 to 5 cover a fact, 6 to 10 compare the procedures, 11 to 15 require clarification, and 16 to 20 cover missing information. Accept a faithful rephrasing. A human correction of the output does not count as a success for the model.
| N° | Question à poser | Réponse ou comportement attendu |
|---|---|---|
| 1 | Who confirms the photo kit? | Louise, according to PA. |
| 2 | What does the photo kit contain? | One camera and two batteries, according to PA. |
| 3 | Where is the audio kit picked up? | At counter B, according to PB. |
| 4 | By what time must the audio kit be returned? | Before 5 PM, Monday to Friday, per PB. |
| 5 | Which fields must be filled in to reserve the photo kit? | First name, pickup date, and return date, according to PA. |
| 6 | Do both forms ask for the same information? | Yes: first name and pickup and return dates, in both PA and PB. |
| 7 | Which kit requires a request further in advance? | The photo kit: two calendar days, versus one for audio. |
| 8 | Which checks and return times are shared? | Item count; return before 5 PM, Monday to Friday. |
| 9 | Does the same person confirm both reservations? | No: Louise for photo, Malik for audio. |
| 10 | On a Tuesday at 10 a.m., can I pick up both kits at counter A? | No: photo at A from 9 AM to 12 PM; audio at B from 2 PM to 5 PM. |
| 11 | Who should I ask to confirm the kit? | Ask whether it is the photo or audio kit. |
| 12 | Which counter should I go to for my kit? | Ask which kit is concerned. |
| 13 | Can I pick up my kit on Tuesday at 3 PM? | Ask which kit: audio yes, photo outside the time slot. |
| 14 | What should my kit contain? | Ask whether it is the photo or audio kit before listing. |
| 15 | I want to pick up a kit on Friday: is one day in advance enough? | Ask which kit: enough lead time for audio, not for photo. |
| 16 | What is the price of the loan? | The corpus gives no price. |
| 17 | What penalty applies for a late return? | No penalty is documented; do not invent one. |
| 18 | What is the camera resolution? | This characteristic is not present in PA. |
| 19 | Can someone else pick up the hardware on my behalf? | The option of pickup by a third party is not specified. |
| 20 | Who steps in for Louise when she is away? | No replacement is indicated. |
Lock in the configuration and start with a single request
Choose a compatible model and tool, then note their versions and license. Record the prompt, the document selection and the settings. In Transformers, max_new_tokens limits the new tokens produced; sampling influences the choice of output. Keep these parameters to compare the runs.
The memory required includes more than the model file: working data and the generation cache also take up space. Transformers in particular describes the cost of the cache as the context grows longer. Start with a single request and then check the longest inputs.
Increasing users and response length at the same time would mix quality and capacity. The 24, 48 or 80 GB benchmarks do not promise that a model or several requests will fit.
Worked example: two procedures, sixteen acceptable answers
Suppose an evaluation of the two procedures PA and PB with the grid above. An answer is acceptable if it respects the source and the expected behavior. Asking for clarification can therefore count as a success.
The table is hypothetical: no model was run to produce it. Sixteen acceptable cases out of twenty gives 80% for this specific set, without estimating reliability on other requests. An invented rule can weigh more heavily than an incomplete answer.
In this scenario, the team first looks at fabrication and ambiguities. A missing passage means revisiting the selection; a passage that is present but misinterpreted points to the prompt or the model. More memory does not automatically fix these errors.
| Famille, 5 questions chacune | Acceptées | À examiner |
|---|---|---|
| Simple information | 5 | None in this example. |
| Cross-references | 4 | One partial answer. |
| Ambiguous requests | 3 | One unjustified assumption and one unnecessary refusal. |
| Missing information | 4 | One invented rule. |
| Total | 16 out of 20 | 4 unacceptable cases. |
Keep a record that explains each verdict
Keep the raw answer, the passages provided, the references, the verdict and its reason. Distinguish a poorly chosen document, an incorrect interpretation and an unsuitable form: "the model is sometimes wrong" does not point to any correction.
Then assess the observed time and the review effort, separating initial loading from subsequent runs. If generation varies, repeat a few difficult questions and keep all the outputs. Choosing only the best would hide the failures; repeating the same inputs does not cover all real-world situations.
- Identifier and exact wording of the question.
- Version of the file and passages actually provided.
- Raw answer and references displayed.
- Human verdict and short justification.
- Test configuration and observations on time or memory.
Change one variable, then check what already worked
Change one identifiable variable: document, passage selection, prompt or model. Replay the failures and several questions that were already acceptable. A fix can degrade another case; keep the old outputs to see it.
Do not keep tuning the prompt until it incorporates every answer from the grid. Then prepare a few new questions: they check whether the fix serves the need beyond the twenty known formulations. This check remains limited to its own conditions.
Decide what comes next and export the test file
Conclude on a precise scope: retrieve a reference, prepare a reviewed draft or ask for clarification. Name the excluded cases. A successful demonstration does not guarantee correct answers for the whole company.
Continue with a supervised trial, fix one specific cause or stop if reviewing costs more than the original work. Before running several simultaneous requests, validate the individual flow and keep a known subset to spot changes.
Save authorized documents, grid, outputs, settings and decision. The rental must cover evaluation and recovery, beyond generation. BriefGPU leaves the choice of tools and processing to the client, without inspecting their content; your control of the results remains necessary.