1. Write the question before preparing the variants
A useful comparison answers a precise decision: does increasing the maximum length avoid truncated answers? Does a level of detail preserve the necessary contours? Does a larger group finish without error? Choose one question and a single parameter. "Finding the best settings" is too broad for a first session.
Call A the reference and B the variant. Note the exact value changed. Changing the model, the dimensions and the number of passes at the same time can produce a different result, without revealing which modification explains it. Keep these trials for separate comparisons.
2. Freeze a small corpus and the judgment rules
Gather the same inputs for A and B. A dozen cases may be enough for a first bounded decision if it includes the project's difficulties: fine text in an image, a long document, missing data or an unusual format. This number is a practical choice, not a statistical guarantee. Avoid including only the examples that already worked.
For each input, write the success conditions before the test. Distinguish a presentation preference from a blocking defect. In a document assistant, a more pleasant answer may still be rejected if it invents a rule. For a product visual, an appealing color does not make up for the disappearance of an important element.
Also set the minimum number of accepted cases, the forbidden defects and the maximum review time. These project-specific thresholds must remain identical when you examine the results.
| To freeze | Example note | Why it matters |
|---|---|---|
| Inputs | case-01 to case-12, same files and same wording | Compare the same difficulties |
| Quality criterion | Answer verifiable in the document and without invention | Prevent a fluent output from masking an error |
| Critical failure | Missing rule presented as certain | Reject a defect even if the total seems better |
| Variable | A: 128; B: 256 new tokens at most | Attribute the difference to an identified change |
3. Keep the same session and the same dependencies
Keep the model, its version, the extensions, the compute engine and the other settings. Use the same environment and the same GPU, without launching another run between the two variants. Note what you cannot control: visible concurrent activity, a version change imposed on you, or a different load. A comparison affected by these changes requires more caution.
If the tool uses a random seed and lets you set it, keep the value the same for each pair of runs. This makes comparison easier, but does not promise identical output everywhere. PyTorch states that full reproducibility is not guaranteed across versions or platforms, even with the same seed.
Create two folders, A and B, with the same list of identifiers. Keep the raw outputs before any manual correction. A fix done after the fact should appear as an extra operation, with its own work time, and should not be attributed to the setting that produced the initial file.
4. Run a control pass, then compare the pairs
First check that a simple input goes all the way to the retrieved file or response. This pass checks the method. Flag a first load or a special preparation separately: do not compare the full startup of A with a B run in which everything is already loaded.
Run the same cases for A and B. Alternate the order if the tool allows it, then replay a few decisive pairs in reverse order. Check that a single result or an order advantage is not driving the whole conclusion.
Also record errors and rejected outputs. If B fails on a large input, do not remove it from the table to improve its average. You can fix the cause in a separate B2 trial if needed. This way you keep a readable comparison instead of a folder where the variant keeps changing as difficulties come up.
5. Separate the accepted result from the observed time
First review quality at full size or in the tool you actually use. If possible, temporarily hide the names A and B when judging. One person can simply review the outputs in shuffled order, then map them back. Keep the criteria you announced and a short reason for each rejection.
For time, choose a single definition: for example, from launch to the saved complete output. Then separate human review and corrections. A stopwatch around a GPU call is not always a measure of completed computation: with PyTorch/CUDA, operations can be asynchronous. Use the measurement method documented by the tool if you want to isolate the computation.
If the repeats overlap or the timing remains approximate, note “inconclusive difference in duration.” A small gap in a single run does not justify a percentage gain presented as certain.
Example: 128 or 256 new tokens for a document assistant
A small team wants complete answers to twelve fixed questions. It keeps the same model, the same documents, the same instruction and the same GPU. Only the response limit changes: A allows 128 new tokens, B allows 256. In Transformers, max_new_tokens caps generation without counting the tokens in the input text. A token is not a word: the limit does not define an exact number of sentences.
Before the trial, the team sets an illustrative rule: at least ten accepted answers out of twelve and no invented rule. The table below is a fictional scenario for reading the results. It does not describe a measured assistant and does not predict the effect of these values on your model.
A reaches the minimum, B gets more accepted answers but keeps a critical flaw. The decision is to keep A for this scope, with a review of the answers, and to look into the cause of B's flaw before another comparison. Greater length is neither proof of quality nor the certain cause of the invention.
The team replays the questions that distinguish A from B and two questions that already passed. If the flaw also appears with A, it suspends the verdict: the initial corpus had not been enough to establish stability. It keeps a record of both runs instead of keeping only the more favorable one.
| Scenario criterion | A: 128 tokens maximum | B: 256 tokens maximum |
|---|---|---|
| Accepted responses | 10 of 12 | 11 of 12 |
| Critical defects | 0 | 1 invented rule |
| Duration | To note in the real session | To note using the same method |
| Compliance with the stated rule | Yes, in this limited scenario | No, despite the best total |
Avoid conclusions your test does not support
Don't choose B because one of its outputs is spectacular while the others degrade. Don't change the corpus along the way, don't count two retries as two different entries, and don't remove a failure from the denominator. Twelve entries presented remain twelve cases to explain, even if some produced no file at all.
The setting you keep applies to the entries, versions, and criteria examined. It may no longer be suitable for a document ten times longer or for different images. Add representative cases gradually. This progression extends the check; it does not turn a small test into a certification of the system.
Keep a short decision and the next check
Your final file contains the corpus, the two configurations, the outputs, the verdicts by identifier, and a few lines of conclusion. State the setting you kept, the reason, and the limitation: "A kept on these twelve questions; two answers still need review; additional long documents are not covered." Another person must be able to understand the choice without attending the session.
Keep A as a control for the next diagnostic. If no variant meets the criteria, don't keep any of them for delivery; formulate a new hypothesis. Set aside time for copying and checking before launching this follow-up. A test finished with a clear limitation already provides a decision; it does not oblige you to keep going until you find a winner.