Testing a Financial Document Classification Prompt
A prompt can make a financial document classifier easier to use. It cannot establish that the classifier is reliable. The useful work is to define the labels, require evidence, and test the output on documents with known answers.
Make the task narrow
Suppose the input is text from one filing or company communication. The desired label is one of 10-K, 10-Q, 8-K, Proxy, Earnings transcript, or Press release. Include unknown for fragments that lack enough evidence. Do not ask the model to infer sentiment, risks, and valuation in the same call; each is a separate task with a separate evaluation set.
A prompt for this task could read:
Classify the document using only its supplied text.
Allowed labels: 10-K, 10-Q, 8-K, Proxy,
Earnings transcript, Press release, unknown.
Return JSON with exactly:
{"label": "...", "evidence": "..."}
The evidence must be a short, exact quote from the input.
If the text does not distinguish the labels, use unknown
and explain the missing evidence. Do not guess from company name.An exact quote makes the decision inspectable. It is not proof that the label is correct: the quoted phrase may appear in more than one document type. Parse the JSON, check that label is allowed, and verify that evidence occurs in the source text. Reject malformed or unsupported output rather than silently accepting it.
Try cases that can defeat a tidy example
Build a small labelled set before revising the prompt. Include ordinary filings and confusing excerpts: an 8-K that mentions a 10-K, a press release quoting an earnings call, a truncated filing cover page, and a proxy statement with financial tables. In each case record the source, the correct label, and which lines justify it. A three-row illustration of the expected behaviour:
| Input excerpt | Expected label | Reason |
|---|---|---|
FORM 10-Q ... quarterly period ended | 10-Q | The form and period agree. |
FORM 8-K ... Item 2.02 Results of Operations | 8-K | The form header identifies the filing, even though it reports earnings. |
Revenue increased 12% this quarter | unknown | No document type is identified. |
These are illustrative snippets, not a measured benchmark. For an actual evaluation, use full, appropriately licensed documents with independently checked labels. Split by company and date where possible so near-duplicate templates do not leak between development and evaluation sets.
Measure the errors that matter
Count correct labels per class and inspect a confusion matrix. Track unknown decisions separately: abstaining on a clear 10-Q is a different failure from confidently calling an 8-K a press release. Check evidence quotations and JSON validity too. Keep the prompt and model version fixed while scoring a held-out set, and retain failed examples when changing either.
If the next task extracts numerical claims, use code to parse and calculate values from the quoted source. A prompt asking for “confidence: high” does not provide calibrated confidence, and a plausible percentage is not a verified calculation.
The deliverable is a repeatable evaluation, including misses and unsupported answers. The prompt is only one input to it. For the parallel problem of selecting a winning result from many attempts, see Deflated Sharpe Ratios: Start with the Selection Problem.