Can an AI judge forecast a drawdown better than four numbers?
AI that reads the news and the filings and tells you what a stock will do is now one of the most heavily sold ideas in retail trading. Almost nobody selling it shows whether the AI adds anything to the simple rule underneath it. We froze that test before a single prediction was made: two AI judges, one criterion, one look.
The window is every pick our screen logs before the open from 26 October through 31 December 2026. The verdict is computed once, on or after 11 January 2027, after the last of those picks has finished grading, and published whether it passes or fails. An interim readout at 150 picks is reported only and can change nothing. Registry id: H-JUDGE1, with the companion rule H-AUTH1.
Why this one
Our own record taught us two things. Nothing we have tested predicts the direction of these stocks: not our own score (Experiment 01) and not any rule in our July sweep. But the size of the drop is partly predictable. Hotter scores draw down further, and a backward-looking check found that picks with a risk event in their recent SEC filings drew down further still. That check cannot prove anything by itself, and its own forward test is running now.
That makes drawdown the honest place to ask whether AI helps. If an AI can’t beat a four-number score at the one thing that is predictable here, the pitch that it can pick winners is in trouble.
It also matches the research. Published work finds that large language models can pull some signal out of financial text, mostly in small stocks and bad news, and that the effect fades as more people use them. The same work finds that their stated confidence runs high and that their stated reasons are often not the real ones. So we grade the AI the way you would grade any new test before letting it make decisions: does it agree with itself, does its 70% mean 70%, and does it add anything to what we already had.
The bar and the two judges
The score (the bar). The plain ThePickLog score (float, relative volume, gap, price), turned into a probability with a model fitted on 476 earlier picks and frozen on 7 October, before this window existed. On those earlier picks it already does a fair job of separating big drawdowns from small ones (AUC 0.667, measured on the data it was fitted to). Both AI judges have to beat it.
Judge 1: a filing reader. Jev, version 1.13.0, pinned. It reads each pick’s recent SEC filings and answers a fixed set of questions about them.
Judge 2: a general AI model from a different company. OpenAI gpt-5.5-2026-04-23, a pinned, dated version, with no tools and no web access, asked for a probability. A second vendor means that when the two judges agree, it is not one model agreeing with itself.
Every reply is checked against the pinned version. A changed model is refused, not quietly accepted. If Judge 2’s version is retired before every pick has been judged, it is reported as insufficient, never swapped for a newer one.
The rule, frozen
What each judge predicts. The probability that the stock trades at least 20% below its opening price at some point that day or in the next five sessions. Our grader already records that for every pick, so nothing new is built on the outcome side. (Whether it closes up that day after costs is also asked, and reported only.)
What each judge sees. Three versions, each asked three times:
A. Numbers only, name hidden. The same four inputs the score uses. Expected to add nothing: it checks whether the AI is just a fancier way of combining the same numbers.
B. Numbers plus filings, name hidden. The same pre-open SEC filings our filing test reads, with the company name and ticker removed. This is the real test.
C. Numbers plus filings, name shown. Identical to B except the company is named. Published research found that hiding a company’s name can make AI forecasts better, because what the model thinks it knows about the company gets in the way. B versus C tests whether that holds in tiny stocks the model probably knows little about.
No look-ahead. Every input existed before the 9:30 open on the trading date: filings accepted before then, plus the screen’s own fields. Both judges are pinned versions released before the window opened, with no web access, so no prediction can have seen its outcome. That is also why predictions may be made in batches after the fact, which the registration calls for at least monthly. The prediction log is append-only: each question is asked once, and the first answer is the record.
Nothing touches the site. No judge’s answer is shown on a pick, used to choose one, or used in a trade. A pass allows advisory use in shadow mode only. Showing it to anyone would need a new registration.
What it has to clear
Frozen at registration. For each judge and each version, all of these, at the single look:
0. Enough evidence. At least 300 graded picks and at least 100 different companies. Short of either, the result is insufficient: not a pass and not a fail.
1. It adds something to the score (the claim). Score plus judge separates big drawdowns from small ones better than the score alone, by at least 0.03 AUC, with an interval that excludes zero, and the improvement is significant after correcting for all six judge-and-version combinations tested (Holm). A version that is missing counts as a failure in that correction, so it can never make the others easier to pass.
2. It ranks picks better than chance, with a 95% interval that treats each company, not each pick, as the unit of evidence.
3. It beats the base rate on accuracy (Brier skill above zero, interval excluding zero).
4. Its probabilities mean what they say once rescaled: fitted on the first 100 picks, then checked on the rest. Its raw probabilities are reported too, and we expect them to be overconfident.
5. It agrees with itself: asked the same question three times, its answers correlate at 0.80 or higher.
Kill conditions. A judge whose repeats correlate below 0.60 is too noisy to be a rater at all. If the two judges agree with each other below 0.20, “AI judgment” here is just one model’s quirks.
When a judge declines to answer, that pick is scored as the plain score’s forecast. Declined picks are never dropped, so a judge that declines everything simply reproduces the score and fails the claim.
The AI’s explanations are not evidence. Each answer logs a one-line reason. Reasons are reported, never used to decide a pass.
When may the AI act alone? (H-AUTH1)
The same window also tests a rule, written down in advance, for when an AI’s call could be trusted without a human. The filing reader’s call is sorted three ways: act alone only when it is far from the base rate and the stock is neither among the thinnest-traded nor carrying a dilution flag; send to a human otherwise; defer to the score when the judge declines. The rule passes only if, on the “act alone” picks, the judge is more accurate than the score with an interval that excludes zero, and its probabilities are calibrated within 10 points. A good rule hands people exactly the cases where the AI is no better. Registered prior: about 1 in 4.
What we expect to happen
Stating a prediction before the data is part of the method, and it is uncomfortable on purpose.
Version A should fail. A language model has no business beating a fitted score at combining the same four numbers. Version B: we put the chance of passing at roughly 1 in 4 for each judge. The score is already strong, and the filing signal we found looking backward is real but modest. B versus C: no direction predicted.
A clean failure would still be one of the most useful results on this site for anyone being sold AI stock analysis. If two AI judges reading the same filings can’t add anything to four numbers anyone can compute, the AI is the expensive part of the product, not the valuable part.
How you’ll be able to check it
The full registration is public and dated: REGISTRATION-H-JUDGE1.md, with every prompt reproduced word for word. Its commit history is the timestamp: you can see it was written before the window opened and every change since. The judge code is kept off the public site, because one version reads third-party SEC filing text, so instead every judge file is frozen by its fingerprint (sha256) in the registration. Change one character and the fingerprint no longer matches. Outcomes come from the same public outcomes.csv every other experiment uses.
Two amendments were made on registration day, both before the first window pick existed. The first makes the grader enforce the window’s end date, which can only shrink the sample to the registered window. The second adds one reported-only analysis (do the judges do better or worse on picks where the raters disagree?) that cannot change any verdict. Both are dated in the registration file.
What this experiment is not
It is not a recommendation to trade anything, and no judge’s output is shown on the site, attached to a pick, or used to choose one. It is not a review of any company that sells AI tools. And whichever way it lands, it describes these two model versions, on these stocks, over this window. It will not be evidence about AI in general.
Get every verdict by email
A verdict lands every few weeks. We'll email you each one, free, the day it publishes — and nothing else. No signals, no picks, no offers. Verdicts stay free and public for everyone either way; this just means you don't have to remember to check back.
One confirmation email first — you're not on the list until you click it. Privacy
Educational and informational only — not investment advice, not a recommendation, and not a broker-dealer. ThePickLog is operated by AMD Ventures, LLC (Florida).
← Experiment 09: volatility contraction (NR7)
Evidence Validation · Score & metrics · Trust & Data
Tools Calculator · Vetting Guide · 7-Check Scorecard
Legal Disclaimer · Privacy · Terms