How to spot made-up numbers in AI research tools

AI tools can produce plausible financial figures with no supporting source, Stock Expert AI founder Sedat Anak says. He recommends checking each number’s source, date and method.

AI research tools are becoming part of financial advisors’ daily work. Advisors use them to draft client letters, prepare meeting briefs, summarize earnings calls and turn stock tickers into research notes. Some of that material is sent to clients.

Language models do not automatically retrieve financial data unless they are connected to a source. By default, they predict text. A revenue growth rate, debt-to-equity ratio or price-to-earnings multiple can look correctly formatted and fall within a plausible range while matching no company filing.

Anak recommends four checks for figures produced by an AI research tool. The first is the source. Each number should identify where it came from, such as a regulatory filing, exchange feed or licensed data provider. A reference to “proprietary data” without more detail does not allow an advisor or client to verify the figure.

The second is the date. Ratios and scores can change when share prices move or companies publish new filings. A number without a reporting or calculation date may be out of date, and readers cannot determine when it was produced.

The third is the calculation method. If a tool gives a company a score or rating, it should disclose the inputs, how they are combined and what the result does not measure. Figures that cannot be explained can be difficult to defend to a client or compliance officer.

The fourth is the origin of the language and figures. Tools should distinguish between commentary generated by a language model and data taken from a documented source. When that distinction is not visible, advisors should treat the figures as potentially generated by the model.

The approach separates data collection from interpretation. Verified sources should provide the figures. An AI model can explain a margin trend, describe a quarter in context or rewrite technical information for a client. It should not create, round or estimate the numbers used in that explanation.

In a September scan of the 3,638 U.S. companies scored by Stock Expert AI, 97.9% of the scores had been recalculated within the previous seven days, according to Anak. The figure illustrates why a calculation date matters for current ratings and ratios.

Clients may also bring chatbot-generated research to meetings. An advisor who can identify where each number came from and when it was calculated can provide information that clients can verify.

Advisors remain responsible for communications sent to clients, regardless of the software used to produce them. FINRA Rule 2210 requires communications to be fair and balanced and prohibits false or misleading statements. The Securities and Exchange Commission’s Marketing Rule prohibits untrue statements of material fact in investment advisers’ advertisements. FINRA said in Regulatory Notice 24-09 that its rules apply when firms use generative AI, as they do with other tools.

Articles by this author