Financial AI is changing faster than the way we measure it.
For years, evaluating financial language models was relatively straightforward. A model could be asked to retrieve a figure from a filing, calculate a ratio, summarize an earnings report, explain a financial concept, or produce an investment memo. These tasks remain useful because they test whether a model can read financial information, reason over documents, and produce outputs that resemble the work of an analyst.
But they increasingly describe only part of what financial AI is becoming.
The current generation of financial systems is moving beyond isolated question answering. Models are gaining access to professional data sources, research tools, financial workflows, portfolio information and, increasingly, trading infrastructure.
Kimi’s recently launched financial industry solution is a clear example of this shift. It combines more than ten professional data sources with dedicated financial Skills covering areas such as financial modeling, research reports, portfolio reviews and earnings analysis. Its stated objective is not simply to answer financial questions, but to connect data retrieval, analysis, modeling and professional deliverables into an end-to-end workflow.
A similar transition is taking place closer to execution.
Interactive Brokers now allows users to connect AI systems including ChatGPT, Claude and Grok to authorized brokerage-account data through MCP. These systems can analyze holdings, cash balances, P&L, historical transactions, options data and risk exposures, and can generate trade instructions for the user to review before submission.
These developments suggest that the unit of financial AI is changing.
It is moving from:
answering a financial question
toward:
performing a financial task.
That distinction has significant implications for how financial AI should be evaluated.
Financial Knowledge Is Not the Same as Financial Work
A model may understand what revenue growth means.
It may know the formula for CAGR.
It may be able to explain the difference between EBITDA and free cash flow.
None of these capabilities necessarily demonstrates that the system can perform a real financial workflow reliably.
Consider a relatively ordinary market-screening request:
Find U.S. equities with a market capitalization above $2 billion, average 20-day dollar volume above $50 million, current relative volume above 1.5, and a close above the previous 30-day high. Then rank the qualifying companies by relative volume and return the top 20.
There is nothing conceptually exotic about this task.
But completing it correctly requires a model or Agent to perform several different operations.
It needs to identify the relevant universe, obtain the appropriate data, understand how each metric is defined, calculate derived values, apply multiple filters, identify a technical event, sort the resulting set and return the correct ranking.
A model can fail at any stage.
It can retrieve stale data.
It can calculate the denominator incorrectly.
It can include the current session in a historical lookback that should exclude it.
It can identify the correct securities but rank them incorrectly.
It can omit qualifying results.
It can return securities that never satisfied the original criteria.
The final answer may look entirely plausible while still being materially wrong.
The same problem becomes more pronounced in crypto and onchain markets.
A user might ask for tokens satisfying a combination of liquidity, market capitalization, netflow, active-wallet growth and transaction-volume conditions over a particular time period.
Or an investor may ask an Agent to identify wallets with a defined profitability profile, specific transaction behavior and exposure to a particular category of assets.
These tasks are not primarily tests of financial vocabulary.
They test whether a system can convert a financial objective into a sequence of correct operations over real data.
This is why the distinction between financial knowledge and financial work matters.
The former asks whether a model understands finance.
The latter asks whether it can produce a correct, verifiable financial result.
Financial Benchmarks Have Already Begun to Move in This Direction
The benchmark landscape is not static.
FinanceBench and earlier financial QA datasets established an important foundation by testing models against financial documents, numerical reasoning and fixed reference answers.
More recent benchmarks have expanded the scope considerably.
Vals’ Finance Agent benchmark evaluates systems that can use web search, SEC EDGAR, document parsing, stored information and historical market prices to answer complex financial questions. The model is no longer expected to rely entirely on information contained in the prompt. It must decide how to use tools to retrieve and analyze the required information.
Rogo’s Big Finance Benchmark goes further into professional workflow evaluation. Its 928 questions cover areas including capital structure, valuation, private capital, earnings analysis and forecasting. Rather than grading only the final answer, Rogo breaks complex work into detailed criteria covering retrieval, definitions and calculations, making it possible to identify where a model’s process failed.
This is an important evolution.
A model that retrieved the right filing but made a calculation error should not necessarily be treated as equivalent to a model that failed to identify the correct source entirely.
Likewise, two models with similar aggregate scores can have very different capability profiles across financial workflows. Rogo’s results illustrate this clearly: different frontier models lead in different classes of financial work rather than one model dominating every category.
These benchmarks increasingly measure how AI performs work rather than simply how much it knows.
But the next transition in financial AI introduces another problem.
The market itself is not static.
Financial Agents Operate in a Changing World
Most benchmark design benefits from fixed questions and fixed answers.
That is relatively easy when evaluating document analysis.
A company’s reported revenue for a completed fiscal year does not change every hour.
A historical filing can be frozen.
A calculation performed using defined inputs can be reproduced.
Markets are different.
Prices change.
Volumes change.
Liquidity changes.
Wallet balances change.
Positions change.
Signals appear and disappear.
Rankings can change within minutes.
This becomes a fundamental problem once financial AI is expected to perform market screening, signal detection, cross-asset analysis or portfolio-aware workflows.
Suppose two models receive the same request:
Find the 20 U.S. equities with the strongest relative volume among stocks currently breaking above their previous 30-day high.
Model A executes the task at 10:00 a.m.
Model B executes it at 11:30 a.m.
Even if both systems perform perfectly, their answers may differ because the underlying market has changed.
A benchmark cannot meaningfully compare those outputs unless both models are evaluated against the same financial reality.
This means future financial-agent benchmarks will increasingly need a concept that traditional static QA benchmarks rarely require:
a reproducible market state.
For dynamic financial tasks, the benchmark must specify not only the question but also the moment in the market to which the question refers.
That may require recording the timestamp, data cutoff, exchange session, market-data version and, for onchain tasks, the relevant block or slot.
All evaluated models should then operate against the same frozen state.
The principle is simple:
Same market. Same tools. Same tasks. Different models.
Without this control, performance differences can easily become differences in data rather than differences in intelligence.
Dynamic Markets Require Reproducible Ground Truth
Once financial benchmarks move into dynamic markets, another issue becomes more important: how the correct answer is created.
In many language-model benchmarks, another model is used as the evaluator.
This approach can be useful when the task is genuinely qualitative.
But it should not become the default source of truth for financial questions that can be verified deterministically.
If the task asks for a revenue CAGR, the benchmark should not need an LLM to decide whether the answer is correct.
The required financial figures can be obtained from an authoritative data source and the calculation performed in code.
If the task asks for securities satisfying a set of screening conditions, a machine rule can be executed against a frozen dataset to produce the correct set.
If the task asks for a ranking, the ordering can be calculated from the underlying metric.
In such cases, the benchmark can construct a reproducible Gold Answer independently of the model being tested.
A robust architecture might look like:
Source Data → Standardized Metrics → Frozen Snapshot → Code → Gold Answer
This matters for more than technical elegance.
Financial evaluation needs provenance.
If a benchmark says that a model is wrong, it should be possible to answer several questions:
Where did the underlying data come from?
What definition of the metric was used?
What market state applied?
What calculation produced the reference answer?
Can the same answer be reproduced later?
Which part of the model’s output was incorrect?
For financial applications, these questions are essential.
Accuracy without traceability is difficult to trust.
Not Every Financial Task Has a Deterministic Answer
There is, however, an important limit to this approach.
Some forms of financial judgment cannot be reduced to a formula.
Chart interpretation is one example.
Experienced discretionary investors often recognize patterns that are difficult to describe through one technical indicator. They may evaluate price structure, volume, consolidation, breakout quality, prior volatility and broader market context simultaneously.
Two charts can satisfy the same simple numerical conditions while only one represents the pattern an experienced trader would actually consider actionable.
The same issue exists in other forms of expert financial analysis.
A model may retrieve all the correct information but still fail to interpret its significance in the way an experienced investor would.
This is where expert labels remain important.
A future financial benchmark will likely require several different forms of ground truth rather than one universal grading mechanism.
For deterministic financial calculations, code should provide the answer.
For screening and ranking tasks, machine-executed rules should provide the answer.
For tasks involving expert visual or investment judgment, carefully constructed human labels may remain the appropriate reference.
The general principle should be:
Use the most objective source of truth available for each task.
AI can assist with task generation, rewriting, quality assurance and error analysis.
It should not automatically become the ultimate authority when a stronger source of truth exists.
From Financial Analyst Benchmarks to Financial Agent Benchmarks
As financial AI evolves, the capability map being tested will need to broaden with it.
The relevant question will no longer be whether a model can answer finance questions in the abstract.
A useful benchmark will increasingly need to evaluate whether an Agent can reliably perform several different classes of work.
Retrieval remains fundamental. A system must locate the correct filing, market datapoint or transaction record rather than inventing one.
Calculation tests whether the model can transform raw facts into derived financial metrics using the correct definitions and periods.
Screening measures whether an Agent can search a large universe and return the complete set of objects satisfying several simultaneous constraints.
Ranking adds another layer. Finding the correct candidates is insufficient if the system cannot order them according to the requested financial criterion.
Detection evaluates whether an Agent can identify market events or changes such as breakouts, unusual volume, capital flows or portfolio-risk changes.
Interpretation tests whether the system can explain the financial meaning of information rather than merely extracting it.
Multimodal understanding becomes increasingly important as Agents encounter charts, tables, filings, dashboards and other visual information.
Tool use tests whether the Agent knows when it should retrieve data, search a filing, query a market database or perform a calculation instead of relying on its internal model knowledge.
State awareness matters because financial work happens over time. A useful Agent should understand that its answer depends on a specific market state and update when that state changes.
Eventually, another capability will become increasingly important:
action.
As AI systems gain access to portfolios and financial accounts, the boundary between analysis and execution is becoming thinner.
Interactive Brokers’ current AI integrations are deliberately designed to keep execution under user control. Connected AI systems can inspect authorized portfolio information and generate instructions, but the investor reviews and submits each order.
That human-in-the-loop model is likely to remain important, particularly as financial agents mature.
But even before autonomous execution becomes a benchmark category, the industry needs to understand whether the analysis that precedes an action is reliable.
An Agent should not be considered capable merely because it can call an execution tool.
The more important question is whether the financial judgment leading to the action can withstand verification.
The Environment Matters as Much as the Model
There is another consequence of this shift.
Model evaluation increasingly cannot be separated from the environment in which the model operates.
A frontier model without access to professional market data may perform worse than a smaller model equipped with well-designed financial tools.
A model given poor tool descriptions may fail even if its underlying reasoning capability is strong.
One Agent may have access to real-time pricing while another relies on public web search.
Comparing them directly would measure the environment as much as the model.
This is why the concept of a common financial Agent harness becomes increasingly important.
When the objective is to compare base models, the surrounding conditions should be controlled.
Each model should receive the same available data, the same tools, the same financial Skills, the same task, the same market snapshot and comparable runtime constraints.
Only then can the resulting difference reasonably be attributed to how effectively the model operates within that financial environment.
This is also why leaderboard scores should be treated as contextual measurements rather than permanent properties of a model.
Models change.
Providers update them.
Data environments change.
Tools improve.
A score should therefore mean something closer to:
Model X achieved this result in Benchmark Batch Y, using Environment Z, against Market Snapshot T.
That description may be less elegant than a single permanent leaderboard number.
It is also substantially more accurate.
Financial AI Is Becoming Portfolio-Aware
The recent expansion of financial AI makes this problem increasingly urgent.
Kimi’s new financial stack reflects the movement from general-purpose intelligence toward domain-specific workflow systems. The model is paired with authoritative data sources, Skills, financial modeling capabilities and institutional deliverables.
Interactive Brokers represents a different part of the same transition. ChatGPT, Claude, Grok and other MCP-compatible systems can now operate with authorized context from a real brokerage account, including positions, cash balances, historical transactions and portfolio risks.
The convergence is clear.
Financial AI is moving through several stages:
Model → Tools → Professional Data → Portfolio Context → Financial Agent
The benchmark must follow the same trajectory.
If evaluation remains centered on static document questions while financial products are moving toward continuous market analysis and portfolio-aware workflows, benchmark performance will become increasingly disconnected from actual user experience.
The question is no longer merely whether a model performs well on financial text.
It is whether it performs reliably in a financial environment.
A Benchmark Is Also a Definition of the Product
There is a broader reason this matters.
Benchmarks do more than measure systems after they are built.
They influence what systems are optimized to become.
If a benchmark primarily rewards financial-document extraction, model developers will optimize for financial-document extraction.
If it evaluates multi-step professional workflows, models will improve at those workflows.
If it eventually includes live-market screening, onchain analysis, chart interpretation, risk reasoning and portfolio-aware tool use, the resulting systems will evolve in a different direction.
In that sense, the benchmark becomes a specification for what the industry means by a financial Agent.
This is one of the questions we have been exploring at Questflow.
As we think about financial Agents that can eventually help investors research markets, interpret signals, understand expert frameworks and operate across multiple financial environments, we also need a more rigorous way to understand what these systems can actually do.
We are therefore increasingly interested in benchmark architectures centered on real financial tasks, reproducible market states, machine-verifiable outcomes and expert judgment where deterministic answers are not available.
The objective is not simply to create another collection of financial questions.
It is to understand whether an AI system can reliably transform a financial request into a correct and verifiable piece of work.
That distinction becomes increasingly important as the consequences of AI outputs move closer to real capital.
Measuring Financial Intelligence in the Agentic Era
Financial AI is entering a different phase.
The first generation demonstrated that language models could understand financial documents.
The second showed that Agents could use tools to perform increasingly realistic analyst workflows.
The emerging generation is beginning to operate with professional data, continuously changing markets, real portfolio context and increasingly direct access to financial infrastructure.
That progression requires a corresponding change in evaluation.
Static questions will remain useful.
Financial documents will remain important.
Traditional analyst work will remain a core capability.
But they will no longer be sufficient on their own.
The next generation of financial benchmarks will need to measure how models operate in a changing environment.
They will need to evaluate complete sets rather than convenient examples.
They will need to test ranking as well as retrieval.
They will need to distinguish between data errors, calculation errors, interpretation errors and tool-use failures.
They will need market snapshots for dynamic tasks.
They will need reproducible Gold Answers wherever possible.
They will need expert labels where judgment cannot be reduced to code.
And they will need to make performance sufficiently transparent that a user can understand not only whether an Agent failed, but how it failed.
This is particularly important in finance.
A fluent answer can be wrong.
A convincing investment narrative can be constructed from incorrect inputs.
A model can identify nineteen correct securities and quietly omit the twentieth.
A portfolio recommendation can appear reasonable while relying on stale holdings.
As financial AI gets closer to decision-making and execution, surface-level plausibility matters less.
Verifiability matters more.
The most important financial benchmark of the next generation may therefore not ask:
How well does this model know finance?
It may ask something more demanding:
Given the same financial world, the same tools and the same task, can this Agent consistently produce the correct work?
That is a much higher standard.
It is also increasingly the standard financial AI will need to meet.