As cancer genomics datasets keep growing in size, querying, and visualizing these data is increasingly challenging. cBioPortal, a visualization platform for cancer genomics data, addresses this need by allowing integration, visualization, and analysis of cancer genomics data in an intuitive environment. To extend the functionality of cBioPortal, Gene Queries can be formulated in a specific query language called Onco Query Language, or OQL for short. This allows for advanced data selections to be performed, and more detailed analysis to be carried out.
An earlier article, introduced AI OQL-Helper, which we now call the OQL Gene Assistant. This helper tool translates natural-language requests into OQL, and is integrated into cBioPortal. The Gene Assistant can make advanced genomic queries more approachable for new users and speed up repetitive query formulation for experienced users.
While one-off example queries demonstrated the potential of this approach, we needed to assess how reliable the Gene Assistant was in providing correct answers. Therefore, we set out to test it systematically: how reliably can different language models translate a broad range of cancer-genomics requests into OQL? Where do errors occur, and how should queries be evaluated before they define patient cohorts?
This article looks back at the completed benchmark project, developed as part of a master's research project at The Hyve. It outlines what was built, the scope of the evaluation, the main baseline findings and the practical lessons for the OQL Gene Assistant. A separate follow-up article will examine how prompt design and temperature settings affected performance.
When a small query difference changes the cohort
When doing the benchmark, we noted several shortcomings in the answers that we got. One in particular attracted our attention, because it is a difficult error to catch in evaluations. For instance, one common query to run is this:
Select all samples with a BRAF mutation from V to E at position 600
BRAF: MUT=V600E
BRAF: MUT
Similarly, for another gene and mutation, we observed similar responses:
Select samples with a KRAS G to C mutation at position 12
KRAS: MUT=G12C
KRAS: MUT
In both cases, the generated OQL query is parsable, and it provides a selection of samples. However, the results are not the same. The queries answer a valid but different biological question. The first selects the specific BRAF V600E alteration; the second selects all BRAF mutations; and similarly for KRAS. The difference is visually small, yet it can substantially change the returned patient group.
This is one of the most common mistakes that LLMs make when translating natural language to OQL: the models tend to broaden the query. We observed more extreme examples of this. For instance, one model broadened the requested KRAS single-variant cohort into a multi-gene pathway query in four of five runs, increasing the cohort four- to five-fold. In the BRAF example, another model repeatedly produced a non-existent alteration token, and returned no patients. An empty or plausible-looking result could be misread as a biological finding when it is actually a translation error. Across the two case studies, most observed failures broadened rather than narrowed the intended cohort.
Building a systematic benchmark
A few successful (or unsuccessful) demonstrations cannot show whether non-deterministic tools, such as LLMs, are reliable across the full range of requests users may make. The project therefore created a benchmark for natural-language-to-OQL generation rather than testing only a small collection of showcase examples. The benchmark contains 94 natural-language requests paired with reference OQL queries. The set combines researcher-style questions, examples derived from the OQL documentation, and additions from domain experts. It covers the main query types that OQL enables, including mutations, copy-number alterations, expression, protein abundance and fusions, as well as more detailed constructions involving labels, exclusions, ranges, and gene-specific constraints.
In this benchmark, 25 language models from seven providers were evaluated using the same baseline prompt. Each main configuration was run five times, allowing the project to measure mean performance and variation between repeated outputs. Latency and cost were recorded alongside quality, although this article focuses on reliability.
Evaluating more than whether a query runs
Every generated query passed through a layered evaluation pipeline. First, the cBioPortal OQL parser checked whether the output was syntactically valid. This answers an important but limited question: is the query well formed and executable?
Next, a semantic evaluator compared the biological constraints in the generated query with those in the reference query. Equivalent formatting differences were normalized, while omitted, added or reassigned constraints remained errors. This made it possible to distinguish harmless surface variation from a change that could alter the biological question or selected cohort - and will catch the broadening of queries that we observed.
Figure 1 shows how these evaluation steps fit together, including the cohort-level validation performed for selected case studies.
What the baseline comparison showed
The 25 models that were benchmarked showed a wide range of reliability profiles. Under the shared baseline prompt, mean syntactic validity ranged from 81.1% to 98.7%. Semantic exact match - requiring all normalized biological constraints to be preserved - ranged from 61.1% to 82.6%. No model was perfect, and the leading semantic models formed a close cluster with overlapping confidence intervals, so small differences at the top should not be interpreted as a definitive ranking.
The more informative result was how different syntactic and semantic accuracy could be. DeepSeek V3.1 and Ministral 3B produced valid OQL at nearly the same rate: 94.5% and 93.4%, respectively. However, their semantic exact-match scores were much further apart at 79.4% and 61.3%. A syntax-only evaluation would have hidden an 18.1-percentage-point difference in how often the models preserved the requested constraints. Figure 2 shows the complete baseline comparison across all 25 models, including the variation observed across the five replicate runs.
What this benchmark delivers
The project delivered a curated OQL benchmark, a repeatable evaluation pipeline and a way to inspect errors at syntax, meaning, and cohort level. Together, these provide a foundation for comparing future models and configurations using the same criteria.
The results also clarify what reliable assistance requires. A parser remains valuable because malformed output should be rejected immediately. During development, semantic checks and representative cohort tests are needed to detect valid-looking errors. When a request is underspecified, the safer response may be to ask a clarifying question rather than guess. In the user interface, generated OQL should remain visible and editable so that the researcher can review it before execution; that is also how the Gene Assistant is currently implemented.
From promising output to reviewed assistance
The benchmark confirms that large language models can generate useful OQL across a broad and technically varied set of requests. It also provides a clearer picture of the remaining reliability challenge: outputs that fail syntax are easy to catch, while outputs that preserve the grammar but change the meaning can be harder to recognize.
The practical question is therefore not only, 'Does this query run?' It is also, 'Does this query still mean what I asked?' Measuring both gives the OQL Gene Assistant a stronger foundation for continued development and responsible use in cBioPortal.
Try the OQL Gene Assistant
Explore the OQL Gene Assistant on The Hyve's cBioPortal demo. Treat each generated query as a proposal: review the OQL and confirm that it preserves your intended cohort before running it. Interested in reliable AI-assisted querying or other cBioPortal extensions? Contact The Hyve to discuss your use case.