Introducing the Multimodal Knowledge Agent benchmark

In partnership with Artificial Analysis, we're launching MKA (Multimodal Knowledge Agent), a new benchmark built to evaluate how agents reason across visually complex files to solve real-world knowledge work tasks.

Introducing MKA

Most knowledge work benchmarks can be solved with text alone, without ever needing to look at an image or page layout. Yet, everyday professional work looks very different. Across many industries, day-to-day work is deeply visual — from cross-referencing architectural blueprints and tracing engineering schematics to synthesizing findings into polished decks, reports, and spreadsheets. We built MKA to measure how well agents can handle this visually complex work.

To ground the benchmark in real-world workflows, we partnered with domain experts across 10 verticals and 83 occupations where day-to-day work is inherently visual. Together, these specialists — from architects and biologists to underwriters and data scientists — created MKA's 200-task evaluation set to assist them in their jobs.

Each task reflects a multi-step project that takes a seasoned professional three to eight hours to complete, pairing a realistic prompt and a bundle of multimodal input files with an expert-written rubric of roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design.

Starting today, you can explore the benchmark and evaluate your own models on Artificial Analysis. Every model runs in the same open-source agent harness, Stirrup, with the same tools, sandbox, and turn budget.

An agentic autorater built on a non-Google model grades each deliverable. It opens every file, renders pages, inspects spreadsheet formulas, and checks the work against the expert rubric. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time.

To keep results meaningful, Artificial Analysis runs the 200-task evaluation set and keeps it private so the tasks can't leak into training data or be found by the model via search. We've also released a separate set of 10 public tasks on Hugging Face, so anyone can inspect the format and test their own agents.

Our technical report covers the methodology and results in detail. It shows that frontier models work in very different ways. On the same tasks, the models we tested have extreme differences in efficiency, using between 30 and 130 turns on average to complete each task. We show how strong perception abilities are necessary to solve each task, and we also show how models try and fail to compensate when vision is removed.

Expanding visual frontiers with Gemini 4 Carbon

Gemini 4 Carbon achieves #2 in quality among the models we tested, and completes tasks at 6x lower cost on average compared to the top scoring model.

Getting started

Gemini 4 Carbon is available today via the Gemini API in Google AI Studio. Explore our updated Gemini 4 developer guide and documentation to get started.