ScarfBench: Can Agents Migrate Enterprise Java?

TL;DR

• Applications outlive frameworks. Enterprise Java apps accumulate decades of institutional knowledge — business rules, data semantics, workflows, integrations. Migration has to replace the underlying stack while preserving behavior, and the second half is the hard part.
• The difficulty is that the framework hides the behavior. Java frameworks bury runtime behavior behind their APIs using proxies, reflection, and interceptors, and the code itself doesn't capture any of it. So there is no one-to-one equivalence between Spring and Jakarta; a migrated app can build and run fine while hiding silent failures that only surface at deployment.
• ScarfBench = Self-Contained Application Refactoring Benchmark, built by having IBM subject-matter experts manually port every application across Spring, Quarkus, and Jakarta, with hand-written behavioral tests for each — from curl smoke tests to Playwright browser-interaction checks.
• The headline result: agents are strong at producing migrations that compile and deploy, but only 2–14% of migrations preserved behavior exactly. Compilation is a weak signal that creates a false impression of competence.
• Migration direction is asymmetric: Spring→Quarkus is a very different problem from Jakarta→Spring, and the data strongly suggests some frameworks are hard to migrate to and others hard to migrate from.

Key Points

Why enterprise Java migration is worth benchmarking (~02:36–02:37)

Krishna is a senior research scientist at IBM Research. (The talk opened with a slide failure; he switched to his own laptop mid-talk, leaving a gap around 02:35–02:36.)

Enterprise applications written in Java remain critical. These legacy applications were written against some framework, and the applications themselves often outlive the frameworks. Modernization matters because newer frameworks bring supported runtimes, better security, and better deployment modalities, among other things.

But the real insight is what's sitting inside a deployed application: institutional knowledge embedded over several decades — business rules, data semantics, workflows, integrations. So migration must replace the underlying technology stack while preserving application behavior.

Why it's hard: the behavior isn't in the code (~02:37)

The Java frameworks these legacy applications are written against hide runtime behaviors behind their APIs — proxies, reflection, interceptors. The code doesn't capture any of this; the framework handles it.

The consequence: there is no one-to-one equivalence between an application written in Spring and one written in Jakarta. When you migrate, it may look like the app builds and runs correctly, but there may be silent failures that only show up at deployment time.

The anatomy of a typical enterprise Java application (~02:38–02:39)

Large applications are organized in tiers, each addressing a specific concern: the presentation layer (serving a browser or phone app) down through the data access layer (database reads and writes), plus a cross-cutting concern tying everything together with configuration, security, and so on.

The difficulty is that much of the behavior and business logic is hidden behind the framework and its APIs, and each framework has its own idiosyncrasies — dependency injection, annotations — that dictate how business logic gets embedded. Go hunting for it in the code and you may find none of it.

And although the layers are separate in terms of concerns, a deployed application has all of them interacting with one another, often at the same time — which is what makes the problem genuinely challenging.

The design of ScarfBench (~02:39–02:40)

ScarfBench stands for Self-Contained Application Refactoring Benchmark, with a twofold objective:

1. Measure how effective agents actually are at migrating and modernizing applications from one framework to another.
2. Along the way, by understanding agent trajectories, build a "field manual" for constructing agentic solutions for general modernization.

Two groups:

• Focused applications: self-contained apps for a single layer.
• Whole applications: those layers assembled into a cohesive business use case.

IBM's method was to have a large number of subject-matter experts manually convert each application across Spring, Quarkus, and Jakarta. Each row is an application, and a unit of migration is any app going from one framework to another.

This design also serves two purposes:

1. Every migration has its own idiosyncrasies, and they are asymmetric — Spring→Quarkus is very different from Jakarta→Spring.
2. It exposes how agents handle independent layers — an agent may be excellent at converting the presentation layer while struggling with integration or dependency injection.

The whole-application group captures each app both as a monolith and as an equivalent microservice deployment, again manually converted by SMEs across all frameworks.

Scale, as spoken on stage: roughly 38 applications individually, expanding to 114 variations, with directed transformations totaling about 228. (The published paper's figures differ slightly — see To Verify.)

The real differentiator: hand-written behavioral tests (~02:41–02:42)

The key distinction he emphasized is the handwritten test cases. As each application was migrated, developers wrote tests for the behaviors expected once it's deployed, spanning:

• HTTP checks — a curl smoke test confirming the app deployed in the target framework exposes the same HTTP and HTTPS ports; essentially a health check.
• Smoke tests also cover whether messaging works correctly and whether database updates, commits, and rollbacks behave as expected.
• JSP-page-level checks verifying browser click actions and correct data propagation in the deployed application.
• Playwright tests for apps with a browser endpoint, ensuring user actions on the migrated application look identical to the source app.

Results: it compiles, but the behavior is gone (~02:42–02:43)

They evaluated a large number of coding agents; more results, failure-mode taxonomies, and guidance on building better agents are in the paper. The stage highlight:

Agents are highly effective at migrating applications to the point where they compile and deploy — and not good at maintaining behavior in the target application.

The number: after migration, only 2–14% of migrations had exactly the same behavior as the source application.

The methodological lesson he drew: compilation itself was a weak signal, giving a false indication that agents are good at migration when behavior was not in fact preserved.

The flows also strongly suggest that some frameworks are really hard to migrate to and some are really hard to migrate from — an inherent complexity and asymmetry in how agents move applications between frameworks.

Quotes

"The applications themselves often outlive the frameworks." (~02:36)

A one-line definition of the enterprise modernization problem.

"If we go hunting for it in the code, we may not find any of these." (~02:38)

Business logic lives in the framework's annotations and dependency injection — exactly where a source-reading agent will miss it.

"Compilation itself was a weak signal … that gave us a false indication that the agents are really good at migration, but the behavior was not preserved." (~02:42)

The most important methodological warning in the talk.