Securing AI Agents: From Risk Assessment and Runtime Guardrails to Self-Improvement and Certification

TL;DR

• A three-bucket framework: (1) risk assessment / red teaming — the pentest of AI, run first to surface weaknesses; (2) fundamental protection — train safety principles into the model and the agentic workflow rather than pasting a policy into a system prompt and hoping; (3) guardrails — a plug-and-play "firewall for AI systems" spanning the prompt level, the MCP level, tool calls, and tool trajectories.
• DTap (DecodingTrust-Agent Platform): over 50 hand-built simulation environments across 14 domains, plus a red-teaming agent that autonomously discovers attack strategies. Everything is open-sourced except that red-teaming agent — withheld so target agents don't overfit to its strategies. Outcomes are scored by a verifiable judge that inspects state (did the transaction actually happen? was the file actually deleted?) rather than an uncertain LLM judge.
• Two principles for organizing risk: regulatory compliance (OWASP Top 10, NIST, MITRE, FINRA, EU AI Act) and use-case-driven categories (hallucination, data exfiltration, misuse, …), plus a split by agent type (computer use, tool use) — because a universal secure and compliant agentic system is essentially unattainable, so depth per sector is the realistic path.

Key Points

Framing: the last mile, and what it costs in the real world (~02:47–02:50)

Bo Li introduced himself as a UIUC professor working on AI safety and security research and as co-founder and CEO of Virtue AI, which provides a unified AI security and compliance platform for enterprises in finance, healthcare, retail, and elsewhere. He also noted the news of the moment: Virtue AI had just joined Meta Superintelligence Labs, continuing the mission of providing a trust layer, governance, and observability for large-scale personalized agents.

His central position: AI safety and security is the critical last mile for all AI applications, especially agents and multi-agent systems. Without confidence that these problems are solved to some meaningful extent, deploying agentic systems into the real world stays hard to justify.

The high-level attack surface he listed: prompt injection, jailbreaks, data leakage, hallucination, and — a particular enterprise concern — brand risk. On the real-world side:

• Browser agents such as OpenAI Atlas can be attacked and misled into purchasing items they were never asked to buy.
• A customer-service-style agent a couple of months earlier could be manipulated into leaking an entire customer database — fortunately found by a white hat — with potential losses in the hundreds of billions of dollars.
• Trading markets and finance supply plenty of further examples.

The encouraging part is the convergence of regulation, policy, academia, industry, and the compliance community. His caveat: there is still a very long way to go.

Three buckets: red teaming, fundamental protection, guardrails (~02:50–02:54)

1. Risk assessment (red teaming). His framing was direct: red teaming is to AI what pentesting is to software. Whatever model or agent powers an application, and whichever domain it lands in — finance, healthcare, autonomous systems — the first step in understanding the system is to stress-test it: build the skill sets, autonomous environments, red-teaming strategies, and algorithms that expose its vulnerabilities and weaknesses, so that any subsequent solution or governance has something to work from.

2. Fundamental protection. Assume red teaming has produced a comprehensive list of weaknesses; how do you protect the system? The first track is fixing the model itself: making it not merely the probabilistic, data-driven model we use today, but one with true reasoning and a fundamental component that injects and ensures AI safety and security principles. That is what would let policies like the EU AI Act or FINRA rules actually be ensured rather than handed over as a system prompt — where, in his words, we don't know when or whether the agent will follow it, at quite high uncertainty. He acknowledged this is very challenging, from fixing the model to fixing the agentic workflow, with a great deal of active research already underway.

3. Guardrails. If you want to use and deploy something now, can you have a quick plug-and-play firewall — the traditional cybersecurity concept, applied to AI systems? This layer needs protection from several angles: prompt level → MCP level (inspecting MCP code for vulnerabilities) → tool calls and tool trajectories, so you can understand intent, understand the vulnerabilities, and close the loop on the output end to end.

Together, these three form what he considers the overall architecture for protecting, governing, and securing current agentic systems.

The two hard questions of red teaming, and DTap (~02:54–02:59)

He spent most of the talk on the first bucket, opening with two questions: what are the rules and perspectives for risk assessment, given how large that space is? And how do you provide a comprehensive red-teaming platform and set of strategies for agentic systems?

Start with the attack surface. An agentic system is a core LLM plus tools plus an environment — and the environment is the highly vulnerable part:

• When the agent interacts with a website, that site can carry injections both visible (in review text) and invisible (in hidden forms); his group has many papers showing that simply interacting with an environment is quite dangerous.
• Swap in a database and you get SQL injections; every traditional vulnerability class recurs against frontier agentic models.
• And that leaves aside the agent's supply chain and vulnerabilities in the core model itself.

Two principles for categorizing risk perspectives:

1. Regulatory compliance — frameworks including OWASP Top 10, NIST, MITRE, FINRA (for finance), and the EU AI Act, followed to produce a comprehensive assessment.
2. Use-case-driven — his group has a NeurIPS best paper working from the use-case perspective (hallucination, data exfiltration, misuse, and so on) to map the landscape of the vulnerability space.
3. Plus a split by agent type — since a universal secure and compliant agentic system is extremely hard to build, look at sectors such as computer use and tool use and get more comprehensive, in-depth analysis and protection within each.

That leads to DecodingTrust-Agent Platform (DTap). He stressed that everything is open-sourced except the red-teaming agent, with the paper online; the attack agent is withheld specifically so target agents don't overfit to its attack strategies. Three things make it distinctive:

1. The environment. Red teaming an agent needs a sandbox, and this sandbox differs from the ones agents currently ship with — those typically contain only a file system, whereas a real agent connects to external MCP tools, and the moment you connect to external MCP tools all the vulnerabilities become live. DTap contains over 50 environments, each built one-to-one, so tool injection, environment injection, and similar tests run under control.
2. The red-teaming agent. It provides comprehensive strategies — tool injection, skill injection, prompt injection, and their combinations — paired with a verifiable judge: you can go to the state and confirm whether the agent actually made the transaction or actually deleted the files, so you don't have to rely on an uncertain LLM judge.
3. 14 domains. His challenge to the field: on what basis would we claim 50 environments is enough, or 100, or 5,000? So go by domain instead — 14 of them, each with that domain's own policies (finance, healthcare, and so on) driving how the risk assessment is designed.

With this framework you can evaluate the helpfulness and safety of different agentic frameworks crossed with different models. He emphasized that these are two separate components, and the safety landscape can differ substantially per combination — which is exactly what the platform's ablation studies are for.

Protection and certification: ShieldAgent (~02:59–03:00)

Nearly out of time, he moved quickly through the protection side and ShieldAgent. The process extracts rules from regulatory principles and policies and turns them into an action graph. The payoff is not just a promising model but a component that can provide certification — meaning that at runtime you get guardrail protection carrying a certain level of guarantee, backed by action-graph analysis and certification. The numbers, he noted, are also clearly better.

He closed by widening the frame: in the long term, looking at agentic systems from a population perspective, he hopes to see a trajectory like the internet's — if the community solves this last step together, we should see wide adoption of trustworthy agentic systems.

Note: the self-improvement portion of the announced title was not reached; the delivered talk covered risk assessment, guardrails, and certification.

Quotes

"Red teaming is a pentest for AI models, for agents, for multi-agent." (~02:51)

One line locating red teaming in the security process.

"…rather than just say give it as a system prompt — and we don't know when and whether the agent will follow that, which is quite of high uncertainty." (~02:52)

Why writing a policy into a prompt is not a compliance mechanism.

"The moment you connect to the external MCP tools, all the vulnerabilities could happen." (~02:57)

Why a file-system-only sandbox is inadequate as a red-teaming environment.