Microsoft Launches ASSERT to Help Developers Test AI Behavior With Plain Text Descriptions

The open-source framework turns product rules and safety policies into scored AI behavior tests

Microsoft has released a new open-source tool that lets developers create AI behavior tests by describing expected behavior in plain language. The tool, called ASSERT, is designed to help companies check whether their AI agents and applications behave the way they are supposed to inside specific products, workflows, and policy environments.

ASSERT stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. Microsoft introduced the framework on Tuesday as part of a broader push to help developers build AI systems that can be tested, monitored, and trusted beyond the demo stage. The company says ASSERT can turn high-level descriptions of goals, rules, policies, or intended behaviors into structured evaluations that score how well an AI system follows those expectations.

The launch comes as more companies move from experimenting with AI models to deploying AI agents in real products. That shift creates a new problem. General model benchmarks can test broad capabilities, but they do not always tell a company whether its own AI tool is following its own rules.

Why AI Behavior Testing Is Becoming More Important

AI testing has become one of the most important parts of the generative AI market. Large labs already evaluate models for safety, compliance, hallucinations, sycophancy, bias, and alignment. But those tests are often broad. They may show whether a model performs well in general, but they do not always reflect the specific behavior a company needs in a real application.

That gap becomes serious when AI agents are connected to tools, documents, customer data, email, codebases, calendars, or internal systems. A general-purpose model might be capable, but the application around it still needs rules. Developers need to know whether the system respects permissions, avoids unsafe actions, stays within scope, and produces the right kind of output for the product.

ASSERT is Microsoft’s answer to that application-specific problem. Instead of asking teams to manually write every possible test case, the framework starts with plain-language descriptions. Developers can describe what an AI system should or should not do, then use ASSERT to generate scenarios and tests around those rules.

For example, a company could say that a document research agent should not email people outside the organization, should restrict confidential information to senior executives, and should provide concise summaries based on prior context. ASSERT can use those instructions to generate test cases that check whether the agent follows those limits over time.

How ASSERT Works

ASSERT takes natural-language descriptions of expected behavior and converts them into a structured set of acceptable and unacceptable behaviors. It then generates problem scenarios and test cases, runs them against the target AI system, and scores the results.

The framework can also record the path an AI system takes during a test. That includes intermediate actions, tool calls, and decision points. This matters because AI failures are not always visible in the final answer. An agent may produce a reasonable response while taking an unsafe route in the background, using the wrong tool, exposing sensitive context, or attempting an action it should not have taken.

Developers can also provide system context, tools, and constraints to make the tests more relevant. That makes ASSERT useful for application teams that need to test real product behavior rather than abstract model quality.

The goal is not only to find obvious failures. It is to make AI testing repeatable. Once an AI application is updated, developers can rerun evaluations to see whether the system still behaves correctly. That gives teams a way to catch regressions, where a new model, prompt, tool integration, or policy update accidentally breaks behavior that previously worked.

From Generic Benchmarks to Product-Specific Rules

The AI industry has spent years building benchmarks to compare model performance. These benchmarks are useful, but they are not enough for production AI. A model may perform well on a public benchmark and still fail inside a company’s actual workflow.

A customer support agent, for example, does not only need to answer accurately. It may need to avoid making refund promises, follow escalation rules, protect account data, and stay within a company’s policy language. A coding assistant may need to avoid modifying certain files, using deprecated libraries, or exposing private repository content. A healthcare-facing assistant may need to provide safe general information without crossing into diagnosis or treatment instructions.

These are not universal benchmarks. They are business-specific requirements. ASSERT is designed for that layer of testing, where the question is not simply “Is the model smart?” but “Does this AI system behave correctly in our product?”

That distinction is becoming more important as AI agents gain more autonomy. When an AI system can call tools, retrieve files, trigger workflows, or act across applications, developers need more than a prompt and a hope that the model follows instructions. They need tests that show where the system breaks.

Microsoft’s Broader Trust Stack

ASSERT is part of Microsoft’s wider effort to build a trust layer around AI agents. In a related Microsoft Foundry announcement, the company also introduced the Agent Control Specification, or ACS, a portable runtime control standard meant to place safety and security controls at different checkpoints in an agentic workflow.

Microsoft frames ASSERT and ACS as complementary tools. ASSERT helps teams identify where an AI agent is failing against policies or expected behavior. ACS is meant to help developers place controls at the points where those failures can happen, including input, model behavior, state, tool execution, and output.

That pairing shows where enterprise AI development is heading. Companies do not only want to build AI agents faster. They want to understand, govern, and monitor them after launch. As agents move from internal experiments into customer-facing products, trust becomes a development problem, not just a compliance slogan.

The company also says ASSERT is open source and can work across different AI frameworks, not only Microsoft’s own stack. That matters because many companies are already building with LangChain, CrewAI, LiteLLM, OpenAI tools, Anthropic tools, custom systems, and other agent frameworks. A testing tool that works only inside one vendor’s ecosystem would have limited reach.

Why Developers Need This Now

Developers are under pressure to ship AI features quickly, but the testing culture around AI is still catching up. Traditional software testing works well when code follows predictable rules. AI systems are different because the same prompt can produce different outputs, and agentic systems can choose different paths depending on context.

That makes behavior testing harder. Developers must test not only whether an output is correct, but whether the system followed the right process, respected boundaries, and avoided unsafe intermediate steps. This is especially important when the AI has access to tools or data.

ASSERT gives developers a way to translate product rules into a repeatable evaluation process. That could make AI testing more accessible to teams that do not have dedicated AI safety researchers. A product manager, policy lead, or developer can describe the desired behavior, and the framework can help turn that into testable cases.

Still, ASSERT will not remove the need for human judgment. Generated evaluations are only as useful as the policies and descriptions behind them. Companies will still need to define what acceptable behavior means, review failures, update rules, and decide which risks are acceptable in production.

The Limits of Automated Evaluation

Automated AI behavior testing is useful, but it is not a complete safety solution. AI systems can fail in unexpected ways, and no test suite can cover every possible user input, edge case, or adversarial attempt.

There is also a risk that teams overtrust test scores. A high score may show that an AI system performed well against known scenarios, but it does not prove the system is safe in every situation. For production use, companies still need monitoring, human review, access controls, incident response, and regular updates to test coverage.

ASSERT’s value is that it makes the invisible parts of AI behavior easier to inspect. By recording tool calls, intermediate actions, and reasoning paths through the system, it gives developers more evidence about where failures happen. That can help teams fix problems earlier, before users encounter them in real-world settings.

The framework also reflects a larger shift in AI development. The question is no longer only how to make AI systems more capable. It is how to make them predictable enough for organizations to trust.

A New Layer of Quality Control for AI Agents

Microsoft’s ASSERT launch shows that AI development is moving into a more disciplined phase. The early rush to add chatbots and agents to products is now giving way to the harder work of testing behavior, enforcing policies, and monitoring systems after deployment.

That shift is necessary. As AI agents become more deeply connected to enterprise tools and user data, companies cannot rely only on broad benchmarks or manual spot checks. They need application-specific tests that reflect real policies, real workflows, and real failure modes.

ASSERT gives developers a more practical way to start that process. By turning plain-language rules into scored tests, Microsoft is trying to make AI evaluation part of the normal development cycle rather than a separate expert-only process.

The bigger message is clear: production AI will not be judged only by how impressive it looks in a demo. It will be judged by whether it behaves correctly, repeatedly, and safely when connected to real users, real tools, and real company rules.