New in LangSmith Engine: red teaming and automated testing

We built LangSmith Engine to make fixing agents faster, so developers can spend more time building new capabilities and less time fighting bugs.

Engine is an in-platform agent that automates work at each step of the agent development lifecycle. Agents are well-suited for this work. They excel at analyzing large datasets and writing code, making them great at spotting subtle issues, writing fixes, and monitoring for regressions.

Since launching in May, Engine has analyzed more than 70M traces and diagnosed tens of thousands of issues.

With today’s v2 launch, Engine now takes on more work, helping users ship fixes faster and detect issues before they appear in production. Engine v2: 

• Red-teams your agent to detect bugs and prompt issues proactively
• Detects harder-to-spot issues like inefficiencies and trends in error rate, latency, and cost
• Automatically validates its proposed fixes prior to human review and deployment

What LangSmith Engine does

Engine scans your production traces for areas of improvement, from errors to unmet user requests. Once it has identified an issue, Engine classifies it and groups related traces into a single issue. Each record comes with all the information a user needs to act on it: a root cause, a proposed fix, ground truth examples for your evaluation datasets, and ongoing monitoring.

Engine takes the data that LangSmith collects and puts it into action, helping you move through the agent development lifecycle (ADLC) faster to improve your agent.

With Engine v2, we focused on driving improvements in two of these core tasks: identifying issues and proposing fixes.

Proactively detect agent issues

With Engine v2, we’re introducing Red Teaming, our proactive troubleshooting tool to catch issues like hallucinations and violations of system prompts before they affect users in production. Red Teaming analyzes your agent’s production traces and repos to understand its purpose and behavior, then uses that context to test for weaknesses and identify issues that haven’t yet surfaced in production. You get a list of relevant, verified issues to resolve that anticipate future failures, so you can head them off.

Red Teaming is available today in Private Beta to existing LangSmith Deployment users. You can apply to join the Beta here.

Detect harder-to-spot issues

Engine v2 can now find harder-to-detect issues in production traces that often escape human review until they impact customers and LLM bills. Even when your agent appears to function as expected, subtle degradations in latency can frustrate users, and inefficient agent trajectories waste tokens.

Engine v2 flags these issues so you can resolve them before they impact your business.

• Inefficient work: Engine analyzes agent trajectories to determine whether your agent is taking an inefficient path to accomplish a given task, like making an incorrect or repetitive tool call, and proposes a more efficient alternative.
• Performance trends: Engine tracks trends in key metrics like error rate, latency, and cost.

Every issue is tracked in the same queue, so you have one place to track and resolve all your agent issues.

Ship pre-validated fixes

Engine already provided proposed fixes for the issues it detects. But previously, users were presented with a choice to deploy the fix into production and monitor performance on live traffic, or perform a manual review, test offline, and then deploy the fix. Shipping without testing risks a bad customer experience. Manual tests slow down product development.

For agents using LangSmith Deployment, Engine now validates its proposed fixes before putting them in your queue for review. When Engine v2 spots an issue, it first reproduces the failure by running the offending inputs against your agent in Deployment. Then, it starts the fix development process: proposing a change, testing it against the same inputs, evaluating the results, then adjusting the fix. With each test, the fix gets stronger. Once Engine confirms the fix resolves the issue, it provides the fix to the user in LangSmith, where they can open a PR with a click for quick deployment.

This feature is available in Private Beta today to existing LangSmith Deployment users. Sign up here if interested.

Improving Engine’s baseline performance

In addition to new capabilities, we’re continually improving Engine’s performance against its core tasks. When Engine finds higher-impact issues and produces more effective fixes, users can resolve problems in their agents faster.

We make regular improvements to Engine’s harness, prompts, and underlying models to raise performance against internal and external benchmarks. As we shared in August, Engine is now more than 2x better at detecting issues (as measured by IssueBench) and provides fixes that are 25% more effective (as measured by Terminal-Bench).

What’s next: BYOK for self-hosted Engine

For teams with strict data policies that operate LangSmith in self-hosted environments, our next release of self-hosted LangSmith will support bring-your-own-key (BYOK) for Engine. This enables teams to provide their own model API key to power inference for Engine, while trace data stays within your VPC. We will share more details on BYOK for Engine with our next self-hosted release.

Get started

Engine v2 is available in SaaS deployments for all LangSmith Plus and Enterprise plans. Support for Engine v2 in Self-Hosted plans will be coming shortly. Pricing details can be found here.

If you’re on a Plus plan, you can enable Engine with just a few clicks.

If you’re on an Enterprise plan, you can reach out to your account team to get started.

New to LangSmith? Get started today.