Roadmap

Attach evaluators to playground sessions and see scores inline as you iterate on prompts. Connect test sets to keep prompt iteration and data curation in one loop.

Build a review queue from traces or test set rows, attach a scoring schema, and route it to reviewers. Export finished queues as labeled test sets that feed straight into your evaluators.

Trigger CI and repository workflows when you deploy a prompt. Send deployment events to your own HTTPS endpoint or call GitHub directly with repository dispatch and workflow dispatch.

Connect 150+ external tools (Gmail, Slack, Notion, Google Sheets, GitHub) to your prompts directly from the playground. Authenticate with OAuth, attach tool actions, and execute tool calls with one click.

Refine prompts with AI directly in the playground. Describe what you want to improve and get a refined version with an explanation of the changes.

Multi-organization support, SSO with any OIDC provider, domain verification with auto-join, and a US region.

Create folders and subfolders to organize prompts. Drag prompts between folders and search across everything.

We are making it easy to use and navigate in the playground with large testsets.

Create reusable prompt snippets that can be referenced across multiple prompts. Reference specific versions or always use the latest version to maintain consistency across prompt variants.

Add a button in observability to open any chat span directly in the playground. Creates a stateless playground session pre-filled with the exact prompt, configuration, and inputs for immediate iteration.

A richer editing experience for LLM-as-a-Judge and other evaluators, with inline test runs and the ability to evaluate evaluators against a labeled test set to measure agreement with ground truth.

Build and configure AI agents directly from the Agenta UI. Define agent workflows, select tools, and set up orchestration logic without writing code. Test and iterate on agent behavior in the playground, then deploy to production with versioning and observability built in.

Turn annotated traces into labeled test cases directly from an annotation queue. Export reviewed traces with ground-truth labels as a new or existing test set in one action.

Set usage limits for traces at the project level. Configure a hard cap to stop accepting new traces once the limit is reached, or a soft cap to receive an alert while continuing to accept traces. Gives teams cost predictability and control in production.

We are adding the ability to cache prompts in the SDK.

We are adding the ability to tag traces, testsets, evaluations and prompts. This is useful for organizing and filtering your data.