LogoTrendingDevTools
icon of Promptfoo

Promptfoo

Open-source toolkit for evaluating, testing, and red-teaming LLM apps and model prompts.

Visit Website

Introduction

Overview

Promptfoo is an open-source CLI and library for testing LLM applications rather than relying on trial and error. It lets teams define prompts, test cases, providers, and evaluation criteria, then compare outputs in structured results and matrix views. The project covers both quality evaluation and security testing: its official documentation describes benchmarking prompts, models, and RAGs for a specific use case, while its red-team workflow generates adversarial inputs and measures how an application responds. Promptfoo can run as a command-line tool, a Node.js library, or inside CI/CD workflows, and the project states that evaluations run locally and communicate directly with the selected LLM provider (official introduction).

The product is particularly relevant as an application-testing layer around AI systems. It is not presented as a model provider or a general observability platform. Instead, it helps developers make repeatable tests for systems whose behavior can change with prompts, models, retrieved context, tools, or application code. The official site describes testing agents, RAG applications, chatbots, and other LLM apps for problems such as prompt injection, jailbreaks, data leakage, harmful output, and insecure tool use (red-team overview).

Key Features

  • Automated evaluations: Define test cases and expected behavior, run prompts against one or more providers, and score outputs with deterministic or model-graded metrics. Caching, concurrency, and live reloading are documented as ways to speed iteration (introduction and workflow).
  • Model and prompt comparison: Promptfoo supports side-by-side comparisons and a broad provider catalog, including OpenAI, Anthropic, Azure, Google, Hugging Face, open-source models, HTTP endpoints, and custom JavaScript, Python, Ruby, or shell providers (provider documentation).
  • Red teaming and vulnerability scanning: Its red-team system creates simulated adversarial inputs and evaluates responses for application- and model-layer risks. The official site specifically documents prompt injections, jailbreaks, PII leaks, indirect injections, unauthorized data access, privilege escalation, and other threats (red-team documentation).
  • CI/CD quality and security gates: Teams can run evaluations or red-team scans in pipelines, export JSON, HTML, or JUnit results, and fail builds when thresholds are not met. Official examples cover GitHub Actions, GitLab CI, Jenkins, scheduled scans, and other platforms (CI/CD guide).
  • Local and shared result workflows: The open-source tool includes a local web viewer and sharing workflow. A basic Docker-based server can store shared evaluations, while Enterprise adds team controls, dashboards, API access, monitoring, and remediation features (self-hosting guide and Enterprise overview).
  • Active provider and feature development: The official release notes document new providers, adaptive rate limiting, code-scanning improvements, and red-team plugins (release notes).

Use Cases

Promptfoo fits teams building or maintaining AI features that need repeatable regression checks. A developer can compare several prompts or models against a fixed dataset before changing a production workflow. An evaluation team can add assertions for correctness, relevance, format, or other application-specific requirements. An AppSec team can run adversarial probes against a chatbot, RAG system, or agent to find prompt injection, leakage, harmful-output, and tool-use weaknesses before launch. A platform team can put evaluations and security scans into pull requests or scheduled CI/CD jobs, export machine-readable results, and enforce quality gates (CI/CD documentation).

It also serves organizations that need local execution or infrastructure control. The Community edition runs locally and may be self-hosted, while the official Enterprise On-Prem option is designed for deployments behind a firewall with a dedicated runner and network isolation (pricing, Enterprise deployment options). These paths make Promptfoo suitable for private development environments, internal testing platforms, and teams that cannot send evaluation results to a managed service by default.

Best for

Promptfoo is best for AI application developers, ML or evaluation engineers, AppSec teams, and platform teams that want test-driven development for prompts, agents, and RAG systems. It is a strong fit when the team wants declarative test cases, many provider integrations, local execution, CLI automation, and security-oriented red teaming in the same workflow. Smaller teams can start with the open-source Community version; organizations needing RBAC, centralized reporting, continuous monitoring, SSO, SLAs, or dedicated deployment support should evaluate the paid Enterprise offerings (pricing and feature comparison).

Alternatives

Braintrust is a broader evaluation and observability platform for teams that want managed experiment tracking, datasets, and production feedback around AI systems; it is not simply the same product as Promptfoo's open-source, CLI-first testing and red-teaming workflow. Comet is generally associated with experiment tracking and ML development workflows, so it is a different fit from Promptfoo's specialized prompt evaluation, LLM security scanning, and CI/CD red-team use cases. Langfuse focuses on LLM observability, tracing, prompt management, and production analytics; it can complement Promptfoo, but its primary documented role is not the same as Promptfoo's adversarial testing and pre-deployment evaluation focus.

Pricing

The official pricing page lists Community — Free Forever, with all LLM evaluation features, all model providers and integrations, red teaming up to 10,000 probes per month, custom integration with an app, local or self-hosted operation, vulnerability scanning, and community support (official pricing). The page lists Enterprise — Custom pricing and On-Premise — Custom pricing; no numeric dollar amount is published in the reviewed official source. Enterprise adds custom red-team limits, team collaboration, continuous monitoring, centralized security/compliance dashboards, configurable attack profiles, SSO, granular permissions, API access, managed cloud deployment, and priority support. On-Premise adds deployment on the organization’s infrastructure, complete data isolation, a dedicated runner, and an assigned deployment engineer.

Basic self-hosting is documented through an official Docker image, Docker Compose, Helm chart, or a build from source (self-hosting documentation). However, the same guide warns that this basic self-hosted server is not recommended for production: it uses local SQLite, requires manual persistence management, has no built-in authentication or SSO, has no multi-team or role-based access control, and does not support horizontal scaling because jobs live in server memory and replicas cannot share the database. Enterprise On-Prem is the documented production-oriented route for network isolation and team deployment.

Frequently Asked Questions

Is Promptfoo open source?

Yes. The official repository describes Promptfoo as an open-source CLI and library, and the repository’s LICENSE file contains the MIT license text.

What does the free Community plan include?

The official pricing page says Community is free forever and includes LLM evaluations, model providers and integrations, vulnerability scanning, local or self-hosted execution, and red teaming up to 10,000 probes per month (pricing).

Can Promptfoo test an agent or RAG application?

Yes. Promptfoo’s official red-team documentation discusses testing RAG systems, agents, and chatbots, including application-layer risks such as indirect prompt injection, context-data leakage, and insecure tool or API use (red-team overview).

Can it run in a CI/CD pipeline?

Yes. The official CI/CD guide provides commands for evaluations and red-team scans, output formats including JSON, HTML, and JUnit XML, quality gates, and examples for GitHub Actions, GitLab CI, Jenkins, and scheduled workflows (CI/CD guide).

Is basic self-hosting production-ready?

The official self-hosting guide says it is not recommended for production use cases. It documents SQLite persistence, no built-in authentication or SSO, no multi-team support or RBAC, and no horizontal scalability; the guide directs production deployments needing those capabilities to the Enterprise platform (self-hosting guide).

ℹ️

Affiliate Disclosure: TrendingDevTools may earn a commission if you sign up through links on this page, at no extra cost to you.

Newsletter

Join the Community

Subscribe to our newsletter for the latest news and updates