Awesome AI Testing

A curated list of AI-powered testing tools, frameworks, and resources for QA engineers.

AI is reshaping software testing. This list collects tools, platforms, and resources that use AI or LLMs to generate tests, heal broken locators, triage failures, write assertions in natural language, and more. Both open source and commercial offerings are included, marked with badges so you can filter by what fits your stack.

Contents

• Legend
• Test Generation
• MCP-Based Testing
• Self-Healing Test Frameworks
• AI-Powered E2E Platforms
• Mobile AI Testing
• Visual AI Testing
• Natural Language Test Authoring
• LLM-as-Judge Evaluation
• Test Analytics and Triage
• Code Coverage with AI
• AI Test Data Generation
• Mock and Service Virtualization
• Performance Testing with AI
• AI for Accessibility Testing
• API Testing with AI
• LLM and AI System Testing
• Browser Automation for AI Agents
• Articles and Talks
• Courses and Tutorials
• Newsletters and Communities
• Benchmarks and Datasets
• Related Awesome Lists

Legend

• 🆓 Open source
• 💰 Commercial
• 🆓💰 Open core (free tier or open source with paid features)

Test Generation

Tools that generate test cases from code, requirements, or user behavior using AI.

• Qodo-Cover 🆓 - AI-powered tool for automated test generation and code coverage enhancement.
• EvoMaster 🆓 - First open source AI tool that automatically generates test cases via evolutionary algorithms for REST, GraphQL, and RPC APIs.
• EvoSuite 🆓 - Generates JUnit tests using evolutionary and genetic search-based algorithms for Java.
• Pynguin 🆓 - Automated unit test generation for Python via evolutionary algorithms, from Saarland University.
• Symflower 💰 - Commercial tool that combines symbolic execution, static analysis, and LLMs to generate unit tests for Java, Go, and Kotlin. Free CLI available with limitations.
• CodiumAI / Qodo 💰 - AI assistant generating meaningful tests from code context.
• Diffblue Cover 💰 - Autonomous Java unit test writer using reinforcement learning.
• GitHub Copilot 💰 - AI pair programmer that generates test code in Playwright, Cypress, Selenium across editors.
• Cursor 💰 - AI-first code editor with strong test generation capabilities for major frameworks.
• Claude Code 💰 - Anthropic's terminal-based agentic coding assistant, useful for test suite generation and refactoring.

MCP-Based Testing

Tools and servers that use the Model Context Protocol to give AI agents browser control and testing capabilities.

• Playwright MCP 🆓 - Official Playwright MCP server giving AI agents full browser control through structured accessibility snapshots.
• Playwright CLI 🆓 - Token-efficient CLI for coding agents like Claude Code and GitHub Copilot, with installable skills.
• Chrome DevTools MCP 🆓 - Official MCP server from the Chrome DevTools team, with 26 tools for browser automation, debugging, and performance analysis.
• ExecuteAutomation Playwright MCP 🆓 - Community Playwright MCP server with API testing support and 143 device emulation profiles.
• BrowserTools MCP 🆓 - Browser monitoring and console log access for AI agents via Chrome extension.
• Browser MCP 🆓 - Popular MCP server that automates the user's own local browser, preserving logged-in sessions and avoiding bot detection. Note: limited maintenance activity since mid-2025 but widely used (6.5k+ stars).
• Podium MCP 🆓 - MCP server purpose-built for mobile app testing on Android/iOS simulators using Maestro, with deep Redux state inspection and CI pipeline integration.
• prufa-mcp 🆓💰 - Open-source (Apache-2.0) MCP server connected to Prufa's hosted audit backend; an AI agent runs a QA audit of a web app (analytics, broken flows, security headers, accessibility) and gets back machine-verified findings. First audit is free; further audits and features require a paid plan.
• qmax-mcp 🆓 - MCP server and CLI that gives coding agents independent QA evidence before declaring a web change done: scans a URL for console errors, broken links, accessibility, Core Web Vitals, SEO, and security headers, then generates and runs a deterministic Playwright repro behind a human-approval gate.
• TestChimp 💰 - QA platform with an MCP server and CLI (@testchimp/cli) that lets coding agents read and update test scenarios, coverage and API operation data, with plans and Playwright tests kept in your Git repo.
• Kerno 🆓💰 - The AI Backend Runtime QA Engineer. Runs locally inside your coding agent's loop through MCP and tests each change against your running app and its real databases, queues and services.

Self-Healing Test Frameworks

Tools that automatically repair broken test locators and adapt to UI changes.

• Healenium 🆓 - Self-healing library for Selenium, Appium, and Playwright. Replaces broken selectors at runtime.
• CodeceptJS 🆓 - End-to-end testing framework with built-in AI heal plugin that uses OpenAI, Anthropic, or local models to repair failing steps and propose locator fixes.
• optics-framework 🆓 - Open-source test automation for mobile, web and Smart TV with a locator fallback ladder across XPath, text, OCR and image strategies, plus optional LLM-driven self-healing that recovers failed steps through the same keyword API.
• Testim 💰 - Pioneer of self-healing tests with AI-driven smart locators.
• Functionize 💰 - AI-powered tests that adapt without selectors.
• TestSigma 💰 - AI-driven low-code platform with self-healing across web, mobile, and API.
• Tricentis Tosca 💰 - Enterprise platform with Vision AI for resilient automation.
• Perfecto 💰 - Cloud testing platform with self-healing locators.

AI-Powered E2E Platforms

End-to-end testing platforms with AI at the core.

• TestZeus Hercules 🆓 - World's first open-source testing agent for UI, API, security, accessibility, and visual validations, no code required.
• agent-qa 🆓 - Self-improving QA agent for natural-language web and mobile tests with run memory, UI-change adaptation, and regression detection.
• Endtest 💰 - Agentic AI test automation platform for web and mobile apps, featuring autonomous test generation, natural language authoring, self-healing, and cloud execution.
• Mabl 💰 - Low-code platform with auto-healing and ML-based test maintenance.
• Meticulous 💰 - Records real user sessions and generates regression tests automatically.
• Momentic 💰 - AI-native end-to-end testing platform that writes, runs, and maintains web and mobile tests automatically using natural language.
• Autify 💰 - No-code end-to-end testing platform with AI-driven maintenance.
• Reflect 💰 - No-code regression testing with AI-assisted authoring.
• QA Wolf 💰 - AI-powered QA-as-a-service generating Playwright tests at scale.
• QualityMax 💰 - Independent AI QA platform that generates, executes, and gates tests across web, mobile, API, performance, and security, with self-healing automation.
• Bug0 💰 - Agentic testing platform built on the Planner, Generator, Healer pattern with MCP integration.
• Checksum 💰 - Generates Playwright and Cypress tests from real user sessions.
• Rainforest QA 💰 - No-code testing platform with AI-driven test generation.
• BrowserStack Low Code Automation 💰 - AI-powered low-code platform with self-healing agents, NL test authoring, and real device cloud execution.
• KaneAI 💰 - GenAI-native test agent for authoring, executing, and maintaining tests using natural language.
• Virtuoso QA 💰 - NLP-based codeless E2E platform with self-healing and visual testing.
• Katalon Studio 🆓💰 - Test automation platform with AI features including TrueTest and Visual Testing.
• Testorim 💰 - Describe a web test in plain English and an AI agent plans the steps, runs them in a real Chromium browser, and reports what happened with video, a Playwright trace, and an accessibility scan.
• Kane CLI 🆓💰 - Testing agent and assurance layer for coding agents that turns a PRD, ticket or spec into acceptance criteria and tests, runs them on the real web or mobile app, and reports which requirements a run has proven, with step-by-step evidence.

Mobile AI Testing

AI-powered tools specifically for mobile app testing.

• Maestro 🆓💰 - YAML-based mobile UI automation that reads accessibility tree, no XPath needed.
• Appium 🆓 - Industry standard mobile automation, with AI plugins for self-healing.
• Sofy 💰 - No-code AI mobile testing platform.
• Kobiton 💰 - Mobile device cloud with AI-driven scriptless automation.
• HeadSpin 💰 - Mobile testing platform with AI-driven performance insights.
• testRigor 💰 - Plain English test authoring across web, mobile, and API with AI-driven element identification.

Visual AI Testing

Visual regression and UI verification powered by AI.

• Loki 🆓 - Visual regression testing for Storybook.
• Argos 🆓💰 - Open source visual testing for engineering teams.
• BackstopJS 🆓 - Visual regression testing for responsive web UIs.
• Pixelmatch 🆓 - Pixel-level image comparison library.
• Reg-Suit 🆓 - Visual regression testing workflow with publish/notify integrations for CI.
• Applitools Eyes 💰 - Visual AI platform with cross-browser and cross-device verification.
• Percy 💰 - Visual review and regression testing, part of BrowserStack.
• Chromatic 💰 - Visual and interaction tests for Storybook.
• Happo 💰 - Cross-browser screenshot testing with Playwright, Cypress, and Storybook integrations.

Natural Language Test Authoring

Write tests using plain English (or other natural languages).

• Shortest 🆓 - QA via natural language AI tests, built on Playwright.
• Magnitude 🆓 - AI-native, vision-first testing framework that lets you write E2E tests in plain language.
• Auto Playwright 🆓 - Run Playwright tests with AI through plain text prompts.
• Passmark 🆓 - Open-source AI regression testing framework on Playwright with intelligent caching, auto-healing, and multi-model verification.
• Midscene.js 🆓 - AI-driven UI automation with natural language commands.
• Tusk 💰 - AI agent that auto-generates and runs tests for pull requests.

LLM-as-Judge Evaluation

Use LLMs to evaluate test outputs, assertions, and quality.

• Promptfoo 🆓💰 - Test framework with LLM-as-judge for prompts, models, and RAG pipelines.
• DeepEval 🆓💰 - Pytest-like LLM evaluation framework with built-in judge metrics.
• Ragas 🆓 - Evaluation framework for RAG pipelines using LLM judges.
• Inspect AI 🆓 - LLM evaluation framework from the UK AI Safety Institute.
• Coder Eval 🆓 - Framework for testing AI coding agents (Claude Code, Codex, Gemini) and their skills by running them in a sandbox against declarative YAML tasks. Grades the resulting files and commands with weighted criteria, LLM- and agent-judged rubrics, and skill-activation precision/recall gates, then fails CI on regressions.
• TruLens 🆓 - Evaluation framework for LLM apps with feedback functions and tracing.
• Arize Phoenix 🆓 - Open-source LLM observability and evaluation.
• Weights & Biases Weave 🆓💰 - Weights & Biases toolkit for tracing, debugging, and evaluating generative AI applications with built-in LLM-as-judge metrics and dataset management.
• OpenAI Evals 🆓 - Framework for evaluating LLMs and an open-source registry of benchmarks from OpenAI. No longer actively maintained for new evals, but still widely used as a reference.
• lm-evaluation-harness 🆓 - EleutherAI's framework for few-shot evaluation of language models, backing the Hugging Face Open LLM Leaderboard.
• Langfuse 🆓💰 - Open source LLM observability, tracing, and evaluation platform.
• Helicone 🆓💰 - Open source LLM observability and prompt evaluation platform.
• Opik 🆓💰 - Open-source LLM evaluation and observability platform with automated tracing, LLM-as-judge metrics, and pytest integration for CI pipelines.
• LangWatch 🆓💰 - Open-source LLM evaluation and AI agent testing platform combining end-to-end scenario simulation, observability, and prompt management in a single unified loop.
• Evidently 🆓💰 - Open-source Python library for evaluating, testing, and monitoring ML and LLM systems with 100+ built-in metrics for data quality, drift detection, and LLM output quality.
• LangSmith 💰 - LangChain's platform for testing and monitoring LLM apps.
• Braintrust 💰 - LLM eval platform with experiments, datasets, and observability.
• Patronus AI 💰 - Automated evaluation and security testing for LLMs.
• Respan 🆓💰 - LLM observability, evaluation, and prompt management platform with a multi-provider gateway.

Test Analytics and Triage

AI for failure analysis, flaky test detection, and reporting.

• agenttrace 🆓 - Local-first TUI and CLI for evaluating AI coding agent sessions with cost, token, latency, and failure regression gates.
• ReportPortal 🆓💰 - Open source results management with ML-based failure clustering.
• Allure TestOps 💰 - Test management with AI-driven analytics and flaky detection.
• Sealights 💰 - Quality intelligence platform using ML for test gap analysis.
• Trunk Flaky Tests 💰 - ML-based flaky test detection and quarantine.
• Datadog CI Visibility 💰 - Test analytics with flaky test detection across CI pipelines.
• CloudBees Smart Tests 💰 - Formerly Launchable. ML-driven predictive test selection and flaky test detection.
• BuildPulse 💰 - Flaky test detection and analytics platform.
• flaky-test-detector 🆓 - AI agent skill that detects, analyzes, and eliminates flaky tests across any test runner. Compatible with opencode, Claude Code, Cursor, Windsurf, and GitHub Copilot.

Code Coverage with AI

Tools that use AI to fill coverage gaps and prioritize testing efforts.

• Mutahunter 🆓 - LLM-based mutation testing for stronger test suites.
• Stryker Mutator 🆓 - Mutation testing framework that pairs well with AI test generators.
• coverage-guard 🆓 - AI agent skill that enforces 100% test coverage for any JavaScript/TypeScript project. Works with Vitest, Jest, react-scripts, and more. Compatible with opencode, Claude Code, Cursor, Windsurf, and GitHub Copilot.

AI Test Data Generation

Tools that use AI to generate realistic test data, fixtures, and edge cases.

• Faker.js 🆓 - Standard fake data library, often paired with AI for context-aware data.
• Datafaker 🆓 - Java/Kotlin port of Faker for realistic fake data generation in JVM-based test suites.
• Mockaroo 🆓💰 - Realistic test data generation with AI-suggested schemas.
• Synthesized 💰 - AI-driven synthetic data platform for testing.
• Tonic.ai 💰 - Generate realistic safe test data from production using AI.

Mock and Service Virtualization

Tools for mocking external services, LLM APIs, and dependencies in AI testing pipelines.

• Mountebank 🆓 - Mature open source service virtualization for HTTP, HTTPS, TCP, and SMTP protocols. Supports stubbing, mock verification, and record-playback.
• WireMock 🆓💰 - Industry standard HTTP mocking with 7.1k stars and a native MCP server in the cloud version for AI coding assistants.
• Mockoon 🆓 - GUI-based open source API mocking tool, easy to set up.
• Pact 🆓 - Contract testing framework for microservices and API consumers.
• MSW (Mock Service Worker) 🆓 - API mocking library for browser and Node.js, ideal for frontend AI testing.
• Hoverfly 🆓 - Lightweight service virtualization with proxy-based recording and replay.
• Twinbay 🆓💰 - Stateful copies of third-party APIs (Shopify, Slack, Zendesk and ten more) for testing AI agents. A model compiles outcomes you write in plain language into checks, and Twinbay grades the agent's captured API calls against them.
• OrcaReplay 🆓 - Proxy-based record and replay aimed at coding agents: captures the model API traffic together with the shell exit codes, per-turn file changes and MCP calls, then serves the recorded responses back with the network off so a whole agent session reruns deterministically.

Performance Testing with AI

AI-enhanced performance, load, and chaos testing.

• k6 🆓💰 - Open-source load testing tool, increasingly paired with AI for scenario generation.
• WebPageTest 🆓💰 - Performance testing with AI-suggested optimizations.
• LoadView 💰 - Cloud-based load testing with AI-powered insights.
• Akamas 💰 - AI-driven performance optimization and tuning.

AI for Accessibility Testing

AI-powered accessibility scanners and remediation tools.

• Pa11y 🆓 - Open-source automated accessibility testing tool.
• Axe DevTools 💰 - AI-powered accessibility scanner from Deque.
• Evinced 💰 - AI-driven accessibility testing platform.
• UserWay 🆓💰 - AI-powered WCAG compliance scanner.

API Testing with AI

AI features for API testing, schema generation, and contract validation.

• Keploy 🆓 - Generates API tests and mocks from real traffic, with AI assertion generation.
• Stepci 🆓💰 - API testing framework with AI-suggested assertions.
• Postman 🆓💰 - AI-assisted API test generation and contract validation with Postbot assistant.

LLM and AI System Testing

Tools to test LLM applications themselves (security, robustness, hallucination).

• Garak 🆓 - LLM vulnerability scanner from NVIDIA.
• DeepTeam 🆓 - LLM red teaming for prompt injection, jailbreaks, and data leaks.
• llm-security-scanner 🆓 - Red-team toolkit with OWASP LLM Top 10 alignment and Turkish payload library.
• Giskard 🆓💰 - Testing framework for LLMs and ML models.
• Guardrails AI 🆓💰 - Python framework for validating and structuring LLM outputs using composable validators covering toxicity, PII leakage, and hallucination detection.
• NeMo Guardrails 🆓 - Open-source toolkit from NVIDIA for adding programmable guardrails to LLM-based conversational systems, preventing jailbreaks, topic drift, and unsafe outputs.
• PyRIT 🆓 - Microsoft's Python Risk Identification Tool for generative AI.
• Lakera Guard 💰 - Real-time prompt injection and jailbreak detection.
• Prompt Security 💰 - Runtime prompt injection and data leak protection platform.
• Confident AI 💰 - LLM testing platform built around DeepEval.

Browser Automation for AI Agents

Browser automation libraries designed for or commonly used by AI agents.

• Browser Use 🆓 - Make websites accessible to AI agents.
• Stagehand 🆓 - AI browser automation with predictable yet flexible APIs.
• Skyvern 🆓💰 - Automate browser-based workflows using LLMs and computer vision.
• Steel Browser 🆓 - Open-source browser API for AI agents.
• Patchright 🆓 - Patched Playwright for stealth automation.
• Lightpanda 🆓 - Headless browser written in Zig, optimized for AI agents and scraping workloads.
• AgentQL 🆓💰 - Natural language query language for AI agents to interact with and extract structured data from web pages, with self-healing selectors that integrate with Playwright.
• Notte 🆓 - Web agent framework for building fast, cost-efficient AI agents that automate web interactions using natural language instructions, with browser session management, structured data extraction, and anti-detection support.
• Browserbase 💰 - Cloud browser infrastructure with natural language automation.
• stealthy-auto-browse 🆓 - Stealth browser automation for AI agents: Camoufox (Firefox) in Docker with no CDP exposure and real OS-level input, driven over an HTTP API and MCP; passes Cloudflare, CreepJS and other bot detectors.

Articles and Talks

Essential reading on AI in software testing.

• Best AI Test Generation Tools for Playwright in 2026 - Practical guide to choosing AI test generators.
• Playwright Test Agents: AI Testing Explained - Deep dive into the Planner, Generator, Healer pattern.
• Modern Test Automation with LLM and Playwright MCP - Setting up MCP for test automation.
• 20 Open-Source Projects Redefining AI + Playwright Testing - Roundup of emerging AI Playwright projects.
• Best 12 Generative AI Testing Tools 2026 - Ranked review of top GenAI testing platforms.
• Give your AI eyes: Introducing Chrome DevTools MCP - Deep dive on Chrome DevTools MCP by Addy Osmani.

Courses and Tutorials

Learning resources for AI-powered testing.

• Generative AI in Software Testing - Comprehensive course on Copilot, Claude Code, MCP, and AI agents for QA.
• Test Automation University - Free courses including AI testing modules from Applitools.
• Chrome DevTools MCP Tutorial - DataCamp's hands-on guide to Chrome DevTools MCP with AI assistants.

Newsletters and Communities

• Software Testing Weekly - Weekly testing newsletter with strong AI coverage.
• Test Guild - Podcast and community for automation testing.
• Ministry of Testing - Global testing community with AI-focused tracks.

Benchmarks and Datasets

• StructEval 🆓 - Benchmark for structured-output generation and cross-format conversion across text and renderable formats, with syntax, structural, and visual-fidelity checks.
• HELM 🆓 - Stanford CRFM's open-source Python framework for holistic, reproducible, and transparent evaluation of LLMs and multimodal models across dozens of scenarios covering accuracy, robustness, efficiency, bias, and safety.
• SWE-bench - Benchmark for evaluating LLMs on real software engineering tasks, including test fixes.
• HumanEval - Evaluating large language models trained on code.
• LiveCodeBench 🆓 - Contamination-free holistic benchmark for evaluating LLM coding abilities that continuously collects problems from LeetCode, AtCoder, and CodeForces, covering code generation, execution, and test output prediction.
• WebArena - Self-hostable web environment for building and evaluating autonomous agents on realistic, multi-site tasks.
• ClawBench 🆓 - Open benchmark for evaluating AI web agents on 283 real-world tasks across live websites, with Docker-isolated runs and request interception before irreversible actions.
• OSWorld 🆓 - NeurIPS 2024 benchmark for evaluating multimodal AI agents on open-ended tasks in real computer environments, supporting VMware, Docker, and AWS virtualization.
• tau-bench - Benchmark for evaluating tool-using language agents through dynamic conversations with simulated users and domain-specific APIs.
• AgentBench 🆓 - ICLR 2024 benchmark for evaluating LLMs as autonomous agents across eight environments including OS, database, web browsing, and game tasks.

Related Awesome Lists

• awesome-test-automation - General test automation across languages.
• awesome-testing-tools - General testing tools.
• awesome-ai-agent-testing - Testing AI agents themselves.
• awesome-ai-pentest - AI-assisted penetration testing.
• awesome-llm-security - LLM security tools and resources.
• awesome-mutation-testing - Mutation testing resources.

Contributing

Contributions are welcome. Please read the contribution guidelines before opening a pull request.