weidutech/PentestPi

Autonomous Penetration Testing Agent with External State Memory & Anti-False-Falsification Guardrails

English • 简体中文 • Quick Start • Architecture • Benchmark Validation • State v1 SPEC

---

💡 Overview & Core Philosophy

"Model > Tools > Harness"
The model decides how to think; tools decide what can be observed; the harness provides a reliable workbench so both operate consistently without losing ground.

PentestPi is a production-grade Autonomous Penetration Testing Agent Framework built on top of the Pi agent engine. Designed for real-world enterprise red teaming, complex CTF challenges, and AI cybersecurity benchmarks (such as TSecBench).

In the Baidu Agent+ Cyber Attack & Defense National Competition, PentestPi scored 13,955 / 23,160 points (~60% total score) in just ~4.3 hours, successfully advancing to the Top 15 nationwide.

💎 Core Architecture & Design Principles

PentestPi replaces complex multi-agent blackboard systems and rigid multi-layer orchestration with an engineering-focused, lean architecture:

1. Lean Single-Agent Core: Maximizing Foundation Model Reasoning

• Eliminating Multi-Agent Coordination Overhead: Traditional multi-agent frameworks often expend significant context tokens and execution cycles on task delegation, role-switching, and format reconciliation. PentestPi employs an end-to-end single-agent paradigm, using a unified context from initial reconnaissance to exploitation, minimizing handoff degradation and context down-sampling.
• Unconstrained Tool Execution: Avoids rigid DAG constraints, allowing the foundation model to dynamically reason, select tools, and interact with offline security arsenals based on observed findings.

2. Dual-Dimension Observability (Intent Layer & Traffic Layer)

• 🧭 Intent & Tactical Trail Observability: A lightweight background Observer Agent asynchronously analyzes session logs every 30 seconds, projecting concrete tactical goals and a monotonically progressive milestone trail (e.g. START ➜ S1:Port Recon ➜ S2:Auth Bypass Found ➜ [S3:LFI Flag Extraction]) directly onto the terminal status bar.
• 📡 Wire-Level Raw HTTP Traffic Capture: An automatic background proxy intercepts all outbound terminal HTTP traffic into a single consolidated log raw_traffic.http. Developers and operators can inspect raw wire traffic or export exact reproducing curl commands for PoC validation.

3. Long-Horizon Compaction Strategy: Preserving Critical Findings

• Zero Loss of Key Evidence: Standard LLM agents risk losing credentials, API keys, or subtle error traces during aggressive context truncation.
• Pre-Compaction State Synchronization: By hooking into Pi's session_before_compact lifecycle, PentestPi synchronizes verified facts and parameters to external storage prior to token compaction, maintaining consistency across multi-hour tasks.

4. Establishing an Industry Consensus: State v1 SPEC (pentest_state.md)

Just as CLAUDE.md became the de facto standard specification for AI coding agents, our ambition is for pentest_state.md to become the universal open consensus SPEC for autonomous penetration testing and offensive security agents.

• Not Just an Internal File, but an Open Industry Protocol: Eliminates proprietary agent blackboards and complex graph database lock-in. Defined in clean, verifiable Markdown as the open State v1 SPEC.
• Self-Consistent Offensive Knowledge Graph: Atomically tracks the entire operation via strictly incrementing, non-reusable reference IDs: [F] Objective Facts, [H] Hypotheses, [E] Bounded Experiments, [C] Coverage & Suspensions, [N] Next Actions, and [R] Cognition Revisions.
• Engine-Agnostic & Zero Friction Human Handover: The SPEC is strictly independent of any single agent engine. Any autonomous offensive framework, benchmark runner, or human red team operator can inspect, validate (via the included tools/check_pentest_state.py), and pick up the state seamlessly.

5. Modular Offensive Playbooks Matrix (playbooks/)

• On-Demand Loading & Zero Token Bloat: Replaces prompt-stuffing with modular, actionable playbooks (e.g. Redis unauthenticated takeover, FastCGI RCE, JWT analysis).
• Dual Invocation Pipeline: Supports both autonomous retrieval by the agent upon discovering environmental triggers (via playbooks/INDEX.md) and automatic terminal mount notifications when specific ports or wire signatures are observed. Operable manually via /pentest-playbook.

                    ┌───────────────────────────────┐
                    │     PentestPi 3 Core Rules    │
                    └───────────────────────────────┘
  [Evidence First]      ──► Grounded in objective facts; driven by differential & side-channel signals
  [Giving Up ≠ Falsify] ──► A failed attempt is NOT a dead end; record coverage & reopen criteria
  [External State]      ──► Persist state to Markdown; prevent context degradation; enable resumption

---

⚡ The Pain Points & The PentestPi Solution

Why Traditional Multi-Agent Pentest Frameworks Fail

1. The "Fake Falsification (假证伪)" Trap:
   A Reason agent slices the attack surface into sub-tasks; an Explore agent runs a dictionary probe or payload test. Due to a missing word in the wordlist or an unencoded character, the probe fails. The Explore agent declares "this endpoint is not vulnerable". Downstream agents accept this verdict as an immutable fact and permanently prune the winning exploit path.
2. The High Handoff Tax & Information Degradation:
   A rich exploration trace—including subtle HTTP response lengths, timing variations, and raw headers—gets forcefully reduced into a lossy one-line blackboard summary. Key side-channel clues vanish.
3. The "DFS vs BFS" Fallacy:
   Searching algorithms assume the tree is already known. In real-world pentesting, nodes must be discovered before they can be searched. An agent cannot find /admin/info or understand an obscure JWT HMAC trick through scheduling alone—it requires specialized offensive tools, high-quality dictionaries, and deep model reasoning.

How PentestPi Solves Them

Challenge	Traditional Multi-Agent Frameworks	PentestPi Approach
Memory & Context	Long sessions degrade rapidly; context compaction loses credentials	State v1 External Memory: Structured Markdown (pentest_state.md) written to disk
Negative Signals	Single failure causes irreversible branch pruning	Coverage Tagging: Separates tested methods from untested space; "abandoning" only pauses a path
Context Compaction	Truncation corrupts attack traces and hallucinated states	Anti-Loss Lifecycle Extension: Auto-syncs state in session_before_compact before tokens overflow
Time-Consuming Tasks	Full scans or brute-forcing block the main reasoning loop	Asynchronous Subagents: Delegated to background subagents without freezing main attack thread
Execution runaway	Unbounded Cartesian brute-force explosions freeze the CPU/network	Bounded Probing Discipline: 10s CPU timeout on decoders, strict network latency limits

---

🏗️ Architecture & Operational Loop

flowchart TD
    subgraph Input["Task & Context"]
        A[Target Scope & Authorization] --> B[Pi Main Agent]
        P[Combat Discipline: system_prompt.md] --> B
    end

    subgraph CoreLoop["Cognition & Execution Loop"]
        B --> C[Observe Facts 'F' & Form Hypotheses 'H']
        C --> D[Select Bounded Experiment 'E']
        D --> T[Curated Offline Tools: tools/]
        D --> S[On-Demand Subagents: pi-subagents]
        T --> R[Raw Response & Loot: evidence/]
        S --> R
        R --> E_STATE[Update Mission State: State v1]
        E_STATE --> B
    end

    subgraph ExternalMemory["Durable External State"]
        E_STATE --> M[missions/code/pentest_state.md]
        EXT[.pi/extensions/pentest-pi-compact.ts] -->|Compaction Hook & Periodic Sync| M
        EXT -->|Durable Full Output Save| R
        EXT -->|Resume Session Pointer| B
    end

1. Execution Base (Base Engine)

Built on top of Pi, a lean general-purpose agent engine. The frontier LLM dynamically selects tools and manages reasoning without black-box server dependencies.

2. Combat Discipline (prompts/system_prompt.md)

• Evidence First: Inspect source code, leaks, and error messages before guessing. Never attempt cookie or JWT forgery without the raw token in hand.
• Giving Up ≠ Falsification: A static status code simply means side-channels haven't been observed yet. Mark abandoned paths as Postponed with clear reopening criteria.
• Side-Channel Cost Ordering: Status code / length < body diff < timing blind injection (most expensive comes last).
• Zero Resource Blackholes: Strictly forbids Python Cartesian brute-force loops ($256^3+$); offline decryption scripts are capped at 10 seconds.

3. State v1 Open Industry Specification (docs/state/SPEC.md)

Serving as the security domain's equivalent to CLAUDE.md, this open protocol standardizes offensive agent state into 8 strictly defined sections with persistent, non-reusable IDs:

• [F01] Facts: Verified objective observations.
• [H01] Hypotheses: Deductions and attack theories.
• [E01] Experiments: Commands executed, expected vs actual outputs.
• [C01] Coverage & Postponed: Methods tested, unverified spaces, reopening triggers.
• [N01] Next Actions: Prioritized actionable steps.
• [R01] Revisions: Audit trail of overturned assumptions.
• Schema Validation Tool: Bundled with tools/check_pentest_state.py for automated compliance checking across disparate agent implementations.

4. Anti-Loss Lifecycle Extension (.pi/extensions/pentest-pi-compact.ts)

• Pre-Compaction Sync: Intercepts session_before_compact events to invoke a secondary model that merges newly discovered evidence into pentest_state.md.
• Durable Full-Output Archival: Copies extensive command outputs (details.fullOutputPath) to disk, preventing truncated terminal logs from destroying critical tokens or hashes.
• Atomic State Protection: Validates State v1 schemas before persisting, preventing corrupt or truncated states.

5. Tactical Playbooks Engine (playbooks/)

• Indexed Discovery (playbooks/INDEX.md): Compact index mapping port, protocol, and header triggers to tactical guides.
• Action-Oriented SPEC (playbooks/SPEC.md): Standardized 4-section format: Triggers, Pre-checks, Attack Paths, and Pitfalls.
• Autonomous & Reactive: Agent pulls playbooks upon encountering triggers; terminal extension raises proactive notifications when high-signal services (Redis, FastCGI, JWT) are detected.

---

🚀 Quick Start

Prerequisites

• OS: Linux / macOS / WSL2
• Node.js: >= 18.0.0
• Python: >= 3.10
• Pi Agent CLI: Installed and configured with your preferred LLM provider

[!TIP] 💡 Project-Scoped Isolation (Zero Global Pollution)
This project does NOT modify or overwrite your global pi CLI installation. All penetration testing extensions, system prompts, and /pentest-* slash commands are strictly scoped inside this repository via the local .pi/ directory. Running pi in any other directory remains your regular, untouched environment.

1. Installation

# Clone the repository
git clone https://github.com/weidutech/PentestPi.git
cd PentestPi

# Install essential subagent capability
pi install npm:pi-subagents

# Copy environment configuration (Optional: TSecBench / Vision Models)
cp .env.example .env

2. Interactive Combat Terminal (Recommended)

Simply launch pi to start the dedicated PentestPi terminal with auto-managed proxy sidecar and real-time TUI dashboard:

pi

• Clean Standby: Boots cleanly into [standby] mode without polluting the directory.
• Natural Language Mission Binding: Simply prompt "Start attacking a-01" or "Test target c-06"; the dashboard binds to the target mission and prepares evidence storage instantly.
• Progressive Tactical Trail (Observer Agent): An out-of-band observer runs every 30s to distill the target's goal and solidify tactical milestones monotonically (e.g., START ➜ S1:Recon ports ➜ [S2:Audit auth bypass]).

3. Pentest Command Family (/pentest-*)

Type /pen + Tab inside the interactive terminal to access offensive commands:

Command	Description & Examples	Aliases
/pentest-mission	Inspect, switch, or reset active target • /pentest-mission (view active) • /pentest-mission c-06 (switch target) • /pentest-mission reset (return to standby)	/mission
/pentest-traffic	View captured HTTP request counts & raw wire log status	-
/pentest-milestone	View confirmed attack milestones or manually anchor breakthroughs • /pentest-milestone (view active milestone path) • /pentest-milestone Exploit LFI to read flag (anchor breakthrough)	-
/pentest-playbook	Inspect & mount scenario-specific offensive playbooks • /pentest-playbook (list available playbooks & triggers) • /pentest-playbook show redis_unauth (view playbook content) • /pentest-playbook apply redis_unauth (mount playbook to active session)	-

4. Wire Log Audit & PoC Reproduction (tools/traffic)

All raw HTTP wire frames are consolidated chronologically into a single IDE-compliant file:
missions/<mission>/evidence/raw_traffic.http

Inspect and reproduce flows using the built-in query CLI:

# 1. List all flows captured for a mission
python3 tools/traffic/query_traffic.py --mission a-01 --list

# 2. Dump raw wire request and response
python3 tools/traffic/query_traffic.py --dump <traffic_id>

# 3. Generate instant, ready-to-run curl command for PoC reproduction
python3 tools/traffic/query_traffic.py --curl <traffic_id>

5. Automated Batch Scripting (run_mission.sh)

For CI/CD or automated benchmark evaluation:

# Usage: ./run_mission.sh <mission_code> "<prompt_and_scope>"
./run_mission.sh c-06 "Target is 10.10.10.128. Test within scope and save flag to flag.txt upon exploitation."

6. State Validation Linter

Validate State v1 Markdown schemas:

python3 tools/check_pentest_state.py missions/c-06/pentest_state.md

---

🏆 Benchmark Validation & Architecture Evolution

PentestPi's engineering design has been validated in competitive red-teaming environments and standardized benchmarks:

4-Stage Architectural Progression

1. v0 Centralized Blackboard Architecture:
   Separate reasoning and exploration agents coordinating over a shared task graph.
   Issues: Frequent false falsification—sub-agents failed on minor payload encoding differences and falsely marked attack paths as invulnerable, blocking downstream reasoning.
2. v1 File-based Blackboard Coordination:
   Replaced centralized database services with local structured Markdown, but still retained heavyweight handoff protocols. Baseline success rate remained around 16%–27%.
3. v2 End-to-End Single-Agent Pipeline:
   Removed outer planning and handoff layers, enabling the main agent to traverse reconnaissance through exploitation directly. Achieved 11,160 pts (48%), demonstrating the advantage of reducing coordination overhead.
4. v3 PentestPi Production Release (Lean Core + State v1 + Observability)：
   Integrated State v1 external memory, lifecycle compaction hooks, and full-spectrum observability. Reached 13,955 pts (Top 15 nationwide) in the Baidu Agent+ Cyber Attack & Defense Competition.

---

🧰 Offensive Tool Matrix

PentestPi bundles essential offensive binaries, wordlists, and solvers inside tools/ to ensure deterministic execution in air-gapped or sandbox environments:

tools/
├── bin/                   # Pre-compiled offensive binaries
│   ├── nmap               # Fast network recon & banner grabbing
│   ├── arjun              # Advanced HTTP parameter discovery
│   ├── feroxbuster        # High-speed recursive directory fuzzing (Rust)
│   ├── jwt_tool           # Comprehensive JWT auditing & tampering
│   ├── flask-unsign       # Flask session decoding, cracking & signing
│   ├── tblind             # Automated differential timing/boolean blind SQLi
│   ├── kpa-probe          # Known-plaintext keystream difference solver
│   ├── z3 / python-tools  # SMT constraint solvers (reverse engineering & crypto)
│   └── radiff2 / rizin    # Binary disassembly and patch analysis
├── SecLists/              # Curated offensive wordlists
└── wordlists/             # High-yield internal port lists, parameter names & paths

---

📂 Project Structure

PentestPi/
├── .pi/
│   ├── extensions/
│   │   └── pentest-pi-compact.ts   # Core: State auto-sync & anti-loss compaction extension
│   └── settings.json               # Pi settings declaring npm:pi-subagents
├── prompts/
│   └── system_prompt.md            # Combat Discipline (human-readable operational rules)
├── missions/                       # Runtime mission workspaces (gitignored)
│   └── <mission-code>/
│       ├── pentest_state.md        # State v1 persistent task intelligence
│       └── evidence/               # Raw HTTP responses, loot, and flags
├── playbooks/                      # Modular offensive playbooks (on-demand loading)
│   ├── SPEC.md                     # Playbook authoring specification
│   ├── INDEX.md                    # Trigger mapping & tactical index
│   ├── services/                   # Service exploitation guides (Redis, FastCGI)
│   ├── web/                        # Web vector guides (JWT auditing)
│   └── custom/                     # User-defined tactical playbooks
├── tools/                          # Built-in offensive tool matrix, scripts & wordlists
├── docs/                           # Specifications and state schema documents
│   └── state/                      # State v1 SPEC, TEMPLATE, and EXAMPLES
├── run_mission.sh                  # 1-Click mission launcher
└── .env.example                    # Environment configuration template

---

🗺️ Roadmap

• State v1 Specification: Structured F/H/E/C/N/R schema & automated linter.
• Anti-Loss Compaction Extension: Pre-compaction state merging & tool log persistence.
• Offensive Toolchain Bundling: Curated binaries for web, binary, crypto & SQLi.
• Battle Validation: Top 15 finish in Baidu Agent+ National Hackathon (13,955 pts).
• State v2 Topology: Multi-host lateral movement and network pivot tracking.
• Dynamic Model Routing: Fast model for reconnaissance, high-reasoning model for complex exploitation.
• Automated Report Generation: One-click compilation of pentest_state.md into executive and technical PDF/HTML reports.

---

🤝 Contributing

Contributions from the cybersecurity and AI agent communities are warmly welcomed!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/CoolFeature)
3. Commit your Changes (git commit -m 'Add CoolFeature')
4. Push to the Branch (git push origin feature/CoolFeature)
5. Open a Pull Request

---

⚖️ Legal & Ethical Disclaimer

[!CAUTION] PentestPi is developed solely for authorized penetration testing, red teaming assessments, and CTF security research.
Users must obtain explicit written authorization before testing any network, system, or digital asset. The developers assume no liability and are not responsible for any misuse, damage, or legal consequences caused by this software. Use responsibly and abide by all applicable cyber laws and regulations.

---

📄 License

This project is open-sourced under the Apache License 2.0.

---

_{Built with precision for security researchers, red teamers, and AI agent developers.}