wordlist_generator

A wordlist built from your target's own words, so you test the routes, subdomains and parameters that public wordlists don't have.

The problem with shared wordlists

Most hunters fuzz with the same few lists: SecLists raft, Assetnote, OneListForAll. They are good lists, and that is the problem. On a popular program, every path in them has been requested by hundreds of people before you, and whatever they turned up has been reported.

What is left tends to be named after the target itself: a product name, an internal team, a field from an old form. No public list contains ledger.example.com or /billing/reconciliation, but the target may well mention both. It might be in an HTML comment, a footnote, or the name of an input.

The idea

Read what the target already shows and turn it into requests nobody else is sending. tomnomnom described the approach in "Who, What, Where, When"; this tool does the reading for you:

1. It collects the target's known URLs from gau and fetches the pages.
2. It pulls out words from the text, links, forms and comments.
3. It drops every word that is already in common lists (raft and the 10,000 most common English words), plus noise such as hashes, IDs and tracking parameters.

What comes out is short, usually hundreds to a few thousand words rather than tens of thousands, and specific to this one target. If someone else fuzzes the same program with raft, your list overlaps theirs very little.

What you get

The target's own vocabulary

Mode	What it extracts	Example
(default)	words from the pages' text	reconciliation, chargebacks
-dir	directory names from the URLs, dot-directories included	.git, ledger-sync
-f	file names, dotfiles included	.htaccess, export-2023.csv
-p	parameter names, ready for x8 or Arjun	invoiceId, authCode
-c	HTML comments	TODO remove /internal/debug

With --links it also follows the links and form fields on the pages, not just the archived URLs.

Related words, found by meaning

A target that uses invoice, payment and refund probably also has something called billing, receipts or refund-policy, even if the archives never saw it. With -x the tool groups the words it found by meaning and suggests words that fit inside each group. A small model runs on your CPU, so the target's words never leave your machine. It also recombines the target's own name parts: delete_account and user_profile give delete_user.

Hidden parameters

-p lists the parameter names the target already uses, and -p -x suggests new ones built from them: if the target uses invoiceId and user, it tries userId. It adds the most common real-world parameter names that sit close to the ones it found. Parameters seen in one form are worth trying on every other route; that is how many IDOR and mass-assignment bugs are found.

Use it next to a big list, not instead of one

The generated list covers what is specific to the target; raft and Assetnote still cover the common paths every site has. Run both. The suggestions from -x are guesses ranked by how well they fit, so head -n 500 takes the best ones. On real directories held out from 9 bug bounty targets, mixing 500 suggestions into raft found as many paths as raft alone at the same size, so treat -x as a way to explore the target's own topics. The details are in docs/expansion.md.

Only test targets you are allowed to test.

Install

Needs Python 3.11+ and gau.

$ go install github.com/lc/gau/v2/cmd/gau@latest
$ git clone https://github.com/SomeKirill/wordlist_generator
$ cd wordlist_generator
$ pip3 install -r requirements.txt
$ pip3 install -r requirements-semantic.txt   # optional, only for -x neighbours

Quick start

$ ./wordlist_generator.py -d example.com > words.txt              # words from the pages
$ ./wordlist_generator.py -d example.com -dir > dirs.txt           # directory names
$ ./wordlist_generator.py -d example.com -p --links > params.txt   # parameter names
$ ./wordlist_generator.py -d example.com -dir -x --limit 1000 > new-dirs.txt
$ ffuf -u https://example.com/FUZZ -w dirs.txt -w new-dirs.txt

Only the wordlist goes to stdout, so redirecting is safe. ./wordlist_generator.py -h lists every option.

Documentation

• Usage: every flag, defaults, how pages are fetched, exit codes.
• Expansion: how -x works, the cluster width (--coverage), measured results and limits.
• Guide: recipes with ffuf, feroxbuster and x8.
• Contributing: development setup, tests, code layout.
• Changelog.

Credits

• Noise filters are based on BonJarber's clean_wordlist.
• Denylists: raft-large-directories-lowercase, google-10000-english, dutch, RFC 1866.
• -x downloads on first use (nothing third-party is bundled): the snowflake-arctic-embed-xs model (Apache-2.0), Assetnote wordlists for --pool web and --pool params, and wordfreq (data CC BY-SA 4.0) for --pool web.
• The pictures embed Outfit and DM Mono (SIL Open Font License).

Licensed under the Apache License 2.0.