NIPS — Network Intrusion Prevention System

English · 简体中文

A server-side IPS that intercepts traffic on Linux, scores each packet through a rule engine plus an anomaly detector, and drops malicious packets via iptables.

Why "NIPS"? The short name is load-bearing: it is the iptables chain (NIPS), the API token variable (NIPS_API_TOKEN), the systemd unit prefix (nips-*) and the Prometheus metric prefix (nips_*). The repository name, Network-Security-Based-On-ML, describes the project; NIPS is what the code actually calls it.

Before contributing, read ARCHITECTURE.md and CONTRIBUTING.md. CI rejects simulation/mock code in networksecurity/.

How it works

Incoming Traffic
      |
      v
[Rule Engine] ------> decides: blacklist, whitelist, protocol filter,
      | abstain       rate limit, signatures   (always present)
      v
[Mounted learning detectors] ------> BLOCK on anomaly
      |
      v
[ALLOW]

The rule engine is the permanent part of the chain: deterministic, and the only stage that runs on a default configuration. Traffic it does not decide falls through to whatever detectors you mount.

The learning detectors are a plug-in, and they are off by default:

engine:
  ml:
    enabled: false        # true turns detection on for whatever is listed below
    detectors:
      - uses: kitsune     # or `lucid`, or `my_package.module:MyDetector`
      - uses: lucid
        enforce: false    # shadow mode: verdicts recorded, never enforced

With enabled: false, no ML module is imported or constructed: deleting networksecurity/engine/kitsune/ and networksecurity/engine/lucid/ leaves a working, rules-only deployment. With it on, a detector that cannot be built is logged and skipped rather than taking the process down. The interface a third-party detector implements (process_packet / configure / ready / status) and the fail-closed rules that follow from the switch are in ARCHITECTURE.md — The Detector Contract; networksecurity/engine/threshold_detector.py is a complete example to copy.

What detection here does and does not achieve is measured, not asserted: see Measured results and limits.

LUCID (a CNN-based DDoS detector) is optional in its own right as well: it needs TensorFlow (pip install -e ".[lucid]" or pip install tensorflow) and a trained model at engine.lucid.model_path. Without a model that loads, it is not mounted at all — it no longer sits in the chain as an inactive detector the status page would still list.

Algorithms

Kitsune (NDSS'18) — AfterImage incremental statistics (90 features) + a KitNET autoencoder ensemble. Trains online, no labels needed. When link-layer headers are absent (live NFQUEUE), the MAC channel uses a (protocol, ttl) proxy key so it never collapses to zero variance. Grace periods (fm_grace_period, ad_grace_period) allow warmup before detection starts, and the first 10% of the AD grace after it is spent calibrating the anomaly threshold on the frozen model; through all of that packets are logged but not blocked.
LUCID (IEEE TNSM 2020) — 1D CNN over 10-packet flow windows (11 features/packet). Off by default; needs a trained model and engine.lucid.model_path set in config. Produce the model with scripts/train_lucid.py (see "Training LUCID" below).

Note on protocol filtering: the rule engine's protocol allowlist is TCP(6) and UDP(17); anything else it inspects is blocked, including ICMP(1). In live interception, however, only TCP and UDP are redirected into NFQUEUE (interception.intercept_icmp is off by default) — so there ICMP is not inspected and not blocked: the host's own firewall decides.

Turn interception.intercept_icmp: true on to bring ICMP into the pipeline, then allow individual types through engine.rule_engine.allowed_icmp_types. Blocking the whole protocol also breaks Path MTU Discovery (type 3, "frag needed"), which blackholes large connections — a type list is the useful setting, not an all-or-nothing ban.

ICMPv6(58) has one exception to that list: the types that maintain the link itself — neighbour and router solicitation/advertisement (135/136/133/134) and "packet too big" (2) — always pass, because a host whose neighbour discovery is queued and dropped has not blocked an attacker, it has taken itself off the network. ICMPv6 echo is not in that set and stays behind the protocol filter (allow-by-type covers ICMPv4 above). Offline pcap runs (cli.py test --pcap) do exercise the protocol filter, since those packets reach the engine whatever their protocol.

Quick Start

Requirements

Python 3.12+
Linux for live interception (nfqueue + iptables, root required)
macOS / other platforms for development and offline pcap testing

1. Clone

git clone https://github.com/zimingttkx/Network-Security-Based-On-ML.git
cd Network-Security-Based-On-ML

2. Install

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Development / CI — adds pyarrow (the bundled datasets are Parquet) and scapy
# (offline pcap testing).  The runtime image installs neither.
pip install -r requirements-dev.txt

# Optional: the LUCID CNN detector needs TensorFlow, which is not part of the
# default install (the adapter stays inactive without it).
pip install -e ".[lucid]"     # or: pip install tensorflow

3. Configure

config/config.yaml drives both the engine and live interception. The blocks an operator actually touches:

interception:
  nfqueue_num: 0
  intercept_icmp: false     # redirect ICMP into NFQUEUE so the type policy applies
  safe_ips:                 # IPs that are never blocked (loopback is protected)
    - "127.0.0.1"
    - "::1"
engine:
  kitsune:
    fm_grace_period: 5000   # feature-mapping training packets
    ad_grace_period: 50000  # anomaly-detector training packets
    threshold_percentile: 99.0
  rule_engine:
    allowed_protocols: [6, 17]   # TCP, UDP; everything else blocked
    allowed_icmp_types: []       # ICMP types that pass despite protocol 1 not being listed,
                                 # e.g. [0, 3, 4, 8, 11] to keep PMTUD and ping alive.
                                 # ICMPv6 link maintenance always passes, whatever this says
    rate_limit:
      window_seconds: 1.0
      max_connections_per_window: 100
blocking:                    # BLOCK escalation policy (see "Live Interception")
  strikes_threshold: 5       # BLOCKs inside the window before a temp ban
  strikes_window: 300.0
  temp_ban_seconds: 600.0
  temp_ban_count_to_perm: 3  # completed temp bans before a permanent ban
api:
  auth_token: ""             # empty = auth disabled (dev only); NIPS_API_TOKEN overrides
  cors_origins:              # explicit allowlist — "*" is not supported
    - "http://localhost:8000"
    - "http://127.0.0.1:8000"

key	effect
`engine.ml.enabled` / `engine.ml.detectors`	whether learning detection runs at all, and which detectors to mount (built-in short names, or `package.module:ClassName` with a `params:` block). Off by default. A detector entry with `enforce: false` runs in shadow mode — its BLOCK verdicts are counted, alerted and rendered on `/metrics`, but no packet is dropped and no ban escalates.
`engine.kitsune.*`	grace periods, threshold percentile, learning_rate (passed to AfterImage)
`engine.lucid.model_path`	set a path to enable LUCID; empty string disables it
`api.auth_token`	set to enable authentication; empty string disables auth (development mode)
`api.host` / `api.port`	what `python app.py` binds to
`interception.safe_ips`	add IPs that must never be blocked (loopback included by default)
`interception.intercept_icmp` / `engine.rule_engine.allowed_icmp_types`	ICMP policy (see the protocol-filtering note above)
`storage.*`	event database path, row cap and retention window (see "Alerts, audit and metrics")
`logging.*`	level, rotating file target and syslog forwarding

On engine/start the API/CLI read the interception, engine, blocking, and api blocks from this file and apply them at runtime. If the file is missing or malformed, each loader falls back to safe defaults (loopback protection included) rather than crashing.

4. Run the API

python app.py
# /docs, /redoc and the OpenAPI schema are all disabled in production.

5. CLI

python cli.py start                  # start live interception (Linux, root)
python cli.py stop                   # stop live interception (via API)
python cli.py status                 # engine status
python cli.py block 1.2.3.4          # block an IP (POST /api/v1/rules/blacklist)
python cli.py unblock 1.2.3.4        # unblock an IP (DELETE /api/v1/rules/blacklist/{ip})
python cli.py whitelist --ip 10.0.0.0/8   # whitelist a subnet (rejects /0 default routes)
python cli.py unwhitelist --ip 10.0.0.0/8 # remove from whitelist
python cli.py rules                  # list blacklist/whitelist entries
python cli.py reload                 # apply edited rules.json / config to the running engine
python cli.py alerts --last 20       # stored alerts, newest first (via API)
python cli.py alerts --source-ip 203.0.113.7 --action block
python cli.py alerts --since 2026-09-19T00:00:00 --format csv > alerts.csv
python cli.py audit --last 20        # who changed which rule, and the outcome
python cli.py audit --result 401     # rejected management attempts
python cli.py test --pcap sample.pcap  # offline detection test (no root needed)

API Reference

Method	Endpoint	Description
`GET`	`/health`	Health check
`GET`	`/api/v1/status`	Engine status, detectors, blocked IPs (incl. kernel-level), detection-loop health
`GET`	`/api/v1/stats/overview`	Traffic and blocking statistics
`GET`	`/api/v1/alerts`	Stored alerts: `limit`, `offset`, `source_ip`, `action`, `since`, `until`, `format=json\|csv\|jsonl`
`GET`	`/api/v1/audit`	Management audit trail: `limit`, `offset`, `actor`, `result`, `since`, `until`, `format`
`GET`	`/api/v1/rules`	Current blacklist and whitelist
`GET`	`/api/v1/blocks`	Live escalation state (observing / temp-banned / perm-banned)
`POST`	`/api/v1/rules/blacklist`	Add IP to blacklist
`DELETE`	`/api/v1/rules/blacklist/{ip}`	Remove IP from blacklist
`POST`	`/api/v1/rules/whitelist`	Add IP/CIDR to whitelist
`DELETE`	`/api/v1/rules/whitelist/{ip}`	Remove IP from whitelist
`POST`	`/api/v1/rules/reload`	Re-read rules.json and the live engine knobs from config.yaml
`GET`	`/api/v1/signatures`	Declared signature rules and their hit counts
`POST`	`/api/v1/signatures`	Add or edit a signature (re-posting an id replaces it)
`DELETE`	`/api/v1/signatures/{id}`	Remove a signature
`POST`	`/api/v1/engine/start`	Start live interception (Linux, root)
`POST`	`/api/v1/engine/stop`	Stop interception and clean up iptables
`GET`	`/metrics`	Prometheus text exposition (token-guarded like `/api/v1/*`)

The two DELETE routes take {ip:path}, so CIDR entries are removable too (/api/v1/rules/blacklist/10.0.0.0%2F8 or the unencoded form).

Authentication: when api.auth_token is set in config.yaml (or the NIPS_API_TOKEN env var is present), every /api/v1/* call must carry the header X-API-Token: <token>. An empty token disables authentication (development only — the server logs a warning at startup). /health stays open (liveness probes).

Alerts, audit and metrics

Detection events and every management action are written to SQLite (WAL) at storage.events_db (default data/events.db) — they survive a restart, and retention_days plus max_rows bound the file.

Alerts (/api/v1/alerts, cli.py alerts) — one row per BLOCK verdict, plus whitelist changes. format=csv|jsonl exports for a SIEM; each request is capped at 1000 rows.
Audit (/api/v1/audit, cli.py audit) — one row per rule/engine change and per refused attempt (401/422), with the peer address, method, path, target and outcome. The API has one shared token, so actor identifies the host, not a named user.
Metrics (/metrics) — packets processed/blocked, detector state, blacklist sizes, temp-ban counters, event-store health.

The detection path never waits on the disk: record_alert only enqueues into a bounded buffer and a background thread writes in batches. If the buffer overflows or a batch fails, the counters in nips_alert_events_dropped_total / nips_event_store_write_errors_total rise and /api/v1/status reports them under event_store — an incomplete trail is visible, not silent. When the database is unusable, reads fall back to the most recent 500 in-memory events and event_store.degraded is true.

logging.file adds a rotating log file and logging.syslog_address forwards to syslog (platform socket, or host:port over UDP); an unreachable target is reported and skipped rather than blocking startup, and a sink that starts failing later (daemon restarted, disk full) reports once per 1/10/100/1000 lost records instead of printing a traceback per record.

Signature rules

A blacklist answers "is this source bad"; the rate limiter answers "is anyone sending too fast". Neither answers "drop TCP/22 traffic from 203.0.113.0/24 once it exceeds 50 sessions a minute" — banning the subnet would silence the legitimate users behind it, and the global limiter cannot be scoped to one source and one port. A signature is that conjunction:

# watch first: counts matches, drops nothing
python cli.py signature add --id ssh-brute --src 203.0.113.0/24 \
    --protocol tcp --dport 22 --min-packets 50 --window 60 --action log
python cli.py signature list                    # rules with their hit counts
python cli.py signature add --id ssh-brute --src 203.0.113.0/24 \
    --protocol tcp --dport 22 --min-packets 50 --window 60    # now enforce
python cli.py signature delete ssh-brute

field	meaning
`src` / `dst`	IP or CIDR; a `/0` default route is refused
`protocol`	`tcp`, `udp`, `icmp`, or a number
`dport` / `sport`	0-65535. Giving a port without a protocol defaults to TCP — those byte offsets are an echo id/sequence in ICMP, so matching them there would be nonsense
`tcp_flags`	exact match on the 6-bit field (`0x02` = SYN); only valid with TCP
`min_packets` + `window_seconds`	fire only after N matches from one source inside the window
`action`	`block` drops that packet inline; `log` counts it and lets it continue to the ML stage

A rule with no matchers is refused: with action=block a single stray call would drop all traffic.
Evaluation is declaration order, first match wins, after the blacklist (so a listed source is reported as listed) and before the global rate limit (so a scoped rule is not masked by it).
A signature BLOCK is enforced inline per packet, like any other rule-engine verdict: it does not count strikes and installs no kernel DROP. That is deliberate for rate-conditioned rules — a persistent kernel rule would outlive the condition that triggered it. To ban a source outright, use the blacklist.
Per-source hit counters are LRU-bounded (10k sources per rule); uncapped, a spoofed flood would grow the table one entry per packet and turn a detection feature into memory exhaustion.
Signatures persist in rules.json under "signatures" and follow the hot-reload rules above: a hand edit applies within 30 s (or via cli.py reload), and one malformed entry rejects the whole file rather than applying part of it.
API: GET/POST /api/v1/signatures and DELETE /api/v1/signatures/{id}; validation lives in the engine, so the API, the file and hot reload refuse exactly the same specs. Upserts and removals are audited.

Hot reload

rules.json and config/config.yaml are watched by mtime, so a running engine picks up edits within 30 s; POST /api/v1/rules/reload (or cli.py reload) applies them immediately. Rule and rate-limit edits need no restart — and restarting would be costly, since Kitsune re-trains from zero. What does need one is named below, and named by the reload itself.

rules.json is applied with replace semantics, so deleting an entry really stops enforcing it (startup uses merge, which only adds).
engine.rule_engine.rate_limit.*, allowed_protocols and allowed_icmp_types take effect on the next packet.
A malformed file is rejected wholesale: the live rules stay exactly as they were, /api/v1/status raises reload.failures, and the attempt is audited as reload_failed.
Entries the kernel would refuse anyway (loopback / safe_ips) are swept as at startup and reported as dropped_unenforceable.
Kitsune's fm_grace_period, ad_grace_period, threshold_percentile, learning_rate and max_autoencoder_size are not re-applied — they describe how the detector was trained and shaped, so they need a restart. So do engine.ml.enabled and engine.ml.detectors: the chain is imported and constructed before the first packet, and with ML off nothing is imported at all, so applying the switch live would report ML enabled over a chain that was never mounted. The reload summary names every one of them, and verify_engine_module.py RL7 checks that list against the config schema — a knob cannot be added and silently ignored.

Layout

app.py                         # FastAPI application entry point
cli.py                         # CLI management tool
config/
  config.yaml                  # Engine/interception configuration
networksecurity/
  engine/                      # Detection engine
    detector.py                # BaseDetector interface + PacketInfo
    assembly.py                # Mounts the detectors config asks for (engine.ml)
    threshold_detector.py      # Complete worked example of the contract
    verdict.py                 # Verdict, Action, ThreatLevel types
    pipeline.py                # DetectionPipeline (multi-stage chain)
    rule_engine.py             # IP blacklist/whitelist, rate limiting
    signature_engine.py        # Declarative matchers: CIDR/protocol/ports/flags + rate threshold
    block_policy.py            # BLOCK escalation: strikes → temp ban → permanent ban
    kitsune/                   # Kitsune anomaly detector (NDSS'18)
      afterimage.py            # 90-dim incremental statistics
      kitnet.py                # Autoencoder ensemble
      kitsune.py               # Orchestrator
      detector_adapter.py      # BaseDetector adapter
    lucid/                     # LUCID DDoS detector (IEEE TNSM 2020, optional)
      cnn.py                   # 1D CNN model
      dataset_parser.py        # Flow buffer and feature extraction
      detector.py              # Orchestrator
      detector_adapter.py      # BaseDetector adapter
  interception/                # Linux traffic interception
    nfqueue_handler.py         # NFQUEUE binding and packet capture
    packet_parser.py           # Raw IPv4/IPv6 packet parser
    iptables.py                # iptables rule management
    interceptor.py             # Live interceptor (nfqueue + pipeline)
  features/                    # Feature extraction
    flow_extractor.py          # Per-flow statistical features
    feature_registry.py        # Feature set registry
  data/                        # Data loading
    dataset_loader.py          # NSL-KDD, CICIDS2017, UNSW-NB15 (CSV / Parquet)
    pcap_loader.py             # PCAP file reader
  observability/               # Durable events and metrics (storage only)
    alert_store.py             # SQLite alerts + audit trail, batched non-blocking writer
    metrics.py                 # Prometheus text exposition
    log_setup.py               # level / rotating file / syslog routing
  utils/                       # Shared helpers
    config.py                  # config.yaml loading (engine / api / blocking / storage / logging)
    validation.py              # IP/CIDR validation and blacklist refusal rules
    reload.py                  # ReloadProbe: mtime watch over rules.json + config.yaml
scripts/                       # Benchmarks, evaluation & regression checks
  benchmark.py                 # Throughput + rule-engine accuracy
  benchmark_nslkdd.py          # NSL-KDD detection benchmark
  attack_simulation.py         # Large-scale attack simulation
  build_unsw_pcap.py           # Rebuild real-traffic pcaps from the bundled UNSW-NB15 flows
  capture_labels.py            # Label sidecar format: truth next to the capture, not in it
  capture_truth.py             # Shared truth-chain checks: independence, alignment, evaluator refusal
  live_nfqueue_topology.sh     # Rootless netns topology for the live NFQUEUE CI job
  train_lucid.py               # Train the LUCID CNN and write engine.lucid.model_path
  evaluate_pcap.py             # End-to-end pcap evaluation (per attack category)
  verify_*.py                  # Module regression checks, incl. the CI FPR guard

Live Interception (Linux only)

# 1. Install nfqueue library
pip install NetfilterQueue

# 2. Run with root privileges
sudo python -c "
from networksecurity.interception import Interceptor
from networksecurity.engine import DetectionPipeline
from networksecurity.engine.kitsune.detector_adapter import KitsuneDetector

pipeline = DetectionPipeline()
pipeline.add_detector(KitsuneDetector())

interceptor = Interceptor(pipeline)
interceptor.start()  # Blocks. Ctrl+C to stop.
"

The interceptor:

Installs iptables rules to redirect traffic into NFQUEUE
Leaves loopback traffic untouched — everything arriving on lo is ACCEPTed before the NFQUEUE rules, and loopback sources (127.0.0.0/8, ::1) are never eligible for a permanent block (host-local traffic cannot be an attacker; blocking the DNS stub 127.0.0.53 would silently break host DNS)
Leaves SSH (port 22) untouched
Redirects only TCP and UDP into NFQUEUE unless interception.intercept_icmp is on; with it on, allowed_icmp_types decides which ICMP types the engine then accepts (ICMPv6 link maintenance passes regardless)
Enforces BLOCK verdicts through an escalation policy (blocking: in config.yaml) that applies only to ML-detector BLOCKs. Rule-engine verdicts (blacklist hit, rate limit, protocol filter) are deterministic and already enforced inline on every packet, so they never count strikes and cannot escalate — this also guarantees an operator's blacklist entry can never be modified by the ban lifecycle.
- A single ML BLOCK only inline-drops that packet and counts a strike against the source.
- Crossing strikes_threshold inside the rolling window triggers a temp ban — kernel DROP plus a rule-engine blacklist mirror with a TTL, lifted automatically on expiry (only the mirror is removed; an operator's own entry is never touched).
- Repeated temp bans escalate to a permanent ban, which is mirrored into rules.json; on the next start it is loaded back into the rule engine and enforced per-packet in userspace — the kernel DROP itself is not reinstalled.
Removes all of its iptables rules on shutdown

Dual-stack, with an honest fallback. IPv4 and IPv6 TCP/UDP are redirected into NFQUEUE, parsed (including the IPv6 extension-header chain) and blocked through ip6tables. If ip6tables is unavailable the interceptor starts anyway, refuses every IPv6 block instead of pretending, and reports the gap: ipv6_intercepted: false in /api/v1/status and nips_ipv6_intercepted 0 in /metrics. Check that gauge on any dual-stack host — a silently uninspected second address family is exactly the failure an operator would not notice until an incident.

Three parsing limits remain, all fail-closed and all counted in nfqueue_parse_failed: non-first IP fragments carry no transport header, so they are dropped rather than misread; AH/ESP packets cannot be walked without authenticating them, so they are dropped rather than parsed as if the ciphertext were a TCP header; and an IPv6 extension chain deeper than six hops is treated as crafted. Interceptor reads safe_ips and nfqueue_num from config.yaml; a missing or unparseable file falls back to safe defaults (loopback protection included) rather than starting unprotected.

A detection timeout drops only the in-flight packet (fail-closed); it never commits a permanent block, so a slow verdict cannot ban a legitimate IP.

Training dataset preparation

DatasetLoader (networksecurity/data/dataset_loader.py) loads NSL-KDD, CICIDS2017, and UNSW-NB15 as labeled CSV or Parquet for supervised training of LUCID/Kitsune. It assumes each file is already a header-bearing CSV with the dataset's standard column names (a .parquet suffix is read as Parquet instead) — it does not detect or convert headers, nor does it handle the raw headerless NSL-KDD .txt distribution. Preparing the files is the user's responsibility before calling DatasetLoader.

Required layout per dataset:

Dataset	Expects	Notes
NSL-KDD	CSV with header, 43 columns: 41 features in the standard NSL-KDD order, then `difficulty`, then `label`	The official `KDDTrain+.txt` / `KDDTest+.txt` are headerless — add the 41 standard feature names + `difficulty` + `label` before loading. Binary label: `normal`/`normal.` → 0 (benign), anything else → 1 (attack).
UNSW-NB15	CSV with header, binary `label` column (0/1), plus `id` and `attack_cat` metadata	`attack_cat` is dropped automatically (it leaks the label).
CICIDS2017	CSV with header, `Label` column (capital L), plus `Flow ID` / `Timestamp` / `Source IP` / `Destination IP`	Those four metadata columns are dropped automatically. `BENIGN` → 0, everything else → 1.

Categorical columns are one-hot encoded (get_dummies, drop_first), missing values filled with 0, and the result is returned as float32. For aligned train/test encodings use train_test_split(), which fits the encoding on the training split and reindexes the test split to the same columns.

Training LUCID

LUCID is the one detector that is supervised, so it needs a model file before engine.lucid.model_path can point at anything:

# 1. check what your labels imply — no TensorFlow needed, writes nothing
python scripts/train_lucid.py --pcap capture.pcap \
    --attackers 203.0.113.0/24 --victims 10.0.0.1 --inspect

# 2. fit and save (pip install -e ".[lucid]" first)
python scripts/train_lucid.py --pcap capture.pcap \
    --attackers attackers.txt --victims 10.0.0.1 --out models/lucid_cnn.h5

--inspect prints how many complete windows the capture yields, the attack/benign balance and how many flows expired before filling a window — the numbers that tell you whether the addresses you named label anything at all before you spend a training run. Notes:

Attacker/victim entries may be single addresses or CIDRs. A malformed one is refused rather than skipped: a typo in an attacker list silently unlabels exactly the traffic the model was meant to learn.
Under LUCID's convention a window is an attack if a majority of its packets involve an attacker or a victim address on either side — so listing a victim marks every flow toward it as attack traffic. Name attackers only if your capture contains ordinary traffic to that host.
Training reuses the online feature path, so a model cannot be fitted on one representation and scored against another.
One-sided labels, zero complete windows, or fewer than four windows are refused with the reason printed, instead of producing a model that predicts one class and looks healthy.

Measured results and limits

These are numbers this repository actually produces, with the command that reproduces each one. They are not stable — Kitsune's autoencoders are initialised from an unseeded RNG (the evaluator takes --seed to pin one) — so treat them as ranges.

what	result	reproduce with
Rule engine alone vs. a single-source SYN flood	80.0% of 500 SYN packets at 1 ms spacing blocked (the first 100 fit inside the shipped 100 connections/s budget), and 0 of 2 000 normal packets blocked — 1.80% of the same 2 000 once the anomaly detector is in the path	`python scripts/verify_fpr_regression.py`
The same attack spread over 2 000 spoofed sources at 10 000 pkt/s	0.0% (17 of 60 000) — per-source rate never crosses the limit, and per-host modelling cannot see it	`python scripts/attack_simulation.py --full`
Volumetric floods on synthetic traffic (UDP, ICMP)	100% (ICMP by protocol rule, UDP by the anomaly stage)	same
Whole synthetic run, every phase counted	58.5% attack packets detected, 26.72% of normal packets blocked — 36 239 of those 36 408 blocks land in the last phase, where every normal packet is flagged once four-tenths of the stream is attack traffic	same
Real UNSW-NB15 reconstruction, Kitsune after training	false-positive rate 0.4–1.4% (median 0.9%), detection rate 0.3–0.8% over twelve autoencoder seeds; one draw in twelve still lands at 20.9% / 13.6% (`--seed 2`)	`python scripts/verify_real_capture_quality.py`, or per draw: `python scripts/evaluate_pcap.py --seed N`
Real traffic off this host: 107,115 inbound packets, several minutes, no attack labels	rule engine blocked 1.5% of ordinary traffic; Kitsune blocked 98.4% of it before the threshold was calibrated on the frozen regime, 0 of 47,115 after	`python scripts/evaluate_pcap.py --pcap cap.pcap --no-labels --buckets 10` (and `--calibration-packets 0` for the old operating point)
Offline pipeline throughput	~660–870 pkt/s, single process	either of the above

The bundled "real capture" benchmark used to leak its own labels; this is what it says now. scripts/build_unsw_pcap.py drew attack flows' source addresses from one documented block and normal flows' from another, and scripts/evaluate_pcap.py read the label back out of those addresses — the answer key sat on the wire, so any source-address rule scored 100% for free and the "detection rate" measured the reconstruction rather than the detector. Addresses are now drawn independently of the label; the labels live in a sidecar (unsw_reconstructed.labels.json, written by scripts/capture_labels.py) that the evaluator takes as an explicit input, and a capture without one is refused instead of scored (--no-labels reports verdicts and block reasons only). scripts/capture_truth.py holds that line, and scripts/verify_capture_truth.py runs it on every pull request: it permutes the labels across the address partition read back out of the capture to check the source address cannot predict them (with a self-test on a deliberately leaky partition — a check that cannot fail is not a check), checks the retired 175.45.176.*-means-attack rule no longer separates the classes, joins the labelled flows to the capture (same flows, same endpoints, same packet counts), and runs the evaluator four times — missing label file, --no-labels, a sidecar describing none of the capture, and a sidecar carrying attack flows only — each of which must end without rates rather than with the numbers an empty or one-class labelled set produces: a window with five attack packets and no normal ones used to print false positive rate 0.0% over an empty denominator, which no reader can tell from a clean run. verify_real_capture_quality.py imports the same checks and refuses to spend half an hour measuring on a capture whose truth it cannot trust.

The reconstruction flattered the detector; traffic off a real host said otherwise. Feeding several minutes of this machine's own inbound traffic (107,115 packets, no labels, so everything blocked is a false positive by definition) through the shipped configuration, the rule engine blocked 1.5% of it — the same plane that blocked 0 of 2 000 synthetic normal packets, whose generator never puts a real host's per-source load on the wire. Kitsune blocked 98.4%. --buckets says why, and the answer was not drift: the flagged fraction was already 89% in the first bucket after training, so the operating point was wrong from the first packet it judged. The threshold was the percentile of scores taken while the output layer's normalisation was still being updated, applied to scores taken after it froze — two different scales. A stationary synthetic stream does not reproduce that (the legacy threshold flagged 0.0% of it), so this is a finding about real traffic's variety, and the fix is in the detector: calibrate the threshold on the first packets scored after the freeze. On the same capture that took the flagged count from 51,285/52,115 to 0/47,115, while K11 keeps asserting a genuine distribution shift is still caught. What this capture cannot say is anything about detection rate — no attacks in it, and --no-labels refuses to print rates for that reason. Whether Kitsune is worth arming is still unmeasured on realistic traffic, which is what enforce: false exists for.

Why the numbers moved, and why they are the shipped configuration. KitNET fits its feature map on the first fm_grace packets, then freezes its output normalisation and its threshold from the next ad_grace, so those two windows decide everything after them. This capture reconstructs 1 680 flow records with concentrated packet mass (the largest flow carries 8 398 of 82 523 packets; the top 1% carry 35%), and the evaluator used to shorten the grace periods to 2 000 + 30 000 to fit training and a detection window into one capture. At 2 000 packets the fitting window is simply whatever arrived first — 43 flows, one of them a third of the window — so redrawing the same flows re-ordered the stream and moved the fitted model to another regime: 9 autoencoder groups instead of 10, threshold 0.65 instead of 1.18, and 52% of packets blocked instead of 1.3%, with identical flow composition and no code change. That is a property of the draw, not of the detector. The evaluator now trains at the values config/config.yaml ships (5 000 + 50 000, where the fitting window covers 112 flows and the largest contributes 13%), which keeps both draws out of that regime. What is left is the initialisation: KitNET draws its autoencoder weights from the global RNG, and over twelve seeds this capture's false-positive rate ran 2.1–10.5% with detection at 0.9–2.9% when the threshold came from the training pass. Calibrating it on the frozen regime pulls eleven of the twelve draws to 0.4–1.4% false positives with 0.3–0.8% detection (median 0.9%); the twelfth, --seed 2, still lands at 20.9% and 13.6%. So per-initialisation instability is narrowed, not cured — which is why verify_real_capture_quality.py scores three fixed draws and bounds their median rather than bounding a single run: a bound on one unseeded run sits inside that spread and would go red on roughly one run in five with nothing changed. Reproduce any draw with python scripts/evaluate_pcap.py --seed N. The reconstruction's numbers stay modest for a further reason: it is rebuilt from flow records — real addresses, sizes and timing, but no real per-packet structure for the model to separate on. The same code on traffic off a real host flagged 98.4% of it before that calibration existed, which is the row above.

Why a flood's detection rate depends on how it is spread: the synthetic rows are different attacks with the same name. A concentrated flood trips a per-source connection limit; a distributed one is invisible to it by construction, and Kitsune scores per host, so spoofed sources each look like a well-behaved client. This is the boundary of the approach, not a tuning gap — the same reason the reconstruction's detection rate is low.

Where the synthetic run's 26.72% false-positive rate comes from: not evenly. Of the 36 408 blocked normal packets, 169 are in the 100 000-packet training phase and 36 239 are in the final mixed phase — every normal packet in it, once four-tenths of that phase's traffic is attack. A per-host model learns what a host looks like, and attack traffic is exactly what changes that, so the detector's own notion of normal is poisoned by the traffic it is meant to catch. That is a property of the design rather than of the threshold, and it is a second reason the learning detectors ship disabled.

Calibrate blocking: before trusting either number. Strikes are counted per BLOCK verdict, i.e. per packet, so at the 0.4–1.4% packet-level false-positive rate eleven of twelve draws measure on this capture — and at the 98.4% a real capture measured before the threshold was calibrated — a legitimate source sending a few hundred packets inside strikes_window reaches the shipped strikes_threshold: 5 and takes a 10-minute kernel ban; repeated cycles persist a permanent one. Measure on your own traffic (cli.py test --pcap, watching nips_alert_events_written_total) and set the threshold to a multiple of what your traffic produces.

Reproducing this on your own hardware

scripts/benchmark.py — trains Kitsune on synthesized normal traffic, then reports rule-engine accuracy, training/detection throughput, and attack detection rate.
scripts/benchmark_nslkdd.py — downloads NSL-KDD, maps flow records to synthetic packets, trains Kitsune on normal flows, and reports precision/recall/FPR.

Why detection on NSL-KDD is weak here: NSL-KDD records are flow-level summaries, not packet captures. Mapping each flow to a few packets throws away the timing and burst patterns that Kitsune learns from. Volumetric attacks (DoS, probe) survive the mapping better than content attacks (R2L, U2R), which look like ordinary TCP at the packet level. Treat the per-attack numbers as a statement of that limitation, not a measured accuracy claim.

The rule engine itself is exact: blacklist/whitelist, protocol filtering, and rate limiting are deterministic and always applied before the ML stage. Rate limit counts only TCP SYN (ACK clear) and UDP datagrams; established TCP sessions (ACK/data/FIN) do not consume the budget.

Offline testing with your own traffic

The detection pipeline can be exercised without root or iptables, which is the practical way to measure what it would do on the traffic you actually carry:

Real pcap (the one that tells you something): capture packets and run them through the pipeline offline.
```
# capture several minutes of live traffic (requires root for the sniff)
sudo python -c "from scapy.all import sniff, wrpcap; wrpcap('cap.pcap', sniff(iface='en0', timeout=300))"
# offline detection — no root needed
python cli.py test --pcap cap.pcap
# with ground truth absent, the evaluator reports verdicts and never rates
python scripts/evaluate_pcap.py --pcap cap.pcap --no-labels --buckets 10
```
Two things about your own traffic were measured, not assumed, and both will bite otherwise. A sniff on a host sees both directions, so every outbound connection counts against that source's rate limit: on one capture the host's own proxy tunnel (one source, 1,490 UDP packets/s) made the rule engine block 91% of packets. Filter to traffic destined for this host before drawing conclusions, or expect the rate limiter to be the only thing you measure. And Kitsune needs 55,000 warm-up packets plus a 5,000-packet threshold calibration before it judges anything, so a 30-second capture exercises the rule engine alone; --buckets reports the flagged fraction per time slice, which is how a mis-set operating point (high from the first bucket) is told apart from traffic drift (climbing across buckets). This surfaces the real false-positive rate (e.g. legitimate ICMP being blocked by the protocol filter — in this offline path ICMP does reach the engine, unlike live interception with intercept_icmp: false), which the synthetic simulation does not.
The synthetic simulation and the bundled reconstruction are covered by the table above; their limits are stated there rather than repeated here.

Fail-closed behavior

With learning detection enabled, a packet the rule engine does not decide must be scored by something. If every mounted detector that could score has raised or tripped its circuit breaker, the pipeline raises DetectionUnavailable and that packet is dropped rather than waved through: a silent outage is safer than allowing unknown traffic while the detector you paid for is dead. Turning engine.ml.enabled off is the opposite case — a decision, so undecided traffic is allowed, and it never triggers this path. A detector mounted without its model (ready: false) is treated as not deployed: it neither decides nor counts as coverage.

The status API exposes detection_unavailable_drops, broken_detectors, ml_enabled, ml_consulted, ml_idle and detector_status so an operator can tell these states apart instead of guessing from one boolean — detector_status["KitsuneDetector"]["trained"] is where "mounted and counting" differs from "producing verdicts yet", and the threshold_source beside it says which ruler the operating point came from: calibration (scored after the normalisation froze), training (the legacy pass, whose scale does not apply to detection), or pending while neither has been measured.

Deployment

Docker

# Build and start
bash deploy.sh build
bash deploy.sh start

# Verify tests pass (runs all verify_* scripts in container)
bash deploy.sh test

# View logs
bash deploy.sh logs

# Stop
bash deploy.sh stop

Notes:

deploy.sh runs 8 verification scripts (verify_engine_module, verify_interception_module, verify_block_lifecycle, verify_live_exposed_bugs, verify_fpr_regression, verify_features_module, verify_data_module, verify_management_plane) instead of pytest.
The container runs as non-root user nips for security. Bind-mount rules.json from the host — it must exist before docker compose up or you'll get an IsADirectoryError.
The container is the management plane only. NFQUEUE and iptables need the host network stack, so the interceptor runs on the host (cli.py start or the systemd unit). Deploying only the container gives you an API that reports and edits rules but drops no traffic — a dashboard, not a prevention system.
rules.json contains only persistent blacklist entries (operator-added + escalated permanent bans). Temp-ban mirrors live in the ephemeral tier and are never written to disk.

Linux host

# Install dependencies
pip install -r requirements.txt

# Run API server
python app.py

# Or use CLI directly (requires root for live interception)
sudo python cli.py start

systemd

Two units under deploy/systemd/, meant to be installed as nips-api.service and nips-interceptor.service after editing the /opt/nips paths (use systemctl edit drop-ins rather than editing the shipped file):

sudo cp deploy/systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now nips-api nips-interceptor

They are deliberately separate: only the interceptor holds the privileges that rewrite the firewall, and the web-facing process does not. The interceptor needs real root (cli.py start checks geteuid()), so capabilities alone will not satisfy it. Stop behaviour matters — the SIGTERM handler removes the NFQUEUE redirect, so TimeoutStopSec is generous; killing it faster leaves a kernel redirect with nothing draining the queue, which stalls traffic until the nfqueue timeout.

Remote administration

The API listens on api.host/api.port and speaks plain HTTP. It has one shared token and no per-user identity, so it is designed to sit behind a TLS-terminating reverse proxy that also enforces source addresses or client certificates — not to be published directly. The shipped docker-compose binds it to 127.0.0.1:8000 for the same reason.

The CLI targets http://127.0.0.1:8000 by default; point it elsewhere per invocation or by environment:

python cli.py --url https://nips.internal:8443 --token "$NIPS_API_TOKEN" status
NIPS_API_URL=http://10.0.0.5:8000 python cli.py alerts --last 20

Documentation

ARCHITECTURE.md — layer design, data flow, module boundaries, red lines
CONTRIBUTING.md — PR workflow, pre-submission checklist, what we reject
CODE_STYLE.md — coding conventions, import rules, system call validation
SECURITY.md — vulnerability reporting, deployment best practices
CHANGELOG.md — release history
API endpoints: see the "Run the API" section above (/docs, /redoc and OpenAPI are disabled in production)

Contact

Author: 梓铭
Email: [email protected]
Issues: GitHub Issues

License

MIT — see LICENSE

zimingttkx/Network-Security-Based-On-ML

Gemessener Sternzuwachs

Über uns

README.md