AWS Streaming Fraud Detection

Português | English

Educational open source Data Engineering project on AWS for fraud detection with synthetic data. The repository provides a local demo without AWS, a batch processing path with Amazon S3, Parquet, AWS Glue Data Catalog and Amazon Athena, plus an optional streaming processing mode with Amazon Kinesis Data Streams.

Purpose

This project shows how to model synthetic financial events, validate data contracts, apply explainable risk rules, generate partitioned Parquet files and prepare analytics with Amazon Athena. It is not a production anti-fraud system and does not claim PCI DSS, LGPD or equivalent compliance.

Local demo

python -m pip install -r requirements.txt
python -m fraud_detection demo --transactions 1000 --seed 42

The demo writes files to data/local/, which is ignored by Git, and prints a summary with analyzed transactions, suspicious transactions, JSON files, Parquet files and the main triggered reasons.

Architecture

flowchart LR
  A[Python generator] --> B[Event validation]
  B --> C[Local raw or Amazon S3 Raw]
  C --> D[Rule engine]
  D --> E[Risk score and enrichment]
  E --> F[Partitioned Parquet]
  F --> G[AWS Glue Data Catalog]
  G --> H[Amazon Athena]
  A -. optional .-> K[Amazon Kinesis Data Streams]
  K -. optional .-> L[AWS Lambda consumer]
  L -.-> C

Modes

• Local: does not require an AWS account, does not use credentials and generates deterministic data with --seed.
• AWS batch processing: the producer sends JSON to Amazon S3 Raw, processing generates Parquet in Amazon S3 Processed, AWS Glue Data Catalog catalogs the table and Amazon Athena queries the data.
• Optional streaming processing: Amazon Kinesis Data Streams can be enabled by Terraform with enable_streaming = true; it is disabled by default because it may create recurring costs.

Main capabilities

• Typed Python package in src/fraud_detection.
• Safe synthetic data without full names or full card numbers.
• Testable rules with identifier, weight, description and evidence.
• Risk score, risk level, triggered reasons and rule version.
• JSON and Parquet writing partitioned by year, month, day and hour.
• Modular Terraform with encrypted Amazon S3 buckets, public access blocking and optional streaming.
• Athena queries in analytics/queries.

AWS quick guide

cd terraform/environments/dev
cp terraform.tfvars.example terraform.tfvars
# edit unique_suffix before planning
terraform init -backend=false
terraform plan

Do not run terraform apply before reviewing costs, names and permissions. The project does not configure a remote backend automatically.

Documentation

• Architecture
• Local demo
• AWS deployment
• Batch processing
• Streaming processing
• Fraud rules
• Data contract
• Athena analytics
• Security
• Costs
• Learning path

Contributions

See CONTRIBUTING.md. There are beginner-friendly ideas for documentation, tests, analytics, Terraform, Python and observability.

Learning resources

This repository is free and independent. For a structured Portuguese course with additional lessons and projects, see the maintainer's course: Engenharia de Dados na AWS: do Zero aos Projetos Reais.