Anton

This project is an open-source tool that extracts and organizes data from the TON blockchain, efficiently storing it in PostgreSQL and ClickHouse databases.

If you have any questions, you can ask them in the Telegram group.

Overview

Before you start, take a look at the official docs.

Consider an arbitrary contract. It has a state that is updated with any transaction on the contract's account. Each state has the contract code and data. The contract data can be complex, but developers typically provide get-methods in the contract, which can be executed to retrieve the necessary data. The TON has standard contracts (such as TEP-62, TEP-74), and they have predefined get-method names. Therefore, you can attempt to match accounts found in the network to these standards by checking the presence of the get-methods. Contract standards also specify TL-B constructor tags (or operation ids) for each acceptable message to contract, defined as the first 32 bits of the parsed message payload cell. So you if you know standard of a given contract, you can determine the type of message to it (for example, NFT item transfer) by parsing the first 32 bits of message body.

Anton allows you to define the contract interface in just one JSON schema. Format of every schema is described in detail in abi/README.md. Every schema comprises contract get-methods, as well as incoming and outgoing message schemas for the contract. Once contract interfaces are defined and stored in the database, Anton begins scanning new blocks on the network. The tool stores every account state, transaction, and message in the database. For get-methods without arguments in the contract interface, Anton emulates these methods and saves the returned values to the database. When a message is sent to a known contract interface, Anton attempts to match the message to a known schema by comparing the parsed operation ID. If the message is successfully parsed using the identified schema, Anton also stores the parsed data.

To explore contract interfaces known to this project, visit the abi/known directory. This will provide you with an understanding of the various contract interfaces already supported and serve as examples for adding your own.

Currently, Anton offers a REST API for retrieving filtered and aggregated data from the databases. To see example queries, refer to the API.md file.

To explore how Anton stores data, visit the migrations' directory.

Project structure

Folder	Description
abi	get-methods and tlb cell parsing
abi/known	contract interfaces known to this project
api/http	JSON API Swagger documentation
docs	only API query examples for now
config	custom postgresql configuration
migrations	database migrations
internal	database repositories and services implementation

Internal directory structure

Folder	Description
core	contains project domain
core/rndm	generates random domain structures
core/filter	describes filters
core/aggregate	describes aggregation metrics
core/repository	implements database repositories with filters and aggregation
app	contains all services interfaces and their configs
app/parser	service determines contract interfaces, parse contract data and message payloads
app/fetcher	service concurrently fetches data from blockchain
app/indexer	service scans blocks and save parsed data to databases
app/rescan	service parses data by updated contract description
app/query	service aggregates database repositories
api/http	implements the REST API

Starting it up

Cloning repository

git clone https://github.com/tonindexer/anton
cd anton

Running tests

Run tests on abi package:

go test -p 1 $(go list ./... | grep /abi) -covermode=count

To run the tests you might need to provide a path to the emulator library. For example:

CGO_LDFLAGS="-L /Users/user/go/src/github.com/tonkeeper/tongo/lib/darwin/ -Wl,-rpath,/Users/user/go/src/github.com/tonkeeper/tongo/lib/darwin/ -l emulator" \
  go test -p 1 $(go list ./... | grep /abi) -covermode=count

Run repositories tests:

# start databases up
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d postgres clickhouse

go test -p 1 $(go list ./... | grep /internal/core) -covermode=count

Running linter

Firstly, install golangci-lint.

golangci-lint run

Configuration

Installation requires some environment variables.

cp .env.example .env
nano .env

Name	Description	Default	Example
DB_NAME	Database name		idx
DB_USERNAME	Database username		user
DB_PASSWORD	Database password		pass
DB_CH_URL	Clickhouse URL to connect to		clickhouse://clickhouse:9000/db_name?sslmode=disable
DB_PG_URL	PostgreSQL URL to connect to		postgres://username:password@postgres:5432/db_name?sslmode=disable
FROM_BLOCK	Master chain seq_no to start from	1	23532000
WORKERS	Number of indexer workers	4	8
RESCAN_WORKERS	Number of rescan workers	4	8
RESCAN_SELECT_LIMIT	Number of rows to fetch for rescan	3000	1000
LITESERVERS	Lite servers to connect to		135.181.177.59:53312 aF91CuUHuuOv9rm2W5+O/4h38M3sRm40DtSdRxQhmtQ=
DEBUG_LOGS	Debug logs enabled	false	true

Building

# building it locally
go build -o anton .

# build local docker container via docker cli
docker build -t anton:latest .
# or via compose
docker compose -f docker-compose.yml -f docker-compose.dev.yml build

# pull public images
docker compose pull

Running

We have several options for compose run via override files:

• base (docker-compose.yml) - allows to run services with near default configuration;
• dev (docker-compose.dev.yml) - allows to rebuild Anton image locally and exposes databases ports;
• prod (docker-compose.prod.yml) - allows to configure and backup databases, requires at least 128GB RAM.

You can combine it by your own. Also, there are optional profiles:

• migrate - runs optional migrations service.

Take a look at the following run examples:

# run base compose
docker compose up -d

# run dev compose (build docker image locally)
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d

# run prod compose
# WARNING: requires at least 128GB RAM
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

To run Anton, you need at least one defined contract interface. You can find some known interfaces in the abi/known directory. These can be added during the initial startup of the indexer:

# If you are running the indexer without Docker, 
# you need to specify the directory containing the known contracts:
anton indexer --contract-dir /root/anton/abi/known

# If you are running it using Docker, 
# there is no need to specify the contracts directory,
# as they are already included in the container:
docker compose up -d indexer

Database schema migration

# run migrations service on running compose
docker compose run migrations

Reading logs

docker compose logs -f

Taking a backup

# starting up databases and API service
docker compose                      \
    -f docker-compose.yml           \
    -f docker-compose.prod.yml      \
        up -d postgres clickhouse web

# stop indexer
docker compose stop indexer

# create backup directories
mkdir backups backups/pg backups/ch

# backing up postgres
docker compose exec postgres pg_dump -U user db_name | gzip > backups/pg/1.pg.backup.gz

# backing up clickhouse (available only with docker-compose.prod.yml)
## connect to the clickhouse
docker compose exec clickhouse clickhouse-client
## execute backup command
# :) BACKUP DATABASE default TO File('/backups/1/');

# execute migrations through API service
docker compose exec web anton migrate up

# start up indexer
docker compose                      \
    -f docker-compose.yml           \
    -f docker-compose.prod.yml      \
        up -d indexer

Using

Showing archive nodes from global config

docker run tonindexer/anton archive [--testnet]

Inserting contract interface

To add interfaces, you need to provide Anton with a contract description. It will select any interfaces not already present in the database, insert them, and initiate rescan tasks for messages and account states.

# add from stdin
cat abi/known/tep81_dns.json | docker compose exec -T web anton contract addInterfaces --stdin
# add from file
docker compose exec web anton contract addInterfaces "/var/anton/known/tep81_dns.json"

Deleting contract interface

To delete an interface, provide a contract description along with the specific contract name you wish to remove. Anton will then delete the contract interface and its associated operations from the database and initiate rescan tasks to remove all parsed data related to this interface from messages and account states.

docker compose exec rescan sh -c "anton contract deleteInterface -c nft_item /var/anton/known/*.json"

Updating contract interface

To update a contract interface, you need to provide both the contract description and the specific name of the contract you're updating. Anton will then compare the provided contract interface description against the existing interface in the database. If there are any differences, Anton initiates rescan tasks to reparse data and fix these changes. This process may involve adding, deleting, or updating get-methods and contract operations.

docker compose exec rescan sh -c "anton contract updateInterface -c telemint_nft_item /var/anton/known/telemint.json"

Adding address label

docker compose exec web anton label "EQDj5AA8mQvM5wJEQsFFFof79y3ZsuX6wowktWQFhz_Anton" "anton.tools"

# known tonscan labels
docker compose exec web anton label --tonscan