Testnetのクローラを稼働させる
Bitcoinのノードネットワークの状態を確認するのに便利なサイトが
このサイトはmainnetのネットワーク状況をクローリングして表示してくれるけど、Testnetのネットワークをふと確認してみたいと思い、クローラーをセットアップしてみる。クローラー自体はbitnodesで使われているPythonベースの実装があるので、それを利用する。
クローラーのプロビジョニング
プロビジョニングの方法は、Wikiに記載されている。 今回用意したサーバー
Amazon EC2 (t2.medium)
Ubuntu 18.04 LTS
Hostnameの設定
localhostのホスト名を追記
code: console
$ sudo vi /etc/hosts
127.0.0.1 localhost testnet-crawler.dev.chaintope.com
hostnamectl コマンドを実行して、新しいホスト名を指定。
code:console
$ sudo hostnamectl set-hostname testnet-crawler.dev.chaintope.com
必要なパッケージのインストール
code:console
$ sudo apt-get update
...
$ sudo apt-get -y install apt-transport-https build-essential dirmngr htop python-dev python-virtualenv sudo tcl tcpdump unzip
.bashrcの更新
.bashrcにredisのソケットとパスワードを設定。
code:.bashrc
export REDIS_SOCKET=/tmp/redis.sock
/etc/sysctl.confを設定
code:/etc/sysctl.conf
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.tcp_syncookies=1
net.ipv4.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.accept_source_route=0
net.ipv6.conf.all.accept_source_route=0
net.ipv4.conf.all.log_martians=1
net.core.rmem_default=33554432
net.core.wmem_default=33554432
net.core.rmem_max=33554432
net.core.wmem_max=33554432
net.core.optmem_max=33554432
net.ipv4.tcp_rmem=10240 87380 33554432
net.ipv4.tcp_wmem=10240 87380 33554432
net.ipv4.ip_local_port_range=2000 65500
net.core.netdev_max_backlog=100000
net.ipv4.tcp_max_syn_backlog=80000
net.ipv4.tcp_max_tw_buckets=2000000
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=5
net.ipv4.tcp_slow_start_after_idle=0
net.core.somaxconn=60000
fs.file-max=1000000
vm.swappiness=10
vm.min_free_kbytes=1048576
vm.overcommit_memory=1
/etc/security/limits.confを設定
code:security/limits.conf
* soft nofile 1000000
* hard nofile 1000000
THPの無効化
code:console
$ sudo vi /etc/systemd/system/disable-transparent-huge-pages.service
Description=disable-transparent-huge-pages
Type=oneshot
ExecStart=/bin/sh -c "echo "never" | tee /sys/kernel/mm/transparent_hugepage/enabled"
ExecStart=/bin/sh -c "echo "never" | tee /sys/kernel/mm/transparent_hugepage/defrag"
WantedBy=multi-user.target
$ sudo systemctl enable disable-transparent-huge-pages.service
$ sudo systemctl start disable-transparent-huge-pages.service
txqueuelenの設定
code:console
$ sudo vi /etc/systemd/system/set-txqueuelen.service
Description=set-txqueuelen
Type=oneshot
ExecStart=/sbin/ifconfig eth0 txqueuelen 5000
WantedBy=multi-user.target
$ sudo systemctl enable set-txqueuelen.service
$ sudo systemctl start set-txqueuelen.service
再起動
Torのインストール
code: console
$ sudo vi /etc/apt/sources.list.d/tor.list
$ sudo -s
# gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add -
# apt-get update
# apt install tor deb.torproject.org-keyring
..
# vi /etc/tor/torrc
Log notice file /var/log/tor/notices.log
# service tor reload
Redisのインストール
code:console
$ tar xvfz redis-5.0.3.tar.gz
$ cd redis-5.0.3
$ make
$ make test
$ sudo make install
$ cd utils
$ sudo bash install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running redis server
Please select the redis port for this instance: 6379 Selecting default: 6379
Selected default - /etc/redis/6379.conf
Selected default - /var/log/redis_6379.log
Selected default - /var/lib/redis/6379
Selected config:
Port : 6379
Config file : /etc/redis/6379.conf
Log file : /var/log/redis_6379.log
Data dir : /var/lib/redis/6379
Executable : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Success!
Starting Redis server...
Installation successful!
この段階で、/etc/init.d/redis_6379が生成されてる。
code:console
$ sudo update-rc.d redis_6379 defaults
$ sudo vi /etc/redis/6379.conf
unixsocket /tmp/redis.sock
unixsocketperm 777
maxclients 50000
maxmemory 34326183936
maxmemory-policy volatile-lru
notify-keyspace-events K$z
activerehashing no
client-output-buffer-limit normal 512mb 256mb 300
client-output-buffer-limit replica 512mb 256mb 300
client-output-buffer-limit pubsub 512mb 256mb 300
再起動
クローラーの起動
code:console
$ unzip master.zip
$ virtualenv ~/.virtualenvs/bitnodes
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/ubuntu/.virtualenvs/bitnodes/bin/python2
Also creating executable in /home/ubuntu/.virtualenvs/bitnodes/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
$ source ~/.virtualenvs/bitnodes/bin/activate
(bitnodes)$ cd bitnodes-master/
(bitnodes)$ pip install -r requirements.txt
(bitnodes)$ bash geoip/update.sh
... geoip/以下に地理情報がダウンロードされる。
Testnetの設定ファイルの作成
conf/以下の設定ファイルのテンプレートをコピーしてtestnet用の設定ファイルを作成する。
crawl.py
seederやaddrメッセージでノード情報を収集するPythonスクリプト
Testnet用の設定ファイルを追加
code:console
$ cp conf/crawl.conf.default conf/crawl.0b110907.conf
$ vi conf/crawl.0b110907.conf
# Logfile
logfile = log/crawl.0b110907.log
# Network magic number
magic_number = 0b110907
# Default/fallback port number
port = 18333
# Redis database number
db = 0
# List of DNS seeders to get a subset of reachable nodes
seeders =
testnet-seed.bitcoin.jonasschnelli.ch
seed.tbtc.petertodd.org
seed.testnet.bitcoin.sprovoost.nl
testnet-seed.bluematt.me
# Number of concurrent workers (greenlets)
workers = 700
# Print debug output
debug = False
# Public IP address for network interface
source_address = 0.0.0.0
# Protocol version to use for outgoing version message
protocol_version = 70015
# User agent (BIP 0014) to use for outgoing version message
# -----------------------------------------------------------------------------
# NOTE TO USERS
# Please consider changing the user agent before running an instance of this
# crawler. This is so that users will not confuse your crawler with another
# instance that is already running and generating data for the project.
# -----------------------------------------------------------------------------
user_agent = /bitnodes.earn.com:0.1/
# Services to use for outgoing network address message
services = 0
# Set to 1 to receive all txs
relay = 0
# Socket timeout
socket_timeout = 15
# Run cron tasks every given interval
cron_delay = 10
# Take full network snapshot at most at every given interval
snapshot_delay = 240
# Max. age for peering node to be included in crawl set
max_age = 28800
# Limit max. peers per node to be included in crawl set
peers_per_node = 500
# Attempt to establish connection with IPv6 nodes
ipv6 = True
# Limit max. nodes per IPv6 network prefix
ipv6_prefix = 64
nodes_per_ipv6_prefix = 1
# List of excluded ASNs
exclude_asns =
# List of excluded IPv4 networks
exclude_ipv4_networks =
0.0.0.0/8
10.0.0.0/8
100.64.0.0/10
127.0.0.0/8
169.254.0.0/16
172.16.0.0/12
192.0.0.0/24
192.0.0.0/29
192.0.0.170/32
192.0.0.171/32
192.0.0.8/32
192.0.2.0/24
192.168.0.0/16
192.175.48.0/24
192.31.196.0/24
192.52.193.0/24
192.88.99.0/24
198.18.0.0/15
198.51.100.0/24
203.0.113.0/24
240.0.0.0/4
255.255.255.255/32
# List of excluded IPv6 networks
exclude_ipv6_networks =
# Exclude IPv4 bogons
exclude_ipv4_bogons = True
# Exclude IPv6 bogons
exclude_ipv6_bogons = False
# Attempt to establish connection with .onion nodes
onion = True
# Tor proxy is required to connect to .onion address
tor_proxy = 127.0.0.1:9050
# List of initial .onion nodes
onion_nodes =
4crhf372poejlc44.onion
5ghqw4wj6hpgfvdg.onion
5k4vwyy5stro33fb.onion
bitcoinostk4e4re.onion
bk5ejfe56xakvtkk.onion
btcnet3utgzyz2bf.onion
czsbwh4pq4mh3izl.onion
e3tn727fywnioxrc.onion
evolynhit7shzeet.onion
i2r5tbaizb75h26f.onion
jxrvw5iqizppbgml.onion
kjy2eqzk4zwi5zd3.onion
pqosrh6wfaucet32.onion
pt2awtcs2ulm75ig.onion
szsm43ou7otwwyfv.onion
td7tgof3imei3fm6.onion
tfu4kqfhsw5slqp2.onion
vso3r6cmjoomhhgg.onion
xdnigz4qn5dbbw2t.onion
zy3kdqowmrb7xm7h.onion
include_checked = False
# Relative path to directory containing timestamp-prefixed JSON crawl files
crawl_dir = data/crawl/0b110907
ping.py
pingやジェネシスブロックへのinv、addrメッセージを定期的に送信し、ノードとの接続を維持するkeepaliveの実装
Testnet用の設定ファイルを追加
code:console
$ cp conf/ping.conf.default conf/ping.0b110907.conf
$ vi conf/ping.0b110907.conf
# Logfile
logfile = log/ping.0b110907.log
# Network magic number
magic_number = 0b110907
# Redis database number
db = 0
# Max. number of concurrent workers (greenlets) in a pool
workers = 2000
# Print debug output
debug = False
# Public IP address for network interface
source_address = 0.0.0.0
# Protocol version to use for outgoing version message
protocol_version = 70015
# User agent (BIP 0014) to use for outgoing version message
# -----------------------------------------------------------------------------
# NOTE TO USERS
# Please consider changing the user agent before running an instance of this
# crawler. This is so that users will not confuse your crawler with another
# instance that is already running and generating data for the project.
# -----------------------------------------------------------------------------
user_agent = /bitnodes.earn.com:0.1/
# Services to use for outgoing network address message
services = 0
# Set to 1 to receive all txs
relay = 1
# Socket timeout
socket_timeout = 30
# Run cron tasks every given interval
cron_delay = 10
# Limit max. nodes per IPv6 network prefix
ipv6_prefix = 64
nodes_per_ipv6_prefix = 1
# Redis TTL for cached RTT
ttl = 10800
# Attempt to establish connection with .onion nodes
onion = True
# Tor proxy is required to connect to .onion address
tor_proxy = 127.0.0.1:9050
# Relative path to directory containing timestamp-prefixed JSON crawl files
crawl_dir = data/crawl/0b110907
resolve.py
各ノードのホスト名とGeoIPデータの解決。
Testnet用の設定ファイルを追加
code:console
$ cp conf/resolve.conf.default conf/resolve.0b110907.conf
$ vi conf/resolve.0b110907.conf
# Logfile
logfile = log/resolve.0b110907.log
# Network magic number
magic_number = 0b110907
# Redis database number
db = 0
# Print debug output
debug = False
# Redis TTL for cached hostname and GeoIP data
ttl = 86400
export.py
列挙されたノード情報をJSON形式でエクスポート
Testnet用の設定ファイルを追加
code:console
$ cp conf/export.conf.default conf/export.0b110907.conf
$ vi conf/export.0b110907.conf
# Logfile
logfile = log/export.0b110907.log
# Network magic number
magic_number = 0b110907
# Redis database number
db = 0
# Print debug output
debug = False
# Relative path to directory containing timestamp-prefixed JSON export files
export_dir = data/export/0b110907
seeder.py
到達可能なノードをDNSシーダー用のDNSゾーンファイルにエクスポート
Testnet用の設定ファイルを追加
code:console
$ cp conf/seeder.conf.default conf/seeder.0b110907.conf
$ vi conf/seeder.0b110907.conf
# Logfile
logfile = log/seeder.0b110907.log
# Default/fallback port number
port = 18333
# Redis database number
db = 0
# Print debug output
debug = False
# Relative path to directory containing timestamp-prefixed JSON export files
export_dir = data/export/0b110907
# Minimum age for exported nodes
min_age = 28800
# Relative path to DNS zone file containing exported nodes in A/AAAA records
zone_file = data/zone/seed.bitnodes.io.zone
# Initial template for DNS zone file
template = zone.tmpl
# Number of A records to export into DNS zone file
a_records = 25
# Number of AAAA records to export into DNS zone file
aaaa_records = 15
# Number of TXT records to export into DNS zone file
txt_records = 25
pcap.py
Redisのpcapファイルのメッセージを保存
Testnet用の設定ファイルを追加
code:console
$ cp conf/pcap.conf.default conf/pcap.0b110907.conf
$ vi conf/pcap.0b110907.conf
# Logfile
logfile = log/pcap.0b110907.log
# Network magic number
magic_number = 0b110907
# Redis database number
db = 0
# Print debug output
debug = False
# Redis TTL for cached messages
ttl = 10800
# Number of round-trip time (RTT) values to keep for each node
rtt_count = 36
# Number of messages for the same inv to cache from each pcap file
inv_count = 1000
# Tor proxy
tor_proxy = 127.0.0.1:9050
# Relative path to directory containing pcap files
pcap_dir = data/pcap/0b110907
Testnet用の起動ファイルを作成
code:console
$ cp start.sh start_testnet.sh
$ vi start_testnet.sh
設定ファイルが一通り作れたら起動 $ bash start_testnet.sh