Blockchain Performance Benchmarking topics
#benchmarking #performance
Corda, HyperLedger Fabric, Quorum等ethereumベースのチェーンのパフォーマンス計測の観点や計測値の定義を洗い出しておく
代表的な3つの基盤で最も引用されていそうな物を一旦追う
Performance Evaluation of the Quorum Blockchain Platform
TL; DR;
Benchmarking対象: Quorum, 750tps
コンセンサスアルゴリズム別、inputの大きさ、イベントの大きさ、別にRead, Writeのレイテンシを計測
Authors: Arati Baliga, Subhod I, Pandurang Kamat and Siddhartha Chatterjee
https://arxiv.org/pdf/1809.03421.pdf
著者らの所属企業: https://www.persistent.com/
AI等もやっているグローバルに拠点のある企業のようだ
First AuthorはNYU
https://scholar.google.ca/citations?user=NIZTAwwAAAAJ&hl=en
https://twitter.com/aratibaliga
Last AuthorがUC TexasのComputer Architectureの教授のようだ
https://rocketreach.co/siddhartha-chatterjee-email_3340977
https://www.cs.utexas.edu/people/faculty-researchers/siddhartha-chatterjee
tps, 750txs / sec
ベンチの条件を見てるとtpsっておまじないみたいなものだと分かる
計測しているもの
per consensus Algorithm (IBFT, Raft)
null, wirteって区分があるから、何なのかは明確にしたい
per blocktime (100, 1000 ms)
tps (transactions per sec) (vs input transactions rate)
transaction latency (vs input transaction rate)
Read Latency
per read set size (1000b -> 1Mb)
per key value store size
per tx payload, & event payload size
Write Latency
Per write set size (1000b -> 1Mb)
per key value store size
per tx payload, & event payload size
https://gyazo.com/a1fbfc510038b703a3e9147912969c4e
https://gyazo.com/77f61f5b9c9256133aaf6e0845de805b
https://gyazo.com/c98f3701bf1c141f20ada0c9edaae22b
HyperLedger Blockchain Performance Metrics
TL;DR;
HyperLedger ProjectのPerformance & Scale Working GroupのWhite Paper
The group comprises participants from Intel, Redhat, Huawei, etc. as well as R3
用語の整理、メトリクスの整理、Blockchainのパフォーマンス計測のポイントと具体例を提示
https://www.hyperledger.org/wp-content/uploads/2018/10/HL_Whitepaper_Metrics_PDF_V1.01.pdf
実際に色々なベンチマーク系のペーパーで使われているかは確認中
計測のお気持ちを表した図
https://gyazo.com/35426b42ba6d5a1f39886fa217a579b4
用語集
Consensus
Commit
in a traditional database, a commit = point where a tx is written to the database
in BC: defining when this happens across multiple nodes
Finality
once a transaction is committed, it cannot be reversed
voting-based consensus with immediate finality
lottery-based consensus with probabilistic finality.
Network Size
the # of validating nodes participating in consensus in the SUT
Observer Nodeは含まない
Validationを行うNodeが1つで、他に色々なNodeがいるが、究極的にはValidationを行うNodeがSPOF担っている場合は# of Network Sizeは1とする
Query
ad-hocな操作やsearchをBC上のデータセットに対して行うことを指す
The SUT may not be built to execute these queries in a performant way.
off-ledgerのDBやキャッシュにより効率的なQuery操作を行うこと、その効果も計測の対象とする
Mertics V1ではQuery performancはscope外とする(え)
Reads
stateを書き換えない操作
internal reads
BCのノードの内部的な操作を指している
txのバリデーションのためにdataをfetch等
external reads
外から要求されて取る場合
getTx(), getBalance()的なメソッドが呼ばれるケース
Metrics V1ではreads = external Reads
State and Global State
State = contents of the database at a point in time
The log of transactions (the blockchain data structure) is not generally considered part of state
Global State = a state shared across all nodes
Cordaの場合はGlobal Stateは基本的にはNotaryのtx -> 使用済みor notのマップのみ、残りは設計に依るか
Transaction
state transition
typically proposed by clients
evaluated against list of rules, and smart contracts
if valid ocmmit the tx -> state change.
メトリクス
Read / Write Latency
クライアント視点の値
Latency = Time when response received – submit time
Read Throughput
Read Throughput = Total read operations / total time in seconds
Transaction Latency (Write Latency, Read Latency)
ネットワーク全体で見た時の値
txがネットワーク経由でusable(stateの書き換わりを検証できる等?)になるまでにかかる時間
This includes the propagation time and any settling time due to the consensus mechanism in place
全Nodeで測る必要がある
propagation time
measured using all nodes in the SUT. Eyal et al provide a helpful definition for the amount of time for a percentage of the network to commit the transaction1
settling time
probilistic / deterministic
何%がtxをusableであるかのthresholdを設定する必要がある
deterministicな場合は100%にしないと意味がない
Transaction Latency = (Confirmation time @ network threshold) – submit time
Transaction Latency is a network-wide view of the amount of time taken for a transaction’s effect to be usable across the network.
Transaction Throughput
Transaction Throughput = Total committed transactions / total time in seconds @ # committed nodes
transaction throughput is the rate at which valid transactions are committed by the blockchain SUT in a defined time period
1ノードの値ではない
SUT全体でcommitされるまでにかかる時間
テスト環境として考慮すべきもの
Consensus Protocol
Geographic distribution of nodes
Hardware environment of all peers.
processor spec
# of cores
memory
etc
Network Model
ネットワークの構造等、ボトルネックになりそうなものがないか
Number of nodes involved in the test transaction
Software component dependencies.
Test tools and framework.
Type of data store used.
CouchDB?, H2, Postgres ...?
Workload
Transaction Characteristics
Complexity: How compute-intensive is the logic of the smart contract?
Data access patterns: Do the patterns of reads and writes model the production use?
Dependencies: Do the transactions model the transaction and data dependencies of your production use?
Size: How large are the transactions? This has implications for the propagation times across the network.
DLT Performance Considerations
https://medium.com/corda/dlt-performance-considerations-7e1a2cc964b0
R3はHyperLedgerのPSWGに参加しているので検証観点や用語の定義などはHyperLedgerの基準に準拠している(下記はブログに書かれた一例)
上記ブログに書かれた気にするべきポイント一覧
あんまり大したことはかいてないので割愛
How is transaction defined for this claim?
How is throughput defined for this claim, considering what were the start and stop points for measuring TPS?
What number of peer nodes are meant to receive a copy of the transaction, AND are involved in validating the transaction, e.g., is the platform broadcasting, is it limiting interaction between parties?
What consensus model is used to validate the transactions? What protocol? Is it probabilistic? What is the consensus delay?
What data store is used?
What peer communication protocol is used?
What were the software and hardware environment conditions for the test?
What use case(s) did the test workload represent?
What testing tools were used to complete the test?
May the results be replicated by other parties?
What are the performance SLA for the target network?
パフォーマンス検証の観点でCordaがNotaryという時は、暗黙的にNotary Poolを指している
Performance Benchmarking and Optimizing Hyperledger Fabric Blockchain Platform
P Thakkar et al.
BLOCKBENCH: A Framework for Analyzing Private Blockchains
2017
SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data
citation 302
Tien et al.
NUS (National University of Singapore)
https://www.comp.nus.edu.sg/~ooibc/
他にもチェーン比較系のサーベイ論文書いてる
https://www.comp.nus.edu.sg/~ooibc/blockchainsurvey.pdf
2018
https://github.com/ooibc88/FabricSharp
これのチーム
ベンチの対象: Etherem PoW vs Parity PoA vs HyperLedger Fabric
Metrics
Throughput
# of successful txs /sec
Latency
response time
Scalabilty
the changes in throughput and latency when incresing # of nodes & concurrent workloads
Fault tolerance
?(細かく見ていない)
Workloads
https://gyazo.com/6b5666edf9eb57fe7890cbceb17b2850
YCSB
simple put and get
Smallbank
bank operationをシミュレートしたこの手のベンチマークではおなじみのワークロードだそう
https://github.com/ooibc88/blockbench/blob/master/benchmark/contracts/ethereum/smallbank.sol
https://okachimachiorz.hatenablog.com/entry/20130331/1364709005
https://github.com/hyperledger/caliper-benchmarks/tree/master/benchmarks/scenario/smallbank
EtherId
Doubler
WavesPresale
VersionKVStore
IOHeavy
CPUHeavy
DoNothing
https://gyazo.com/2e37cd8e85e364685d8d7c690c65d148
https://gyazo.com/ff104af8b368d4fe953d6db9cd03b87f
https://gyazo.com/4ae0e6dbada26ef12f643cc235f12fa3
https://gyazo.com/279e0d5dff8172d1e4629bec56a4a16a
https://gyazo.com/80c21ece70dc2dd4930a49f1fbde6c98
https://gyazo.com/562ee6cdf3fc8e60344db4c28a6f57cb
On Scaling Decentralized Blockchains
Financial Cryptography & Data Security 16'
Kyle et al.
メモ
用語集
http://www.olapcouncil.org/research/spec1.htm
OLAP