Audit Application Form
Let's talk
Polkadot Benchmarking:
Main Findings
Author: Sergey Govyazin
Blockchain Developer
Blockchain Developer
Foreword
MixBytes.Tank & Tank.bench
Polkadot under synthetic load using MixBytes
Additional readings and tutorials
Тest 1: 21 validators, a localized cluster, stable connection
Test 2: 49 validators, a localized cluster, stable connection
Test 3: 49 validators, a geographically distributed cluster, stable connection
Test 4: 49 validators, a localized cluster, 3% packet loss
Test 5 - 49 validators, a localized cluster, 5% packet loss
Test 6: 99 validators, a localized cluster, stable connection
Test 7: 100 validators, a geographically distributed cluster, stable connection
Test 8 - 99 validators, a localized cluster, 7% packet loss
Test 9 - 100 validators, a geographically distributed cluster, 7% packet loss
Test 10: 31 validators, a localized cluster, stable connection
Conclusion
Foreword
Testing is often neglected or put on the back burner. At MixBytes, we don't think development must be the only priority.
With the help of MixBytes Tank - our tool for complex testing and benchmarking - we decided to challenge several blockchain projects, Polkadot goes the first in the series. Out of its numerous released versions, we chose 3 and version 0.6.18 demonstrated the best results.*
The main thing - the network withstood our heavy load attack, other details of our hard work are presented in a comprehensive report below. Buckle up!
*Before the report publication version 0.7.18 was released. We chose one of the most comprehensive tests to track and demonstrate new version performance (please see test 10).
The log slices, graph screenshots and test cases mentioned across the report can be found here.
With the help of MixBytes Tank - our tool for complex testing and benchmarking - we decided to challenge several blockchain projects, Polkadot goes the first in the series. Out of its numerous released versions, we chose 3 and version 0.6.18 demonstrated the best results.*
The main thing - the network withstood our heavy load attack, other details of our hard work are presented in a comprehensive report below. Buckle up!
*Before the report publication version 0.7.18 was released. We chose one of the most comprehensive tests to track and demonstrate new version performance (please see test 10).
The log slices, graph screenshots and test cases mentioned across the report can be found here.
MixBytes.Tank & Tank.bench
MixBytes.Tank is an easy configurable tool for a fast and simple deployment of blockchain networks. One command is enough to launch different blockchain networks with the desired parameters, such as the number of nodes, the number of validators, machine computing power, etc.
Tank.bench is a tool allowing to create diverse loading for blockchains, mainly by transactions. It is integrated into Mixbytes.Tank. To load a cluster deployed using MixBytes.Tank, just run one command.
In order for these utilities to work with a specific blockchain, it is necessary to write the corresponding bindings, i.e. the code specific to a particular blockchain. Tank requires an instruction for deploying Polkadot nodes, and the bench - the code that creates transactions and sends them to the network.
To run a Polkadot network for subsequent performance measurements and troubleshooting, the MixBytes team developed 2 tools: a Polkadot binding for MixBytes.Tank and a Polkadot binding for Tank.bench.
The comprehensive code is located in the following repositories:
https://github.com/mixbytes/tank
https://github.com/mixbytes/tank.ansible-polkadot
https://github.com/mixbytes/tank.bench-polkadot
Tank.bench is a tool allowing to create diverse loading for blockchains, mainly by transactions. It is integrated into Mixbytes.Tank. To load a cluster deployed using MixBytes.Tank, just run one command.
In order for these utilities to work with a specific blockchain, it is necessary to write the corresponding bindings, i.e. the code specific to a particular blockchain. Tank requires an instruction for deploying Polkadot nodes, and the bench - the code that creates transactions and sends them to the network.
To run a Polkadot network for subsequent performance measurements and troubleshooting, the MixBytes team developed 2 tools: a Polkadot binding for MixBytes.Tank and a Polkadot binding for Tank.bench.
The comprehensive code is located in the following repositories:
https://github.com/mixbytes/tank
https://github.com/mixbytes/tank.ansible-polkadot
https://github.com/mixbytes/tank.bench-polkadot
Polkadot under synthetic load using MixBytes.Tank & Tank.bench. Research results
The corresponding Tank.bench binding and 2 profiles were written for Polkadot research. One of them loads blockchain with transfer transactions, the other "stakes" funds.
Polkadot v.0.6.18 with the disabled OnlyStakingAndClaims plugin for transfer transactions was tested.
Each test is performed using a specific configuration of the tank and bench. Configuration details are described in the respective repositories.
Tests were made using the machines of Google Cloud Engine and DigitalOcean cloud providers.
Large machine configuration:
Ubuntu 18.04.3 (LTS) x64; Disk = 80GB; Memory = 8192MB; VCpu = 4;
*The number of deployed machines exceeds the specified number of regular validators by 2 items, as in addition to regular validators, a boot-node and a monitoring machine are also launched.
For tests with less than 60 machines, the logs are fully presented, while for tests with a large number of machines, the logs are the last 100 lines of each machine log.
During testing, Polkadot released several new versions. To find out if the problems, identified during previous benchmarks, were fixed, a new version 0.6.18 was tested.
Tank.bench-polkadot was updated, as the previous version used for testing polkadot 0.6.2 failed. After updating the dependencies, the load module resumed its work. Below are some results of our tests.
Polkadot v.0.6.18 with the disabled OnlyStakingAndClaims plugin for transfer transactions was tested.
Each test is performed using a specific configuration of the tank and bench. Configuration details are described in the respective repositories.
Tests were made using the machines of Google Cloud Engine and DigitalOcean cloud providers.
Large machine configuration:
Ubuntu 18.04.3 (LTS) x64; Disk = 80GB; Memory = 8192MB; VCpu = 4;
*The number of deployed machines exceeds the specified number of regular validators by 2 items, as in addition to regular validators, a boot-node and a monitoring machine are also launched.
For tests with less than 60 machines, the logs are fully presented, while for tests with a large number of machines, the logs are the last 100 lines of each machine log.
During testing, Polkadot released several new versions. To find out if the problems, identified during previous benchmarks, were fixed, a new version 0.6.18 was tested.
Tank.bench-polkadot was updated, as the previous version used for testing polkadot 0.6.2 failed. After updating the dependencies, the load module resumed its work. Below are some results of our tests.
Тest 1: 21 validators, a localized cluster, stable connection
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
producer:
count: 20
type: large
Tank.bench-polkadot
1,2) tps: 20, total-tx: 1000
profile: transfer
3) tps: 200, total-tx: 10000
profile: transfer
4) tps: 400, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a network with 21 validators. Load testing might be performed if no errors arise.
Here and further on TPS parameter of the Tank.bench-polkadot utility does not imply that the blockchain was able to process all the transactions at such speed. The parameter just shows the degree of intensity with which transactions are added to the mempool.
The data re transactions added to the blocks can be found in the graph "Extrinsics for last 2m".
Here and further on TPS parameter of the Tank.bench-polkadot utility does not imply that the blockchain was able to process all the transactions at such speed. The parameter just shows the degree of intensity with which transactions are added to the mempool.
The data re transactions added to the blocks can be found in the graph "Extrinsics for last 2m".
Graphs
Results
The network was deployed in 10 minutes and worked for about 50 minutes, then the test was stopped. Four load sessions in the form of transfer transactions were applied.
It is clear that the network is stable in between load test launches, node cpu consumption does not exceed 10%.
All load tests showed almost the same results: first transactions are immediately added to blocks, the blocks are quickly finalized. At some point, transactions begin to fall into the mempool and reluctantly get into blocks. After some time, the mempool is cleared.
In all load tests logs of the load utility demonstrate that some transactions throw an exception:
RPC-CORE: submitExtrinsic(extrinsic: Extrinsic): Hash:: 1010: Invalid Transaction: Stale
Consequently, these transactions don't reach validators.
The network became much more stable than in the previous series of experiments. Validators are no longer disconnected due to incomprehensible exceptions. There's no delay in the block schedule, validators almost don't miss finalization. The problem of a large number of transactions filling the pools and not getting into blocks still remains.
It is clear that the network is stable in between load test launches, node cpu consumption does not exceed 10%.
All load tests showed almost the same results: first transactions are immediately added to blocks, the blocks are quickly finalized. At some point, transactions begin to fall into the mempool and reluctantly get into blocks. After some time, the mempool is cleared.
In all load tests logs of the load utility demonstrate that some transactions throw an exception:
RPC-CORE: submitExtrinsic(extrinsic: Extrinsic): Hash:: 1010: Invalid Transaction: Stale
Consequently, these transactions don't reach validators.
The network became much more stable than in the previous series of experiments. Validators are no longer disconnected due to incomprehensible exceptions. There's no delay in the block schedule, validators almost don't miss finalization. The problem of a large number of transactions filling the pools and not getting into blocks still remains.
Logs
Log slices of this test are given in the following files:
- 1_1.log (before network launch)
- 1_2.log (before load test 1 start)
- 1_3.log (before load test 2 start)
- 1_4.log (before load test 3 start)
- 1_5.log (before load test 4 start)
- 1_6.log (before network stop, after load test 5 start).
Test 2: 49 validators, a localized cluster, stable connection
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
producer:
count: 48
type: large
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
3) tps: 400, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a network with 49 validators. Load testing might be performed if no errors arise.
Graphs
Results
The network was deployed in 15 minutes and worked for about 1hr 30 minutes, then the test was stopped. Three load sessions in the form of transfer transactions were applied.
It is clear that the network is stable in between load test launches, node cpu consumption does not exceed 20%. However, cpu consumption increased as opposed to the previous test.
The results and some errors are also similar to those of the previous test.
It is clear that the network is stable in between load test launches, node cpu consumption does not exceed 20%. However, cpu consumption increased as opposed to the previous test.
The results and some errors are also similar to those of the previous test.
Logs
Log slices of this test are given in the following files:
- 2_1.log (during load test 1)
- 2_2.log (after load test 1 launch)
- 2_3.log (after load test 2 launch)
- 2_4.log (after load test 3 launch)
Test 3: 49 validators, a geographically distributed cluster, stable connection
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
producer:
regions:
Europe:
count: 20
type: large
Asia:
count: 20
type: large
North America:
count: 8
type: large
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
Description
The test checks the behaviour of a network with 49 validators. Validators are geographically distributed.
Graphs
Results
The network was deployed in 15 minutes and worked for about 1hr, then the test was stopped. Two load sessions in the form of transfer transactions were applied.
The results are similar to those of the previous test. Geographical distribution of the network did not affect its stability. The network performed stably, without errors (except for the described above, they were still present), validators lost finalization only at the beginning of load start.
The results are similar to those of the previous test. Geographical distribution of the network did not affect its stability. The network performed stably, without errors (except for the described above, they were still present), validators lost finalization only at the beginning of load start.
Logs
Log slices of this test are given in the following files:
- 3_1.log (after load test 1 launch)
- 3_2.log (after load test 2 launch)
Test 4: 49 validators, a localized cluster, 3% packet loss
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
packet loss: 3
producer:
count: 48
type: large
packet loss: 3
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
3) tps: 400, total-tx: 100000
profile: transfer
4) tps: 20, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a network with 49 validators. As the network has 3% packet loss, the test will show if minor packet loss affects network stability.
Graphs
Results
The network was deployed in 14 minutes and worked for about 2 hr 30 minutes, then the test was stopped. Four load sessions in the form of transfer transactions were applied.
Packet loss had an impact on CPU and RAM usage by validators. CPU consumption increased to almost 20% in a non-transactional mode, memory consumption increased by 10-20%.
This test included 3 sessions of highload transactions and 1 session of a large transaction number load with low TPS (100,000 transactions at a speed of 20 TPS). Under such load, an average CPU load was 30-40%. The network kept finalizing blocks and kept working stably.
During transaction load, validators have the following log:
Unable to send a validate transaction result: Ok
This might happen due to packet loss. However, such logs do not affect overall network stability.
Packet loss had an impact on CPU and RAM usage by validators. CPU consumption increased to almost 20% in a non-transactional mode, memory consumption increased by 10-20%.
This test included 3 sessions of highload transactions and 1 session of a large transaction number load with low TPS (100,000 transactions at a speed of 20 TPS). Under such load, an average CPU load was 30-40%. The network kept finalizing blocks and kept working stably.
During transaction load, validators have the following log:
Unable to send a validate transaction result: Ok
This might happen due to packet loss. However, such logs do not affect overall network stability.
Logs
Log slices of this test are given in the following files:
- 4_1.log (after the network launch)
- 4_2.log (after load test 1 launch)
- 4_3.log (after load test 2 launch)
- 4_4.log (during load test 3 launch)
- 4_5.log (after load test 5 launch)
Test 5 - 49 validators, a localized cluster, 5% packet loss
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
packet loss: 5
producer:
count: 48
type: large
packet loss: 5
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
3) tps: 400, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a network with 49 validators. As the network has 5% packet loss, the test will show if minor packet loss affects network stability.
Graphs
Results
The network was deployed in 15 minutes and worked for about 2 hours, then the test was stopped. Three load sessions in the form of transfer transactions were applied.
The results are almost the same as in the previous test. There are still network error messages on validators under transaction load. However, this almost does not affect the network itself but increases the load on CPU under active load.
The results are almost the same as in the previous test. There are still network error messages on validators under transaction load. However, this almost does not affect the network itself but increases the load on CPU under active load.
Logs
Log slices for this test are given in the following files:
- 5_1.log (after the network launch)
- 5_2.log (after load test 1 launch)
- 5_3.log (after load test 2 launch)
- 5_4.log (after load test 3 launch)
Test 6: 99 validators, a localized cluster, stable connection
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
producer:
count: 98
type: large
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
3) tps: 400, total-tx: 100000
profile: transfer
4) tps: 400, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a network with 99 validators.
Graphs
Results
The network was deployed in 25 minutes and worked for about 4 hours, then the test was stopped. Four load sessions in the form of transfer transactions were applied. The last session simulated a continuous load.
The results don't differ from the previous ones. When the mempool is full, validators no longer accept transactions but mark them as "transaction invalid".
Also, under heavy load some validators may lose connection, and the following log pops up:
Unable to send a validate transaction result: Ok
CPU consumption of some validators did not decrease after the load was over. However, no additional information was observed in their logs.
The results don't differ from the previous ones. When the mempool is full, validators no longer accept transactions but mark them as "transaction invalid".
Also, under heavy load some validators may lose connection, and the following log pops up:
Unable to send a validate transaction result: Ok
CPU consumption of some validators did not decrease after the load was over. However, no additional information was observed in their logs.
Logs
Log slices of this test are given in the following files:
- 6_1.log (after load test 1 launch)
- 6_2.log (after load test 2 launch)
- 6_3.log (after load test 3 launch)
- 6_4.log (after load test 4 launch)
- 6_5.log (after load test 5 launch)
Test 7: 100 validators, a geographically distributed cluster, stable connection
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
producer:
regions:
Europe:
count: 33
type: large
Asia:
count: 33
type: large
North America:
count: 33
type: large
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
3) tps: 600, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a geographically distributed network with 100 validators.
Graphs
Results
The network was deployed in 25 minutes and worked for about 3 hours, then the test was stopped. Three load sessions in the form of transfer transactions were applied.
Geographical distribution did not affect network stability.
As in the previous test, CPU consumption of some validators did not decrease after the load was over. However, no additional information was observed in their logs.
A log informing about impossible connection under load is still present:
Unable to send a validate transaction result: Ok
Geographical distribution did not affect network stability.
As in the previous test, CPU consumption of some validators did not decrease after the load was over. However, no additional information was observed in their logs.
A log informing about impossible connection under load is still present:
Unable to send a validate transaction result: Ok
Logs
Log slices of this test are given in the following files:
- 7_1.log (after load test 1 launch)
- 7_2.log (after load test 2 launch)
- 7_3.log (after load test 3 launch)
Test 8 - 99 validators, a localized cluster, 7% packet loss
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
packet loss: 7
producer:
count: 98
type: large
packet loss: 7
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
3) tps: 600, total-tx: 100000
profile: transfer
Description
The test checks the behaviour of a localized network with 99 validators and 7% packet loss.
Graphs
Results
The network was deployed in 24 minutes and worked for 1.5 hours, then the test was stopped. Three load sessions in the form of transfer transactions were applied.
7% packet loss affected CPU consumption of validators in idle mode, increasing it by 5%, and prevented one of the validators to synchronize with the network immediately (instead, it happened 10 minutes after the network was deployed).
Packet loss also led to the fact that some validators were losing finalization even without load, but caught up with it after a while.
After the load was over, CPU behaved oddly again. For some reason, it did not decrease after transactions cease to reach a validator.
7% packet loss affected CPU consumption of validators in idle mode, increasing it by 5%, and prevented one of the validators to synchronize with the network immediately (instead, it happened 10 minutes after the network was deployed).
Packet loss also led to the fact that some validators were losing finalization even without load, but caught up with it after a while.
After the load was over, CPU behaved oddly again. For some reason, it did not decrease after transactions cease to reach a validator.
Logs
Log slices of this test are given in the following files:
- 8_1.log (before load test 1 launch)
- 8_2.log (after load test 1 launch)
- 8_3.log (after load test 2 launch)
- 8_4.log (after load test 3 launch)
Test 9 - 100 validators, a geographically distributed cluster, 7% packet loss
Configuration
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
packet loss: 7
producer:
regions:
Europe:
count: 33
type: large
packet loss: 7
Asia:
count: 33
type: large
packet loss: 7
North America:
count: 33
type: large
packet loss: 7
Tank.bench-polkadot
1) tps: 20, total-tx: 1000
profile: transfer
2) tps: 200, total-tx: 10000
profile: transfer
Description
The test checks performance of a geo-distributed network consisting of 100 validators and having 7% packet loss. This mode simulates a real network in which clusters are at a decent distance from each other and packet loss is highly likely.
Graphs
Results
The network was deployed in 26 minutes and worked for 2 hours,then the test was stopped. Three load sessions in the form of transfer transactions were applied.
Cluster geo-distribution and 7% packet loss affect validator CPU consumption in idle mode, increasing it almost by 5%. Some validators did not synchronize with the network immediately but approx. 10 minutes after the network was deployed. This result repeats the previous one.
CPU behaved strangely again after the load ended. For some reason, it did not decrease after transactions ceased to reach validators.
In general, the network worked without any problems. It is interesting that the transaction pool of some validators could not be emptied for a very long time (see the graphs).
Cluster geo-distribution and 7% packet loss affect validator CPU consumption in idle mode, increasing it almost by 5%. Some validators did not synchronize with the network immediately but approx. 10 minutes after the network was deployed. This result repeats the previous one.
CPU behaved strangely again after the load ended. For some reason, it did not decrease after transactions ceased to reach validators.
In general, the network worked without any problems. It is interesting that the transaction pool of some validators could not be emptied for a very long time (see the graphs).
Logs
Log slices of this test are given in the following files:
- 9_1.log (before load test 1 launch)
- 9_2.log (after load test 1 launch)
- 9_3.log (after load test 2 launch)
- 9_4.log (after load test 3 launch)
Test 10: 31 validators, a localized cluster, stable connection
Configuration
Polkadot version: 0.7.18
Polkadot version: 0.7.18
MixBytes.Tank
binding: polkadot
instances:
boot:
count: 1
type: large
producer:
count: 30
type: large
Tank.bench-polkadot
1,2) tps: 20, total-tx: 1000
profile: transfer
3) tps: 200, total-tx: 10000
profile: transfer
4) tps: 400, total-tx: 100000
profile: transfer
Description
The test checks the performance of a network consisting of 31 validators. Load testing is to be carried out in case of error absence.
Graphs
Results
The network was deployed in 11 minutes and worked for about 1 hour, then the test was stopped. Three load sessions in the form of transfer transactions were applied.
It is clear that the network is stable in between load test launches, node cpu consumption does not exceed 10%. During active load, cpu consumption of some nodes is 40%, others - 10-20%. The reason for the difference is unknown.
The network passed the first load test (1000 total tx, 10 tps) without problems. No errors were observed.
The second load test (10 000 total tx, 200 tps) was also successful, however, the following error appeared once on one of the load modules:
It is clear that the network is stable in between load test launches, node cpu consumption does not exceed 10%. During active load, cpu consumption of some nodes is 40%, others - 10-20%. The reason for the difference is unknown.
The network passed the first load test (1000 total tx, 10 tps) without problems. No errors were observed.
The second load test (10 000 total tx, 200 tps) was also successful, however, the following error appeared once on one of the load modules:
The third load test (100 000 total tx, 400 tps) was successful. However, even after its completion, CPU consumption of some node did not lower, it was still at 40%. The reason is not established, the logs are attached.
All load tests showed approximately the same results: the first transactions are immediately added to the blocks, blocks with them are quickly finalized. At some point, transactions begin to fall into the mempool and become very reluctant to reach the blocks. After some time, the mempool is cleared.
All load tests showed approximately the same results: the first transactions are immediately added to the blocks, blocks with them are quickly finalized. At some point, transactions begin to fall into the mempool and become very reluctant to reach the blocks. After some time, the mempool is cleared.
Logs
Log slices of this test are given in the following files:
- 1.log (after load test 1)
- 2.log (after load test 2)
- 3.log (after load test 3)
Conclusion
After conducting tests, we made several conclusions regarding the Polkadot network performance:
- While testing versions 0.4.4 and 0.6.2, we identified several errors which ceased normal validator operation. In some cases, the network failed to demonstrate stable block production and finality even on small localized clusters. Low packet loss could significantly worsen validator functioning. We suggested paying special attention to synchronization of blocks and memory pool between validators and finality patterns in some cases. All the mentioned problems were fixed in version 0.6.18, but for limited transaction inclusion rate issue.
- The version 0.7.18 did not show an improvement in stability compared to version 0.6.18. However, load modules no longer show some errors that occurred during testing version 0.6.18.
- The network withstands a fairly strong transaction load. However, once a transaction pool is full, a validator fails to process transactions until the pool is emptied. This can be regarded as antispam network protection but you can clog all validator pools and thus stop the network (it will be impossible to send transactions).
- For some reason, transactions are included into blocks at a quite limited rate.
- Even under heavy load and network packet loss, validators do not lag behind other validators in finalization (no more than 5 blocks).
- A new method of adding keys to a validator via API (compared to prior, less than <0.6 Polkadot versions) is less convenient and requires additional devops work.
Other posts