What
We would like to be able to evaluate endpoint performance (latency and concurrency) under synthetic network conditions.
Why
We would like to know whether or not a particular SLP (proposed increase in network limits) would cause query performance to suffer under typical network conditions (7 days of retention). As ledgers get denser and more frequent, we want to ensure that queries can still complete in a timely manner, or make informed recommendations about performance mitigations (such as lowering retention windows or increasing hardware requirements).
Details
The current performance evaluation framework supports testing two distinct scenarios:
- How fast can we ingest X synthetic ledgers created with Y profile (for example, 1000 SAC transactions per ledger).
- How much request load can we handle under typical deployment conditions (7 days of retention on mainnet).
We would like to be able to evaluate a 3rd scenario which is a mixture of both of these:
- How much request load can we handle after ingesting X synthetic ledgers with Y profile?
Put differently, we would like to simulate what (2) looks like after (1). For example, if the network supports 1000 SAC TPL (transactions per ledger), what kind of endpoint performance do we get when querying an instance with 7 days of retention?
Proposed Solutions
This issue itself should be used for brainstorming, but we can provide a jumping off point. It seems this can be solved with varying degrees of "accuracy," for lack of a better term:
- History-only simulation: backfill 7 days of synthetic ledgers, live-ingest from pubnet, run the existing load test
- Full simulation: backfill 7 days of synthetic ledgers, simulate real-time ingestion of synthetic ledgers, run the existing load test
The former should be sufficient in the effort vs. results trade-off given the fact that the ingestion performance of individual ledgers is (a) captured by a different leg of the evaluation framework, (b) has minimal variance between SLP-derived ledgers and mainnet, and (c) has minimal impact on query performance. The case can also be made that realistic traffic is better than contrived traffic given the inherent performance variance that occurs ingesting one ledger vs. another in contrast to a uniform stream of SLP-loaded ledgers.
What
We would like to be able to evaluate endpoint performance (latency and concurrency) under synthetic network conditions.
Why
We would like to know whether or not a particular SLP (proposed increase in network limits) would cause query performance to suffer under typical network conditions (7 days of retention). As ledgers get denser and more frequent, we want to ensure that queries can still complete in a timely manner, or make informed recommendations about performance mitigations (such as lowering retention windows or increasing hardware requirements).
Details
The current performance evaluation framework supports testing two distinct scenarios:
We would like to be able to evaluate a 3rd scenario which is a mixture of both of these:
Put differently, we would like to simulate what (2) looks like after (1). For example, if the network supports 1000 SAC TPL (transactions per ledger), what kind of endpoint performance do we get when querying an instance with 7 days of retention?
Proposed Solutions
This issue itself should be used for brainstorming, but we can provide a jumping off point. It seems this can be solved with varying degrees of "accuracy," for lack of a better term:
The former should be sufficient in the effort vs. results trade-off given the fact that the ingestion performance of individual ledgers is (a) captured by a different leg of the evaluation framework, (b) has minimal variance between SLP-derived ledgers and mainnet, and (c) has minimal impact on query performance. The case can also be made that realistic traffic is better than contrived traffic given the inherent performance variance that occurs ingesting one ledger vs. another in contrast to a uniform stream of SLP-loaded ledgers.