Conversation
Contributor
🧪 Performance Evaluation Test #2Commit: ✅ Backfill ingestion — verdict: ok⏳ Backfill ingestion —
|
| Metric | Value |
|---|---|
| Ledgers ingested | 120960 ([64326400 -> 64447359]) |
| Retention window | 120960 |
| Wall-clock (total) | 3h6m1s |
| Ingest phase | 2h34m59s |
| Bulk-load finalize phase | 31m3s |
| Ledgers/sec (ingest) | 13.0 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — 69ffcb1ea9e4
Serial blast per endpoint (ramp-up 1m, duration 3m, error kill switch 75%, blaster aadc1a17595f) against the backfilled RPC (ledgers [64327144, 64448103], handoff wait 1622s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents | 75 | 11225 | 2 (0.0%) | 0.7 | 15.1 | 175.1 | 1670.1 |
| getFeeStats | 250 | 37491 | 0 (0.0%) | 0.5 | 0.6 | 109.4 | 185.2 |
| getHealth | 250 | 37490 | 0 (0.0%) | 0.4 | 0.6 | 121.3 | 206.5 |
| getLatestLedger | 15 | 2242 | 0 (0.0%) | 116.0 | 172.2 | 249.5 | 308.2 |
| getLedgers (limit=5) | 3 | 440 | 0 (0.0%) | 641.5 | 1400.8 | 2052.1 | 2168.8 |
| getNetwork | 100 | 14976 | 0 (0.0%) | 31.8 | 104.3 | 161.7 | 198.5 |
| getTransaction | 75 | 11223 | 0 (0.0%) | 34.9 | 62.6 | 130.0 | 223.9 |
| getTransactions (limit=200) | 10 | 1473 | 0 (0.0%) | 140.2 | 382.5 | 456.4 | 528.9 |
| getVersionInfo | 100 | 14932 | 0 (0.0%) | 32.0 | 127.9 | 240.9 | 294.9 |
getEvents results extended
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents/catch-up | 3 | 476 | 0 (0.0%) | 5.7 | 18.9 | 93.2 | 267.5 |
| getEvents/deep-pager | 17.25 | 2544 | 0 (0.0%) | 0.7 | 53.8 | 199.9 | 1401.9 |
| getEvents/deep-scan | 2.25 | 344 | 0 (0.0%) | 0.7 | 10.3 | 84.9 | 243.2 |
| getEvents/firehose | 1.5 | 219 | 0 (0.0%) | 0.7 | 5.9 | 9.1 | 93.3 |
| getEvents/head-poll | 36 | 5413 | 0 (0.0%) | 0.7 | 7.2 | 133.5 | 235.4 |
| getEvents/tail-poll | 9 | 1341 | 2 (0.1%) | 0.7 | 3.7 | 1273.9 | 10002.4 |
| getEvents/transfer-watcher | 6 | 888 | 0 (0.0%) | 5.3 | 10.2 | 153.7 | 188.7 |
✅ Go endpoint benchmarks — verdict: ok
Baseline v28.0.1 (273f19e4fcb1) vs candidate 69ffcb1ea9e4 — -benchmem -count=10, both refs sequentially on one box; rpcv2 excluded.
benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │
│ sec/op │
GetLedgerRange-8 952.2n ± 1%
BatchGetLedgers-8 747.6µ ± 1%
geomean 26.68µ
│ baseline.txt │
│ B/op │
GetLedgerRange-8 16.00 ± 0%
BatchGetLedgers-8 520.8Ki ± 0%
geomean 2.853Ki
│ baseline.txt │
│ allocs/op │
GetLedgerRange-8 4.000 ± 0%
BatchGetLedgers-8 2.314k ± 0%
geomean 96.21
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
│ baseline.txt │
│ sec/op │
ComputeFeeDistribution/computeFeeDistribution-8 9.396µ ± 0%
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 217.7µ ± 1%
geomean 45.23µ
│ baseline.txt │
│ B/op │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0%
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
│ baseline.txt │
│ allocs/op │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0%
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetEventsTopicFilters-8 2.612m ± 1% 2.572m ± 1% -1.51% (p=0.001 n=10)
GetEvents-8 157.3µ ± ∞ ¹ 160.3µ ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.344m ± ∞ ¹ 1.128m ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 7.859µ ± 0% 7.682µ ± 0% -2.26% (p=0.000 n=10)
JSONTransactions/XDR_format-8 2.583µ ± 0% 2.600µ ± 0% +0.66% (p=0.000 n=10)
GetProtocolVersion-8 42.59µ ± ∞ ¹ 43.94µ ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 88.40µ 86.13µ -2.57%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetEventsTopicFilters-8 191.0Ki ± 1% 190.2Ki ± 0% -0.39% (p=0.029 n=10)
GetEvents-8 43.16Ki ± ∞ ¹ 43.16Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.369Mi ± ∞ ¹ 1.043Mi ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 2.000Ki ± 0% 2.000Ki ± 0% ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8 1.336Ki ± 0% 1.336Ki ± 0% ~ (p=1.000 n=10) ³
GetProtocolVersion-8 7.908Ki ± ∞ ¹ 7.456Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 25.00Ki 23.64Ki -5.44%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetEventsTopicFilters-8 1.526k ± 1% 1.520k ± 0% -0.46% (p=0.039 n=10)
GetEvents-8 621.0 ± ∞ ¹ 621.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 4.723k ± ∞ ¹ 4.122k ± ∞ ¹ ~ (p=1.000 n=1) ³
JSONTransactions/JSON_format-8 25.00 ± 0% 25.00 ± 0% ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8 23.00 ± 0% 23.00 ± 0% ~ (p=1.000 n=10) ²
GetProtocolVersion-8 122.0 ± ∞ ¹ 120.0 ± ∞ ¹ ~ (p=1.000 n=1) ³
geomean 260.7 254.0 -2.59%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ need >= 4 samples to detect a difference at alpha level 0.05
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetPreflight-8 802.4µ ± 1% 803.5µ ± 0% ~ (p=0.631 n=10)
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetPreflight-8 52.10Ki ± 0% 52.10Ki ± 0% ~ (p=0.593 n=10)
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetPreflight-8 290.0 ± 0% 290.0 ± 0% ~ (p=1.000 n=10)
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/rpcv1/sqlitedb
│ candidate.txt │
│ sec/op │
GetLedgerRange-8 941.3n ± 1%
BatchGetLedgers-8 722.7µ ± 2%
geomean 26.08µ
│ candidate.txt │
│ B/op │
GetLedgerRange-8 16.00 ± 0%
BatchGetLedgers-8 476.8Ki ± 0%
geomean 2.730Ki
│ candidate.txt │
│ allocs/op │
GetLedgerRange-8 4.000 ± 0%
BatchGetLedgers-8 3.120k ± 0%
geomean 111.7
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/store
│ candidate.txt │
│ sec/op │
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 209.9µ ± 2%
ComputeFeeDistribution/ComputeFeeDistribution-8 9.493µ ± 0%
geomean 44.64µ
│ candidate.txt │
│ B/op │
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0%
ComputeFeeDistribution/ComputeFeeDistribution-8 0.000 ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
│ candidate.txt │
│ allocs/op │
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0%
ComputeFeeDistribution/ComputeFeeDistribution-8 0.000 ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/35023739904/go-bench/): baseline.txt, benchstat.txt, candidate.txt
Performance Evaluation Test #1
🧪 Performance Evaluation Test #1
Commit: 5845cd221109 (benchmark-old-vs-new)
Run: https://github.com/stellar/stellar-rpc/actions/runs/34909144389
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — 5845cd221109
| Metric | Value |
|---|---|
| Ledgers ingested | 120960 ([64312512 -> 64433471]) |
| Retention window | 120960 |
| Wall-clock (total) | 3h5m34s |
| Ingest phase | 2h34m59s |
| Bulk-load finalize phase | 30m34s |
| Ledgers/sec (ingest) | 13.0 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — 5845cd221109
Serial blast per endpoint (ramp-up 1m, duration 3m, error kill switch 75%, blaster aadc1a17595f) against the backfilled RPC (ledgers [64313208, 64434167], handoff wait 1472s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents | 75 | 11225 | 6 (0.1%) | 0.7 | 15.4 | 136.4 | 1397.8 |
| getFeeStats | 250 | 37493 | 0 (0.0%) | 0.5 | 0.6 | 160.5 | 273.2 |
| getHealth | 250 | 37493 | 0 (0.0%) | 0.5 | 0.6 | 148.2 | 222.5 |
| getLatestLedger | 15 | 2245 | 0 (0.0%) | 140.8 | 276.5 | 362.0 | 465.7 |
| getLedgers (limit=5) | 3 | 442 | 0 (0.0%) | 650.8 | 1490.9 | 2051.1 | 2439.2 |
| getNetwork | 100 | 14978 | 0 (0.0%) | 38.1 | 1922.0 | 2299.9 | 2351.1 |
| getTransaction | 75 | 11225 | 0 (0.0%) | 33.2 | 57.5 | 107.4 | 239.1 |
| getTransactions (limit=200) | 10 | 1473 | 0 (0.0%) | 129.0 | 411.4 | 485.4 | 601.1 |
| getVersionInfo | 100 | 13860 | 0 (0.0%) | 22.7 | 54.8 | 126.6 | 193.5 |
getEvents results extended
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents/catch-up | 3 | 427 | 0 (0.0%) | 7.9 | 37.8 | 90.4 | 252.3 |
| getEvents/deep-pager | 17.25 | 2565 | 1 (0.0%) | 0.7 | 48.4 | 205.4 | 1397.8 |
| getEvents/deep-scan | 2.25 | 308 | 0 (0.0%) | 0.7 | 8.6 | 72.2 | 181.2 |
| getEvents/firehose | 1.5 | 213 | 0 (0.0%) | 0.7 | 5.6 | 23.2 | 141.4 |
| getEvents/head-poll | 36 | 5401 | 0 (0.0%) | 0.7 | 7.3 | 98.8 | 221.1 |
| getEvents/tail-poll | 9 | 1363 | 5 (0.4%) | 0.7 | 4.7 | 628.7 | 10002.4 |
| getEvents/transfer-watcher | 6 | 948 | 0 (0.0%) | 7.8 | 14.4 | 48.9 | 109.8 |
✅ Go endpoint benchmarks — verdict: ok
Baseline v28.0.1 (273f19e4fcb1) vs candidate 5845cd221109 — -benchmem -count=10, both refs sequentially on one box; rpcv2 excluded.
benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │
│ sec/op │
GetLedgerRange-8 956.4n ± 1%
BatchGetLedgers-8 745.6µ ± 2%
geomean 26.70µ
│ baseline.txt │
│ B/op │
GetLedgerRange-8 16.00 ± 0%
BatchGetLedgers-8 520.8Ki ± 0%
geomean 2.853Ki
│ baseline.txt │
│ allocs/op │
GetLedgerRange-8 4.000 ± 0%
BatchGetLedgers-8 2.314k ± 0%
geomean 96.21
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
│ baseline.txt │
│ sec/op │
ComputeFeeDistribution/computeFeeDistribution-8 9.266µ ± 0%
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 203.7µ ± 2%
geomean 43.45µ
│ baseline.txt │
│ B/op │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0%
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
│ baseline.txt │
│ allocs/op │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0%
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetEventsTopicFilters-8 2.535m ± 1% 2.536m ± 1% ~ (p=0.971 n=10)
GetEvents-8 147.3µ ± ∞ ¹ 148.2µ ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.271m ± ∞ ¹ 1.087m ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 7.618µ ± 0% 7.622µ ± 0% ~ (p=0.403 n=10)
JSONTransactions/XDR_format-8 2.529µ ± 1% 2.567µ ± 0% +1.50% (p=0.000 n=10)
GetProtocolVersion-8 41.17µ ± ∞ ¹ 40.53µ ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 84.97µ 82.87µ -2.47%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetEventsTopicFilters-8 190.1Ki ± 0% 189.8Ki ± 0% ~ (p=0.280 n=10)
GetEvents-8 43.16Ki ± ∞ ¹ 43.16Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.369Mi ± ∞ ¹ 1.043Mi ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 2.000Ki ± 0% 2.000Ki ± 0% ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8 1.336Ki ± 0% 1.336Ki ± 0% ~ (p=1.000 n=10) ³
GetProtocolVersion-8 7.907Ki ± ∞ ¹ 7.452Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 24.98Ki 23.63Ki -5.40%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetEventsTopicFilters-8 1.518k ± 0% 1.516k ± 0% ~ (p=0.272 n=10)
GetEvents-8 621.0 ± ∞ ¹ 621.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 4.723k ± ∞ ¹ 4.122k ± ∞ ¹ ~ (p=1.000 n=1) ³
JSONTransactions/JSON_format-8 25.00 ± 0% 25.00 ± 0% ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8 23.00 ± 0% 23.00 ± 0% ~ (p=1.000 n=10) ²
GetProtocolVersion-8 122.0 ± ∞ ¹ 120.0 ± ∞ ¹ ~ (p=1.000 n=1) ³
geomean 260.5 253.9 -2.53%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ need >= 4 samples to detect a difference at alpha level 0.05
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetPreflight-8 792.3µ ± 0% 789.5µ ± 1% -0.36% (p=0.043 n=10)
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetPreflight-8 52.10Ki ± 0% 52.10Ki ± 0% ~ (p=0.337 n=10)
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetPreflight-8 290.0 ± 0% 290.0 ± 0% ~ (p=1.000 n=10)
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/rpcv1/sqlitedb
│ candidate.txt │
│ sec/op │
GetLedgerRange-8 941.0n ± 1%
BatchGetLedgers-8 723.2µ ± 2%
geomean 26.09µ
│ candidate.txt │
│ B/op │
GetLedgerRange-8 16.00 ± 0%
BatchGetLedgers-8 476.8Ki ± 0%
geomean 2.730Ki
│ candidate.txt │
│ allocs/op │
GetLedgerRange-8 4.000 ± 0%
BatchGetLedgers-8 3.120k ± 0%
geomean 111.7
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/store
│ candidate.txt │
│ sec/op │
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 203.0µ ± 1%
ComputeFeeDistribution/ComputeFeeDistribution-8 9.406µ ± 0%
geomean 43.69µ
│ candidate.txt │
│ B/op │
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0%
ComputeFeeDistribution/ComputeFeeDistribution-8 0.000 ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
│ candidate.txt │
│ allocs/op │
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0%
ComputeFeeDistribution/ComputeFeeDistribution-8 0.000 ± 0%
geomean ¹
¹ summaries must be >0 to compute geomean
Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/34909144389/go-bench/): baseline.txt, benchstat.txt, candidate.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This branch removes all XDR views integration so that we may gather experimental results to quantify the improvements made through the work done in #732.
Known limitations
N/A