feat: Introduce FastExcel Benchmark Performance Testing Module - #575
feat: Introduce FastExcel Benchmark Performance Testing Module#575GOODBOY008 wants to merge 4 commits into
Conversation
GOODBOY008
changed the title
|
CI Benchmark Run Completed Successfully: https://github.com/GOODBOY008/fastexcel/actions/runs/17709908635 To view the benchmark reports:
There are a few issues to address:
|
|
I'm really excited about this PR. However, it's quite large, so the code review will take some time. Also, no offense intended, but I'd like to ask: Did you use AI-generated code in this PR? |
@psxjoy Yes, some parts (like the comparison report, memory profiler logic, and quickstart scripts) were AI-assisted.AI is quite effective in these scenarios, I’ve verified them to make sure they work correctly. I noticed the artifact wasn’t accessible, so I’ve uploaded the results for your review. |
psxjoy
added
PR: developing
|
Hi, @GOODBOY008 Regarding this PR, I still have some questions:
Please refer to the above suggestions and make appropriate modifications to the PR content. After that, we will vote on this PR together with other reviewers ASAP. |
|
Hi @delei For the first point, I understand the concern about the PR size — my intention was to split the work into stages, so this submission might look a bit large. Regarding the second and third points: I’m fine with keeping only the JMH core classes for now, but I’d like to highlight the above considerations. |
5a95c2d to
1be72cb
Compare
|
@delei PTAL |
1be72cb to
1824158
Compare
delei
added
PR: require-multiple-approvals
1824158 to
a828caf
Compare
✅ Deploy Preview for fesod ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
a828caf to
3a2c467
Compare
3a2c467 to
4ede42b
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a comprehensive JMH-based benchmark module for the FastExcel library, enabling performance testing and comparisons with Apache POI across various operations (read, write, fill) and dataset sizes. The module includes memory profiling capabilities, test data generation utilities, and comparison benchmarks to validate FastExcel's performance claims.
Changes:
- New
fesod-benchmarkmodule with complete JMH integration and Maven configuration - Benchmark suites for read, write, and fill operations across multiple dataset sizes and file formats
- Memory profiling utilities with GC tracking and detailed statistics
- Comparison benchmarks between FastExcel and Apache POI
- Comprehensive test data generation with configurable characteristics
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 20 comments.
Show a summary per file
| File | Description |
|---|---|
| pom.xml | Added fesod-benchmark module to parent POM |
| fesod-benchmark/pom.xml | New Maven configuration with JMH dependencies and shade plugin |
| fesod-benchmark/benchmark.md | Documentation for running and interpreting benchmarks |
| MemoryProfiler.java | Utility for real-time memory profiling with GC tracking |
| DataGenerator.java | Test data generation with multiple data types and characteristics |
| BenchmarkFileUtil.java | File management utilities for benchmark operations |
| BenchmarkData.java | Data model with 20 fields covering various Excel data types |
| BenchmarkConfiguration.java | Configuration enums for dataset sizes and file formats |
| AbstractBenchmark.java | Base class providing common benchmark functionality |
| WriteBenchmark.java | Write operation benchmarks for different sizes and scenarios |
| ReadBenchmark.java | Read operation benchmarks with multiple listener patterns |
| FillBenchmark.java | Template fill operation benchmarks |
| FastExcelVsPoiBenchmark.java | Comparison benchmarks between FastExcel and Apache POI |
| ComparisonBenchmarkRunner.java | Runner for executing comparison benchmarks |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Test files for different sizes and formats | ||
| private String xlsxSmallFile; | ||
| private String xlsxMediumFile; | ||
| private String xlsEXTRA_LARGEFile; |
There was a problem hiding this comment.
Variable name xlsEXTRA_LARGEFile uses inconsistent capitalization mixing underscores with camelCase. Should be xlsxLargeFile to match the naming pattern of other variables like xlsxSmallFile and xlsxMediumFile.
| private String xlsEXTRA_LARGEFile; | |
| private String xlsxLargeFile; |
| } | ||
|
|
||
| @Benchmark | ||
| public void readXlsEXTRA_LARGE(Blackhole blackhole) throws Exception { |
There was a problem hiding this comment.
Method name readXlsEXTRA_LARGE uses inconsistent capitalization with underscores and uppercase. Should be readXlsxLarge to follow Java naming conventions and match the pattern of other methods like readXlsxSmall and readXlsxMedium.
| public void readXlsEXTRA_LARGE(Blackhole blackhole) throws Exception { | |
| public void readXlsxLarge(Blackhole blackhole) throws Exception { |
|
|
||
| // Stream reading benchmarks | ||
| @Benchmark | ||
| public void readXlsEXTRA_LARGEWithStreaming(Blackhole blackhole) throws Exception { |
There was a problem hiding this comment.
Method name readXlsEXTRA_LARGEWithStreaming uses inconsistent capitalization. Should be readXlsxLargeWithStreaming to follow Java naming conventions.
| public void readXlsEXTRA_LARGEWithStreaming(Blackhole blackhole) throws Exception { | |
| public void readXlsxLargeWithStreaming(Blackhole blackhole) throws Exception { |
|
|
||
| // Different listener types benchmarks | ||
| @Benchmark | ||
| public void readXlsEXTRA_LARGECountingOnly(Blackhole blackhole) throws Exception { |
There was a problem hiding this comment.
Method name readXlsEXTRA_LARGECountingOnly uses inconsistent capitalization. Should be readXlsxLargeCountingOnly to follow Java naming conventions.
| public void readXlsEXTRA_LARGECountingOnly(Blackhole blackhole) throws Exception { | |
| public void readXlsxLargeCountingOnly(Blackhole blackhole) throws Exception { |
| } | ||
|
|
||
| @Benchmark | ||
| public void readXlsEXTRA_LARGECollecting(Blackhole blackhole) throws Exception { |
There was a problem hiding this comment.
Method name readXlsEXTRA_LARGECollecting uses inconsistent capitalization. Should be readXlsxLargeCollecting to follow Java naming conventions.
| public void readXlsEXTRA_LARGECollecting(Blackhole blackhole) throws Exception { | |
| public void readXlsxLargeCollecting(Blackhole blackhole) throws Exception { |
| "Generated {} rows in {} ms ({} rows/sec)", | ||
| rowCount, | ||
| duration, | ||
| duration > 0 ? (rowCount * 1000 / duration) : "N/A"); |
There was a problem hiding this comment.
Potential overflow in int multiplication before it is converted to long by use in a numeric context.
| duration > 0 ? (rowCount * 1000 / duration) : "N/A"); | |
| duration > 0 ? (rowCount * 1000L / duration) : "N/A"); |
| protected void setupBenchmark() throws Exception { | ||
| // Custom setup logic if needed | ||
| } | ||
|
|
There was a problem hiding this comment.
This method overrides AbstractBenchmark.tearDownBenchmark; it is advisable to add an Override annotation.
| @Override |
|
|
||
| protected void setupBenchmark() throws Exception { | ||
| // Custom setup logic if needed | ||
| } | ||
|
|
There was a problem hiding this comment.
This method overrides AbstractBenchmark.setupBenchmark; it is advisable to add an Override annotation.
| protected void setupBenchmark() throws Exception { | |
| // Custom setup logic if needed | |
| } | |
| @Override | |
| protected void setupBenchmark() throws Exception { | |
| // Custom setup logic if needed | |
| } | |
| @Override |
|
|
||
| System.out.printf("Setup comparison benchmark: %s format, %d rows%n", fileFormat, rowCount); | ||
| } | ||
|
|
There was a problem hiding this comment.
This method overrides AbstractBenchmark.tearDownTrial; it is advisable to add an Override annotation.
| @Override |
| private List<BenchmarkData> testDataList; | ||
| private MemoryProfiler memoryProfiler; | ||
| private List<ComparisonResult> localResults = new ArrayList<>(); | ||
|
|
There was a problem hiding this comment.
This method overrides AbstractBenchmark.setupTrial; it is advisable to add an Override annotation.
| @Override |
4ede42b to
c0530e2
Compare
… comparison tests - Introduce fesod-benchmark module with detailed README guide on usage and best practices - Configure Maven profiles for running full benchmarks and quick CI smoke tests - Implement ComparisonBenchmarkRunner to run Fesod vs Apache POI benchmarks with JSON result output - Add FastExcelVsPoiBenchmark with improved benchmark methods for write, read, and streaming operations - Apply JMH best practices: fixed heap size, no manual timing, preloaded data, fair comparisons - Remove memory profiling logic and simplify benchmarks to focus on throughput and stability - Update Apache POI benchmarks to write and read matching complex test data columns - Refactor benchmark result handling to store minimal metrics, removing memory stats - Enhance benchmark documentation with usage instructions, suite descriptions, and dataset sizes
…ollection - Removed unused Alibaba FastJSON2 imports from FastExcelVsPoiBenchmark - Removed session ID and file-based result collection mechanism - Simplified JVM arguments configuration for Fork annotations in benchmark classes - Removed unused ComparisonResult class and related JSON writing methods - Cleaned up unnecessary processing logic in ReadBenchmark row handler - Updated Maven plugin configuration to include .* argument for dataset matching - Deleted comprehensive benchmark documentation that was no longer relevant
fe7819d to
28afc74
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (15)
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/comparison/FastExcelVsPoiBenchmark.java:290
ExcelReaderimplementsCloseable(close() calls finish()). IfreadAll()throws,finish()is skipped and the underlying stream/cache may not be released. Use try-with-resources so the reader is always closed.
try {
ExcelReader excelReader = EasyExcel.read(testFile, BenchmarkData.class, new ReadListener<BenchmarkData>() {
@Override
public void invoke(BenchmarkData data, AnalysisContext context) {
processedRows.incrementAndGet();
blackhole.consume(data);
}
@Override
public void doAfterAllAnalysed(AnalysisContext context) {
// Processing complete
}
})
.build();
excelReader.readAll();
excelReader.finish();
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/comparison/FastExcelVsPoiBenchmark.java:368
- Same resource-safety issue as
benchmarkFesodRead: ifreadAll()throws,finish()is skipped and the reader isn't closed. Use try-with-resources so the reader is always closed.
try {
ExcelReader excelReader = EasyExcel.read(testFile, BenchmarkData.class, new ReadListener<BenchmarkData>() {
@Override
public void invoke(BenchmarkData data, AnalysisContext context) {
batch.add(data);
processedRows.incrementAndGet();
if (batch.size() >= batchSize) {
blackhole.consume(new ArrayList<>(batch));
batch.clear();
}
}
@Override
public void doAfterAllAnalysed(AnalysisContext context) {
if (!batch.isEmpty()) {
blackhole.consume(batch);
batch.clear();
}
}
})
.build();
excelReader.readAll();
excelReader.finish();
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:306
- The vertical fill template references
{.priority}, butBenchmarkDatahas nopriorityproperty. This will either leave the cell blank or fail template resolution, so the benchmark won't reflect a real fill scenario.
Map<String, Object> row1 = new HashMap<>();
row1.put("A", "Dynamic Fill Test");
row1.put("B", "Status");
row1.put("C", "Priority");
Map<String, Object> row2 = new HashMap<>();
row2.put("A", "{.id}");
row2.put("B", "{.status}");
row2.put("C", "{.priority}");
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:180
createSimpleTemplate()says it creates placeholder rows, but it writes only literal strings (no{...}placeholders). As a result,fillSimpleMap/fillSingleObjectwon't actually exercise template variable replacement, making the benchmark results misleading.
// Create a simple template with placeholder rows
Map<String, Object> row1 = new HashMap<>();
row1.put("name", "Simple Fill Test");
row1.put("date", "2023-01-01");
row1.put("version", "1.0");
Map<String, Object> row2 = new HashMap<>();
row2.put("description", "This is a simple fill test");
row2.put("author", "Test Author");
row2.put("status", "Active");
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:335
fillMultipleLists()fillsFillWrapper("data1", ...)andFillWrapper("data2", ...), but the generatedmultiListTemplateFileonly contains{date}/{version}placeholders and no list placeholders. The list fills will be no-ops, so this benchmark won't measure multi-list filling.
private String createMultiListTemplate() {
String templatePath = BenchmarkFileUtil.getTempFilePath(
BenchmarkConfiguration.FileFormat.XLSX, BenchmarkConfiguration.DatasetSize.LARGE, "MultiListTemplate");
// Create multi-list template
Map<String, Object> row1 = new HashMap<>();
row1.put("Report", "Performance Report");
row1.put("Date", "{date}");
row1.put("Version", "{version}");
Map<String, Object> row2 = new HashMap<>();
row2.put("Metric", "Value");
row2.put("Status", "Threshold");
row2.put("Notes", "Comments");
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/utils/BenchmarkFileUtil.java:91
Files.walk(...)returns a Stream that must be closed; the current implementation never closes it, which can leak file handles (especially on Windows) and cause intermittent cleanup failures during repeated benchmark runs.
Files.walk(testDataPath)
.filter(path -> path.getFileName().toString().startsWith("temp_"))
.forEach(path -> {
try {
Files.deleteIfExists(path);
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/core/AbstractBenchmark.java:187
Files.walk(...)returns a Stream that needs to be closed. Not closing it can leak file handles during repeated benchmark runs and make temp-file cleanup flaky.
Files.walk(outputPath)
.filter(path -> path.getFileName().toString().startsWith("temp_"))
.forEach(path -> {
try {
Files.deleteIfExists(path);
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/comparison/FastExcelVsPoiBenchmark.java:170
ExcelWriterimplementsCloseable(close() calls finish()). In this benchmark, ifwrite()orfinish()throws, the writer won't be closed, which can leak temp files/streams and skew benchmark stability. Prefer try-with-resources and rely onclose()for cleanup.
This issue also appears in the following locations of the same file:
- line 273
- line 343
try {
ExcelWriter excelWriter =
EasyExcel.write(outputFile, BenchmarkData.class).build();
WriteSheet writeSheet = EasyExcel.writerSheet("TestData").build();
excelWriter.write(testDataList, writeSheet);
excelWriter.finish();
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/utils/DataGenerator.java:258
DataGeneratoruses a fixed Random seed for reproducibility, but the generated dates depend onLocalDate.now()/LocalDateTime.now(), so the dataset will change over time. That makes benchmark results harder to compare across runs/days even with the same seed.
private LocalDate generateRandomDate() {
LocalDate now = LocalDate.now();
LocalDate fiveYearsAgo = now.minusYears(5);
long daysBetween = java.time.temporal.ChronoUnit.DAYS.between(fiveYearsAgo, now);
long randomDays = Math.floorMod(random.nextLong(), daysBetween);
return fiveYearsAgo.plusDays(randomDays);
}
/**
* Generate random datetime within the last year
*/
private LocalDateTime generateRandomDateTime() {
LocalDateTime now = LocalDateTime.now();
LocalDateTime oneYearAgo = now.minusYears(1);
long secondsBetween = java.time.temporal.ChronoUnit.SECONDS.between(oneYearAgo, now);
long randomSeconds = Math.floorMod(random.nextLong(), secondsBetween);
return oneYearAgo.plusSeconds(randomSeconds);
fesod-benchmark/pom.xml:220
- The
benchmark-testprofile claims to be a quick smoke test, but passing.*will run all benchmarks in the module (including the non-parameterized ones that generate large/extra-large datasets). This makes the CI/smoke profile much heavier than intended.
<arguments>
<argument>-f</argument>
<argument>1</argument>
<argument>-i</argument>
<argument>1</argument>
<argument>-wi</argument>
<argument>0</argument>
<argument>-p</argument>
<argument>datasetSize=SMALL</argument>
<argument>-p</argument>
<argument>fileFormat=XLSX</argument>
<argument>.*</argument>
</arguments>
fesod-benchmark/benchmark.md:88
benchmark.mdsaysEXTRA_LARGEis for “comparison benchmarks only”, but the operations benchmarks (WriteBenchmark,ReadBenchmark) also useDatasetSize.EXTRA_LARGE. This is confusing for users trying to choose an appropriate benchmark size.
| `SMALL` | 1,000 | Quick development feedback |
| `MEDIUM` | 10,000 | Standard CI benchmarks |
| `LARGE` | 100,000 | Performance analysis |
| `EXTRA_LARGE` | 1,000,000 | Stress testing (comparison benchmarks only) |
pom.xml:82
- PR title/description talk about “FastExcel” and link to the
fast-excel/fastexcelrepo, but the actual change adds afesod-benchmarkmodule underorg.apache.fesod.*in this repository. Please align the PR metadata (title/description) with the Fesod codebase to avoid confusion for reviewers and release notes.
<module>fesod-shaded</module>
<module>fesod-examples</module>
<module>fesod-sheet</module>
<module>fesod-benchmark</module>
</modules>
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/WriteBenchmark.java:48
- The module and packages are all
org.apache.fesod.*, but this class-level Javadoc still refers to “FastExcel”. This is confusing in the Fesod codebase and in generated benchmark reports.
/**
* Comprehensive benchmarks for FastExcel write operations
*/
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/ReadBenchmark.java:48
- The module and packages are all
org.apache.fesod.*, but this class-level Javadoc still refers to “FastExcel”. This is confusing in the Fesod codebase and in generated benchmark reports.
/**
* Comprehensive benchmarks for FastExcel read operations
*/
fesod-benchmark/src/main/java/org/apache/fesod/sheet/benchmark/operations/FillBenchmark.java:53
- The module and packages are all
org.apache.fesod.*, but this class-level Javadoc still refers to “FastExcel”. This is confusing in the Fesod codebase and in generated benchmark reports.
This issue also appears in the following locations of the same file:
- line 171
- line 298
- line 322
/**
* Comprehensive benchmarks for FastExcel fill operations
*/
Overview
This PR introduces a comprehensive benchmark performance testing module for FastExcel, implementing the proposal outlined in #572.
Benchmark Results Available
CI Benchmark Run Completed Successfully: https://github.com/GOODBOY008/fastexcel/actions/runs/17709908635
Benchmark Artifacts: The workflow generated comprehensive benchmark reports available in the artifacts:
To view the benchmark reports:
benchmark-resultsartifact from: https://github.com/GOODBOY008/fastexcel/actions/runs/17709908635benchmark-reports/benchmark-comparison.htmlin your browserWhat's Changed
New Module: fastexcel-benchmark
• JMH Integration: Complete Maven configuration with industry-standard Java microbenchmarking framework
• Comprehensive Test Suites:
◦ Comparison benchmarks (FastExcel vs Apache POI)
◦ Memory efficiency specialized tests
◦ Streaming operation performance tests
◦ Microbenchmarks for core components
• Automated Execution: Multi-profile support with configurable dataset sizes and memory settings
• Advanced Features:
◦ Interactive CLI with scenario management
◦ Real-time memory profiling with GC tracking
◦ HTML visualization reports and JSON data export
◦ Performance trend analysis and regression detection
Key Components
Core Framework (
cn.idev.excel.benchmark.core)◦ Abstract benchmark base classes
◦ Configuration management
◦ Memory profiler integration
Test Scenarios (
cn.idev.excel.benchmark.*)◦ Read/Write operation benchmarks
◦ Fill operation performance tests
◦ Streaming benchmarks for large datasets
◦ Memory efficiency analysis
Comparison Benchmarks (
cn.idev.excel.benchmark.comparison)◦ Direct FastExcel vs Apache POI performance comparison
◦ Multi-dimensional analysis (throughput, latency, memory)
Utilities (
cn.idev.excel.benchmark.utils)◦ Test data generation
◦ File management utilities
◦ Reporting and visualization
Automated Scripts (
scripts/benchmark-runner.sh)◦ Profile-based execution (quick/standard/comprehensive)
◦ Configurable parameters and output formats
◦ Regression analysis automation
GitHub Actions Integration
• Workflow (
.github/workflows/benchmark.yml)◦ Manual trigger with
workflow_dispatchfor on-demand benchmarking◦ Java 11 setup with proper classpath resolution
◦ Automated artifact upload for benchmark results
◦ Fixed JMH forking issues for reliable results
Test Scenarios Coverage
• Data Scales: SMALL(1K) → MEDIUM(10K) → LARGE(100K) → EXTRA_LARGE(1M+)
• File Formats: XLSX
• Operation Types: Read, Write, Fill, Streaming
• Memory Analysis: Real-time monitoring, GC pressure analysis, allocation patterns
Benefits
Closes #572