Enhance RedisItemReader/Writer performance using Redis pipelining #4939
Description
RedisItemReader and RedisItemWriter currently process items one by one, which leads to excessive network I/O and causes performance bottlenecks in high-volume jobs.
I propose using Redis pipelining, which bundles multiple commands into a single round-trip.
Below are the results of the Redis benchmark tests on my iMac.
(Environment = Redis cluster via Testcontainers on a local network, chunk size = 1,000)
| Task | 1k items | 10k items | 100k items | 1M items |
|---|---|---|---|---|
| read | 0.845 s | 8.354 s | 80.610 s | 1218.773 s |
| read (Pipeline) | 0.204 s | 1.244 s | 8.733 s | 97.682 s |
| write | 0.858 s | 8.598 s | 79.786 s | 1047.780 s |
| write (Pipeline) | 0.674 s | 0.725 s | 1.726 s | 16.653 s |
| delete | 0.846 s | 8.176 s | 79.777 s | 855.710 s |
| delete (Pipeline) | 0.100 s | 0.160 s | 1.379 s | 13.524 s |
According to the official Redis documentation, pipelining can deliver roughly a 10× performance enhancement, and my benchmarks showed similar results.
For the curious, I’ve put my benchmark source code on my GitHub branch : https://github.com/noojung/spring-batch/tree/redis-benchmark