🚢 The Bigger the Cluster, the Harder it Falls
💡 Fact
- We allow between 3-200 brokers.
200 sounds like a bit too much - it should probably be smaller.
The main issue which requires us to allow a lot of brokers is the storage space. Since we don't support JBOD, allow only a max of 16 TiB per broker and require 50% free space - we can only store up to 8 TiB per broker, which means 2.6 TiB of produce traffic per broker. With 100 brokers - that's 260 TiB. As an example - 500 MiB/s throughput at 7 day retention produces that 288.39 TiB of data - so it limits the calculator's solutions quite a lot.
Once we fix the free space issue, it would allow us to handle more of these workloads with less brokers. The issue needs revisiting with more data and detail.