> Opacus supports (distributed) Poisson sampling without the need to load the entire dataset (e.g., simply by indexing). I suggest the authors do their due diligence and check the implementation at https://opacus.ai/api/_modules/opacus/utils/uniform_sampler.html
Thanks to the reviewer for the pointer. We apologize that our original text in lines 47-49 could be misinterpreted as “Opacus solution *requires* loading the dataset into the memory”. We meant to say that “loading into memory” is one way to facilitate this feature for small datasets. We will revise the text to make it more precise.
To avoid any further misunderstanding between us and the reviewer on this matter, we summarize our view of the Opacus solution here: Opacus Poisson Subsampling works by using a unique identifier to index each example, and sampling the indices. They also provide a distributed sampler where each worker samples from a random subset of indices. This approach relies on a DataLoader that can provide efficient random access --- given an arbitrary index, read the corresponding data point efficiently. If random access is slow, the data pipeline will be slow, even though the (index) sampler is not the bottleneck.
For small datasets, this can be easily supported by loading the data into memory. For large datasets that do not fit in the memory, various technical challenges need to be addressed to make it work. For example, many formats for raw data (e.g. `csv`, typically used for tabular or text data) or serialized data (e.g. `tfrecords` files, commonly used by the Tensorflow Datasets Catalog) do _not_ naturally support random indexing; moreover, random access can be much slower than sequential bulk reading in cases such as when the dataset is stored in a distributed file system. Various workarounds do exist depending on the resources and constraints, for example, by storing each example as an individual file in a fast SSD attached to the trainer machine. But those solutions usually need to be tailored to specific hardware and cluster infrastructures. On the contrary, we provide a **generic solution** that works for a large range of scales of dataset sizes and with **minimal assumption / requirement** on the underlying IO infrastructure.
We agree that an empirical comparison of different technical solutions could be interesting but it is quite difficult to get right and useful as everyone has different configurations of (distributed) file systems, file format of preference, cluster configuration with communication channels of different properties, etc. Therefore it is quite far from the scope of our paper.
Finally, regardless of the underlying technical solution, please note that ours is the _first large-scale_ DP training experiment with Poisson subsampling, comparing its feasibility with other solutions from multiple angles, including accounting, efficiency, and model utility.
> truncated Poisson sampling distorts the probability of each sample being included in a batch
It is true that due to truncation in Poisson subsampling, the probability that an example lands in a batch is reduced slightly. This is handled in the privacy analysis (Theorem 3.3).
Please let us know if there are any further questions.