Indeed, we had only reported the full training+eval loop runtime for the algorithm but not the inference time specifically before.
Let's get into more details.
To classify $N_\text{qy}$ examples given $N_\text{ctx}$ points, TabPFN would contruct a tensor of size ($L=N_\text{ctx}+N_\text{qy}, B=1, d$) with appropriate masking.
As we do not share the context across queries, with TabPFN kNN we need to have a specific context for each point and as such our input tensor is size ($L=N_\text{ctx}+1, B = N_\text{qy}, d$) where the $+1$ is the query point to classify in each batch dimension.
For instance, for 512 query points and 1000 context size/neighbors, the inference speed of TabPFN would be about 0.01s. A naive TabPFN-kNN is about 1s. However using bf16 precision we can lower it to about 0.4s (0.008s for TabPFN).
If we used 128 queries, the number would be 0.008s (TabPFN) vs 0.1s (TabPFN-kNN).
So as we observe, we are slower than TabPFN.
In the 71 datasets in the attached pdf on which the deep learning baselines were able to run according to the Tabzilla design, the largest of the 95 datasets are typically excluded (this is because Tabzilla allocate a time limit for each algorithm and dataset, as such many --slow-- DL baselines do not run on the largest datasets in the given time budget). This is why TabPFN-kNN is really fast. On fig 12, including all datasets the average speed is about 10x slower but still fast considering there is no training.
The fact that TabPFN-kNN is slower (and takes a lot more memory, especially when having to perform backprop as we do when performing fine-tuning end-to-end with retrieval) is why we introduced the approximate NN context sharing for fine-tuning.
All in all, TabPFN-kNN is not the algorithm with the fastest inference speed, TabPFN is much faster and so is XGBoost etc...
However we believe this algorithm still has very important strengths: it scales extremely well with the dataset size (compared to TabPFN whose performance degrades strongly), so in the case of production ML models which have access to a very large dataset but only have to classify new queries at a lower rate, it is a very well suited algorithm.
Furthermore, indexing on large datasets is usually much faster than retraining any ML algorithm, as such it can adapt very fast to changes in data distribution (such as covid for instance) without needing expensive retraining.
Note that the evaluation runtime was never the bottleneck in our research, so this is not what we focused on, but there are other simple ways one could improve upon it. For example, we can use the clustering capabilities of faiss to split the training data into a fixed number of contexts. This way we can ensure many queries will share the same context and harness the speed of TabPFN. While it may degrade the accuracy of the NN search, we think it can be an interesting trade-off if inference speed is an issue.
Thank you for the interesting comment, we will include numbers or a figure specifically concerning the inference speed for clarity.
Have you received the anonymized code link from the AC?