Diffusion Models are Evolutionary Algorithms

We thank you for your time and effort to review our manuscript and for your thoughtful comments on clarity, future directions, and high-dimensional statistics. We have revised the theoretical section to enhance its accessibility for a broader audience. We will update the paper soon. In the discussion section, we further explored the potential and limitations of our method. Specifically, regarding the challenge of open-ended evolution, several straightforward approaches can be directly adapted from the diffusion model domain. For example, the work by Yilun Xu et al., 2023 (https://arxiv.org/abs/2306.14878) introduces additional Gaussian noise to diffusion models, allowing their time steps to move backward. Their experiments demonstrate that this “restart” trick can improve quality. Based on the equivalence we established, we anticipate that such a method could facilitate open-ended evolution while also enhancing the quality of evolutionary results. For high-dimensional statistics, we extended our benchmarks to higher-dimensional spaces and provided more detailed statistical analyses across all benchmark experiments. Results on diversity and fitness are presented in this table:[ link](https://postimg.cc/ctYHbJ6M). For statistical significance, a detailed table including the standard deviation of fitness and diversity is available here:[ link](https://postimg.cc/N9Rvh27M). These analyses demonstrate that our superior performance in fitness and diversity is statistically significant. Additionally, we compared our method to the quality-diversity search method, specifically the MAP-Elite method, using the QD-score—a commonly used metric for assessing the balance between quality and diversity (see this paper https://arxiv.org/abs/2303.12803). The results show that our method, particularly the latent diffusion evolution, performs well across all tested fitness functions. Moreover, our method demonstrates significant improvements in QD-scores at high dimensions. See the detailed results in this table:[ link](https://postimg.cc/3dfkTr0T). **Addressing the reviewer’s concern about validation in real-world applications:** We have extended our reinforcement learning experiments to encompass a broader range of tasks, demonstrating that our method effectively identifies solutions across these RL scenarios. Additionally, we have incorporated statistical tests to strengthen the validation of our results. Please refer to the tables above. We will update the RL tasks in the revised paper soon.

Thank you for such a detailed and thoughtful response. **Evolutionary Dynamics and Mathematical Foundations** Regarding the first concern about mathematical formulation, it's crucial to understand that we are not developing a new algorithm. Rather, our algorithm can be considered as the solution of an equivalence equation: we propose that $A(x) = B(x)$, by identifying the inherent similarities between $A$ and $B$. Here, $A$ is the diffusion model, and $B$ is the evolutionary algorithm. We then demonstrate our solution as the proposed method and establish its practical utility. Within this mathematical framework, we face natural constraints on incorporating arbitrary biological phenomenology. This is analogous to solving a simple equation like $x + 2 = 4$, where one cannot arbitrarily impose additional constraints, such as requiring $x$ to be negative. This mathematical perspective also addresses the concerns about conditional independence in Bayesian formulations. We're not explicitly modeling evolutionary process behavior, but rather deriving solutions from our equivalence assumptions and conditional independence. Nonetheless, our method still inherently incorporates several biological features without explicit design or imposition, **including the points you proposed**. Beyond the examples mentioned in the paper, such as reproductive isolation and natural selection, the Diffusion Evolution method also naturally encompasses genetic drift and gene recombination: 1. **Genetic Drift:** As suggested by the “neutral theory of molecular evolution,” many mutation dimensions do not impact fitness. This is precisely captured by learning or sampling the distribution—these neutral mutations broaden the distribution, facilitating genetic drift. 2. **Gene Recombination:** This feature is more explicitly included in Diffusion Evolution. In Equation 9, we estimate the optimal parameters as a weighted average of their neighbors. While biological recombination typically involves only two individuals per generation, our approach can be interpreted as a coarse-grained recombination over longer timescales. In the meantime, algorithms like CMA-ES are even doing gene recombination more implicitly and are still considered as evolutionary algorithms. We will add further discussions on these aspects in the revised paper and appreciate your feedback, which highlighted areas requiring clarification. **Choice of the density mapping function** Regarding the choice of the density function, it remains task-dependent. While evolutionary algorithms operate on nondifferentiable fitness functions, ensuring smooth probability functions and reasonable gradient values is advantageous, particularly for avoiding excessively small or large gradients. To address this, we are performing a grid search on benchmark functions. We modified the original function to a new form with a control temperature parameter. Higher temperatures smooth the density function, while lower temperatures focus on high-fitness parameters. We first calculate the fitness scale of the functions, defined as the maximum fitness difference around the optimal points in the fitness function. The temperature is then set to $T\times \text{scale}$, where $T$ is a parameter used to control the smoothness of the $g$ function. This rescaling enables the comparison of different fitness functions. Our observations reveal two general trends: 1. Higher temperatures lead the algorithm to sample more low-fitness parameters, resulting in a lower average fitness (see [[figure](https://postimg.cc/3dt8P3bH)]). 2. Increasing the temperature enhances result diversity, as reflected in both entropy (see [[figure](https://postimg.cc/V5244w4K)]) and quality-diversity scores—a metric quantifying the balance between quality and diversity (see definition at https://arxiv.org/abs/2303.12803). Our results are illustrated in [[figure](https://postimg.cc/Z0n61Q0X)]. However, this effect plateaus quickly at approximately $T=1$. Based on these findings, we recommend first estimating the fitness scale and using $T=1$ for initial experiments. These guidelines will be incorporated into the revised paper shortly.

**Statistical Analysis** Regarding point 2 in the weaknesses section, we appreciate your suggestion to include additional statistical analyses. While our primary objective was to establish the theoretical equivalence between diffusion and evolution processes—rather than emphasizing performance comparisons—our method demonstrates competitive performance even without specific optimization-focused design elements. In response to your recommendation, we have enhanced our evaluation in two ways. 1. First, we incorporated more detailed statistical analyses into the benchmark experiments. The low standard deviation of both fitness and diversity indicate that our superior performance compared to other methods is statistically significant and demonstrates stability. Please refer to the updated table 1 ([link](https://postimg.cc/ctYHbJ6M)) and the detailed table ([link](https://postimg.cc/N9Rvh27M)). 2. Second, we expanded our testing to include higher-dimensional fitness functions and comparisons with quality-diversity search approaches, which represent the cutting-edge method with impressive performance in evolutionary algorithms. Please also refer to the tables above. The results from these extended experiments further validate our method's capability to effectively balance exploration and exploitation, particularly when assessed using the quality-diversity score. See the comparison table at[ link](https://postimg.cc/3dfkTr0T). **Writing and grammar** For point 3 in the weakness section, we thank the reviewer for your time and effort. We carefully reviewed the paper and addressed grammar issues and other unclear points. **Distribution Transformations** Regarding point 4 in the weakness section, we agree that the concept of distribution is general and applies to various domains beyond evolution, such as prediction frameworks like the Gödel machine and general AI models like AIXI. However, this universality does not diminish the value of the distribution concept; rather, it provides numerous insights. Through the formulation of distribution transformation, we naturally connect fitness to probability density and obtain diversity. From the perspective of distribution, diversity implies distributions that are not narrow, such as a Dirac-delta function, or implies distributions with multiple peaks. Simultaneously, this concept ties directly to genetic drift, which you previously mentioned. Genetic drift is deeply related to neutral genes. In the framework of distribution, this corresponds to large regions in the parameter space with similar probability density. From a goal-oriented perspective, we aim to develop an evolutionary algorithm capable of finding multiple solutions, allowing for genetic drift, and modeling complex evolutionary processes. In such cases, viewing evolution as a distribution transformation proves to be the most effective approach. It is also worth noting that our connection between diffusion and evolution is not solely based on distribution transformations. Diffusion models inherently involve iterative refinement, sampling diverse solutions, and incorporating intrinsic random mutations. A fair comparison would reveal that approaches like semantic segmentation lack these characteristics.

**Extending Experiments** Regarding point 5 in the weakness section, we appreciate your feedback on the experiments. Initially, we chose simple experiments because our primary focus was on establishing the theoretical connection between diffusion and evolution. However, this does not imply that our method lacks competitiveness. In the revised version, we extended our multi-objective evolution experiments to high-dimensional spaces, shown in the updated table 1 ([link](https://postimg.cc/ctYHbJ6M)). Additionally, we compared our approach to MAP-Elite, a popular quality-diversity search algorithm with impressive performance specifically designed to balance quality and diversity. These experiments demonstrate that our method, particularly the latent method, outperforms traditional evolutionary algorithms (EAs) and quality-diversity search approaches across various tasks, including high-dimensional ones. The comparison with quality-diversity search further highlights that our method surpasses MAP-Elite in effectively balancing quality and diversity. See the attached table for details ([link](https://postimg.cc/3dfkTr0T)). Such strong performance does not come from manually designed or imposed features like sampling, diversity, or genetic drift (which are nonetheless naturally encompassed as a consequence of the equivalence we discovered), which highlights the significant potential of this equivalence when applied to various tasks. Furthermore, we extended our reinforcement learning experiments to a broader range of tasks, including Acrobot, Cart Pole, Mountain Car, Continuous Mountain Car, and Pendulums. To broaden the scope of our testing, we introduced a scaling factor $s$. Specifically, given a default fitness function $f(x)$ for a reinforcement learning task, we transformed it into $f_s(x) = f(sx)$. This approach allows us to test tasks with varying norms of parameters, hence testing different topology of parameter distribution spaces. The experimental results, summarized in this table ([link](https://postimg.cc/HV0kY659)), indicate that for all environments, Diffusion Evolution and Latent Diffusion Evolution consistently achieve the highest rewards across different scaling factors. Moreover, the Latent Diffusion model demonstrates greater stability with respect to the scaling factor, validating the effectiveness of incorporating latent methods. Based on the standard deviations, these performance improvements are statistically significant. These results suggest that our method can serve as a general optimization approach. The statistical significance analysis on benchmark functions is discussed in previous sections. **Choice of Evolutionary Strategies** Lastly, addressing the issue raised in the question section. While these methods were introduced a long time ago, they were selected for comparison based on careful reasoning. Methods like CMA-ES and PEPG optimize parameters by modeling them as Gaussian distributions, and our comparison aims to demonstrate that the ability to fit complex distributions is critical for achieving both quality and diversity. The OpenES method was chosen because it also estimates the moving direction based on neighbors. The key differences are: (1) how the direction is estimated, and (2) in our method, each individual makes its own estimation, whereas OpenES relies on a single shared estimation. This comparison is intended to illustrate how a solution derived from the equivalence between diffusion and evolution can outperform methods that were meticulously designed. We have attached the updated tables and figures and will upload the revised paper shortly.

Thank you for your thoughtful critique. We presented this paper in a theoretical framework, and one of our primary contributions lies in its theoretical framework specifically in demonstrating the mathematics of the equivalence between diffusion and evolution processes. Having rigorously established this equivalence, we can directly apply existing theoretical analyses of diffusion models - including convergence properties - without the need for additional theoretical investigation. This fundamental equivalence enables us to import techniques from diffusion model research, particularly accelerated sampling methods. These methods demonstrate the same acceleration in our evolutionary framework without requiring modifications to other components. The demonstrated equivalence serves as the cornerstone of our theoretical analysis, making such adaptations possible For the benchmark experiments, we selected CMA-ES from a distributional perspective. Since CMA-ES represents a distribution using a Gaussian, it lacks the capability to fit and optimize over complex fitness landscapes, such as those with multiple optima. This allowed us to highlight the flexibility of our method in addressing such challenges. We also agree that incorporating quality-diversity search can enhance the benchmark comparison. To this end, we integrated MAP-Elite into our benchmark and extended the fitness function to higher dimensions. The results showed that our method outperformed MAP-Elite. While maintaining comparable diversity (quantified using entropy) with MAP-Elite, our method achieved significantly higher fitness, demonstrating its ability to balance exploration and exploitation through unbiased sampling. We thank the reviewer for their suggestions to improve our presentation and better highlight our method's advantages over other evolutionary algorithms. We have expanded our evaluation by testing our approach on additional reinforcement learning tasks, which demonstrated its capability to discover effective controllers while maintaining low computational overhead. We included more experiments and statistics and showed that our method can outperform traditional methods on both quality and diversity. Especially in QD-score (https://arxiv.org/abs/2303.12803), we outperform all other methods on high dimensional space, including MAP-Elite, a popular quality-diversity search. Please refer to the [QD-score table](https://postimg.cc/3dfkTr0T) Regarding the question on alpha scheduling, we appreciate your inquiry. Generally, we observe similar behavior to that in the diffusion model domain. For instance, cosine alpha scheduling significantly impacts convergence speed in diffusion models, and we observe the same effect in diffusion evolution when applying cosine alpha scheduling. Our experimental results, presented in [figure](https://postimg.cc/PPXscxLX), compare fitness values across different total evolution times under various alpha scheduling strategies. Compared to the default DDPM alpha scheduling and linear decreasing alpha scheduling, cosine alpha scheduling outperforms the other methods when the number of evolution steps is below 100. While the performance difference becomes less significant with a higher number of steps, the primary motivation for adopting cosine alpha scheduling is to reduce the total number of evolution steps required. We are finalizing the analysis and will upload the paper soon.

Dear Reviewer, Thank you for your detailed feedback on our paper. We have conducted extensive additional experiments to address your concerns about our method's performance and theoretical foundations. Please check our revised PDF manuscript, changes highlighted with blue color. ## Comprehensive Statistical Analysis We have expanded our evaluation across various fitness functions and comparison methods. For performance and diversity metrics, we conducted two key experiments. In the fitness evaluation, our method demonstrated strong performance while maintaining diversity, even when tested in higher dimensions (4, 32, 256) as shown in Tables 1 and 3. Following your suggestion, we compared our approach with MAP-Elite using the QD-score metric. The results show our method outperforms MAP-Elite in high-dimensional spaces while remaining competitive in lower dimensions (Table 4). We've also broadened our reinforcement learning experiments to include Acrobot, Cart Pole, Mountain Car (both discrete and continuous), and Pendulum environments. Our method consistently outperforms CMA-ES across all tasks (Table 5) with robust success rates (Table 6). To test robustness, we introduced a scaling hyper-parameter that modifies the fitness landscape, and our method maintained consistent performance across these variations. ## Theoretical Connections Our new experiments provide stronger support for our theoretical framework. We observed interesting parallels between diffusion and evolution processes. The alpha scheduling behavior shows remarkable similarities in both contexts - the cosine scheduling, originally proposed for diffusion models, demonstrates similar benefits in our evolutionary context when total evolution time is limited (Figure 5). Additionally, the latent method's contribution to performance mirrors patterns seen in diffusion models, particularly in addressing high-dimensional challenges (Tables 1 and 6). Through grid search experiments (Figure 6), we've found that fitness function design follows principles similar to gradient-based training: optimal performance occurs when the fitness function is neither too sensitive nor too insensitive to parameter changes. We propose using an energy-based method (Equation 19) with temperature T as a tunable parameter to achieve this balance. ------- We believe these additional results address your concerns about both empirical performance and theoretical foundations. We would greatly appreciate your thoughts on these new findings.

We appreciate your acknowledgment of our novelty and for pointing out several important items for us to address. In general, the choice of the g function is task-dependent. While evolutionary algorithms can operate with non-differentiable fitness functions, making the probability functions smooth and ensuring reasonable gradient (or some kind of pseudo gradients) values can be beneficial—specifically by avoiding excessively small or large gradients. To further address this question, we conducted a grid search on the benchmark functions. We modified the original g function to a new form with a control temperature parameter. Higher temperatures make the density function smoother, whereas lower temperatures make it more focused on high-fitness parameters. We first calculate the fitness scale of the functions, defined as the maximum fitness difference around the optimal points in the fitness function. Then, we set the temperature to be $\text{scale}\times T$, where $T$ is the parameter used to control the smoothness of the $g$ function. This rescaling allows for the comparison of different fitness functions. We observe two general trends: (1) Higher temperatures lead the algorithm to sample more low-fitness parameters, resulting in a lower average fitness (see [figure](https://postimg.cc/3dt8P3bH)). (2) Increasing the temperature enhances result diversity, as reflected in both entropy (see [figure](https://postimg.cc/V5244w4K)) and quality-diversity scores – a metric that quantifies the balance between quality and diversity (see definition at https://arxiv.org/abs/2303.12803). Our results are shown in [figure](https://postimg.cc/Z0n61Q0X). However, this effect plateaus quickly at around $T=1$. Therefore, we recommend first estimating the fitness scale and then using $T=1$ for initial experiments. We will update these guidance to the revised paper shortly. We also appreciate your last question and find it intriguing. As discussed in the paper, biological systems naturally employ multiple mechanisms to reduce dimensionality and assign importance to certain parameters, a concept known as sloppy model. Beyond this, geometric location and various non-genomic features also play a role. These directions will be examined in forthcoming work.

Dear Reviewer, Thank you again for your thoughtful comments on our work. We have revised our PDF manuscript with all changes highlighted in blue for easier reference. We've conducted analysis on the $g$ function evaluation that you inquired about, as shown in Figure 6 in revised version. We assessed its impact across three key metrics: fitness, diversity, and QD score. Our findings suggest a principle similar to gradient-based neural network training - the fitness sensitivity should be carefully balanced. If it's too high, diversity suffers; if it's too low, fitness improvement becomes challenging. Your question about the latent space prompted us to explore an interesting extreme case. We investigated what happens when we initially introduce latent variables that are completely independent of the parameter space. Specifically, we expanded a 2D parameter space by appending two additional dimensions: $$ (x_1,x_2)\to(x_1,x_2,{\color{blue}x_3,x_4}) $$ This approach mimics geometric distances found in biological systems. The results on the two-peaks fitness function (where (-1,-1) and (1,1) are optimal points) revealed an intriguing phenomenon of spatial isolation in parameters: we observed distinct clusters forming in the additional 2D latent space, where each cluster contains similar gene parameters, see [link](https://postimg.cc/v1xY8Kby) here. This finding not only provides insights into how geometry influences evolution but also suggests a potential clustering method. We believe these new results offer additional perspective on your questions about the role and behavior of latent spaces in our method. We would appreciate your thoughts on these findings.