AI researchers propose Neural Diffusion Processes (NDP), a new approach based on diffusion models, which learns to sample from distributions over functions

Neural Diffusion Processes (NDPs), a proposed denoising diffusion model approach for learning probabilities on function spaces and generating samples of prior and conditional functions. PNDs generalize diffusion models to infinitely dimensional functional spaces by allowing the indexing of random variables. They demonstrate that this model can capture functional distributions similar to the true Bayesian posterior. The two-dimensional attention block is a novel component for building neural networks that relate size and sequence equivariance in the neural network architecture to behave as a stochastic process.

Traditionally, researchers have used Gaussian processes (GP) to specify prior and posterior distributions over functions. However, this approach becomes computationally expensive when scaled up, is constrained by the expressiveness of its covariance function, and struggles to scale to a point estimate of hyperparameters.

In the new newspaper Neural diffusion process, a research team is looking into these questions, proposing neural diffusion processes. The new framework learns to sample from rich distributions over functions at lower computational cost and to capture distributions close to the true Bayesian posterior of a standard Gaussian process.

The article explains that Bayesian inference for regression is advantageous but often expensive and requires a priori modeling assumptions. This is the presumption that underlies the processes of neural diffusion.

The group summarizes its main contributions as follows:

  • The team proposes a new model, the Neural Diffusion Process (NDP), which extends the application of diffusion models to stochastic processes and is able to describe a diverse distribution over functions.
  • The team takes special care to incorporate symmetries and known properties of stochastic processes, such as exchangeability, into the model, which facilitates the learning process.
  • The team demonstrates the capabilities and adaptability of NDPs by applying them to various Bayesian inference tasks, such as prior and conditional sampling, regression, hyperparameter marginalization, and Bayesian optimization.
  • The team also presents a novel global optimization method using NDP.

The proposed NDP is a method based on a denoising diffusion model to learn probabilities from a function and generate samples of prior and conditional functions. It allows complete marginalization on GP hyperparameters while reducing the computational load compared to GPs.

The team analyzed the quality of samples of existing state-of-the-art generative models based on neural networks. Based on their findings, they developed NDP to generalize diffusion models to infinite-dimensional functional spaces by allowing the indexing of random variables over which the model diffuses.

The researchers also adopted a new two-dimensional attention block to ensure input dimensionality and sequence equivariance and to allow the model to take samples from a stochastic process. Therefore, NDP can take advantage of stochastic processes, such as exchangeability.

Source: https://arxiv.org/pdf/2206.03992.pdf

In their empirical study, the team assessed the ability of the proposed NDP to generate high-quality conditional samples, marginalize kernel hyperparameters, and maintain input dimensionality invariance.

The results demonstrate that NDP can capture functional distributions close to the true Bayesian posterior while simultaneously reducing computational requirements.

The researchers note that while the number of diffusion steps improves NDP sample quality, it also slows down inference times. According to the authors, future research could investigate inference acceleration or sample parameterization techniques to address this issue.

Future work- The researchers found, as with other diffusion models, that the sample quality of an NDP increases with the number of diffusion steps T. This results in slower inference times compared to different architectures like GANs. To solve this problem, techniques for accelerating the inference process could be implemented. The method proposed by Watson et al. is of particular interest to PNDs as it provides them with a moral and distinct metric to assess the quality of the sample, namely the marginal likelihood of the corresponding GP. In conclusion, parameterizing the samples in the Fourier domain could be an intriguing alternative method.

This Article is written as a summary article by Marktechpost Staff based on the paper 'NEURAL DIFFUSION PROCESSES'. All Credit For This Research Goes To Researchers on This Project. Checkout the paper, code (coming soon), ref article.

Please Don't Forget To Join Our ML Subreddit

James G. Williams