Bayesian Field Theory

Chapter 8. Summary

Chapter 8 Summary In this book we wanted to develop a tool box for constructing prior models within a nonparametric Bayesian framework to empirical learning, and to exemplify its use for problems from different application areas. Nonparametric models, or field theories in the language of physics, allow typically a more explicit implementation of a priori information than parametric approaches. This is especially important for complex learning tasks which rely essentially on an adequate implementation of available a priori information. After introducing the general nonparametric Bayesian framework (Chapter 2), the basic building blocks for constructing prior models, Gaussian process prior factors, were studied in Chapter 3 for general density estimation problems . In the formulation of the framework, it proved to be advantageous to include from the very beginning both, dependent variables (V) an d independent variables (V), as this allows density estimation, regression, and classification to be treated on the same footing. Already Gaussian prior factors, or generalized free fields in the language of physics, turned out to provide quite flexible prior models. By choosing the appropriate covariance they allow approximately invariant functions to be reconstructed, ranging from functions which are approximately invariant under infinitesimal translations (smooth functions), to approximately periodic functions and approximate fractals (Sections 3.6, 3.8.5, and 7.1.5). In contrast to exact symmetries, which are of fundamental importance for many theories in physics, the concept of approximate invariance or approximate symmetry seems to be more adequate for typical practical learning problems. If a numerical solution by discretization becomes intractable for a nonparametric model, one can resort to a parametric model, for instance, additive models, decision trees, projection pursuit techniques, or neural networks. In Chapter 4 parametric methods are interpreted as variational approximations of nonparametric models. This defines a systematic procedure to derive prior terms for parametric approaches from an underlyingnonparametric prior 309 310 CHAPTER 8. SUMMARY model. A large improvement in flexibility of prior models can be achieved by taking into account hyperparameters (Chapter 5). Technically the theory then becomes intrinsically non-Gaussian. In practice, the difficulty is to find the right balance between the additional flexibility introduced by hyperparameters and the information contained in the training data, because models which are too flexible do not generalize from training data to non-training data. For Gaussian prior factors hyperparameters parameterize the mean (Section 5.3) and/or the covariance (Section 5.4). For instance, parameterized means allow the solution of a parametric model to be taken as the mean of a Gaussian prior factor for a nonparametric model (Section 5.3.2). In image completion problems hyperparameters can adapt the scaling, rotational, and translational degrees of freedom of reference images (representing means of prior factors) to the training data (Section 6.6). Parameterizing the mean of a Gaussian process is especially convenient as this does not require adaption of a normalization factor. Generalizing hyperparameters from numbers to functions leads to models with hyperfields, corresponding to full interacting field theories (Section 5.6). Hyperfields are useful to adapt prior models locally. An integer hyperfield, for instance, can reflect local distortions in an approximately periodic potential (Section 7.1.5). An example of a higher dimensional hyperfield is an optical flow field which adapts a reference image in an image completion problem to the training data. An alternative to models with hyperfields are the nonGaussian models discussed in Section 5.7. Models with an integer hyperparameter, where the summation over that hyperparameter can be carried out exactly, define the prior mixture models which are treated in Chapter 6. Mixtures of Gaussian prior factors are able to model multimodal prior surfaces and can thus take into account a moderate number of alternative Gaussian prior models simultaneously. Based on prior mixtures a method has been proposed to build nonparametric models from verbally formulated a priori knowledge. To exemplify their use and to check computational feasibility, the basic variants of the prior models discussed in this book have been studied numerically for selected likelihood models, each representing a typical application class of empirical learning. The numerical case studies include: General density estimation with Gaussian prior factors (Section 3.4) and with prior mixtures (Section 6.3), Gaussian regression for approximately invariant functions (Section 3.8.5), and an image completion problem with hyperparameters and prior mixture model (Section 6.6). As an example of special interest for physicists, Chapter 7 deals with the likelihood model of quantum theory. The empirical learning problem studied in this chapter is also called...