GP Site

Are all covariance functions known to be positive-definite?

Before we answer that question, let's define the meaning of positive-definite.

Positive-definiteness
Bochner Theorem (1955) states that a continuous function is positive definite if and only if it is the Fourier transform of a finite, non-negative measure.

And in linear algebra, a positive-definite matrix is a Hermitian matrix such that in many ways this matrix is analogous to a positive real number.

Let M be an n × n Hermitian matrix. The transpose of a vector, a, is denoted by a^T, and its conjugate transpose denoted by a^*. Matrix M is positive definite if and only if it satisfies the following property:

For all non-zero vectors z ∈ Cⁿ,

z^* M z > 0

Gaussian Process Model Definition
A Gaussian process model is a mathematical model for a nonlinear relationship, say f(z), which depends on some explanatory variable, z in this case. The model is commonly used to model all possible nonlinear relationships, such that in a random function model, any one particular nonlinear function is a single realisation of the random functions. Thus, the model is simply the class of realisations for the random functions.

The Gaussian process model places a probability distribution over the set of all possible relationships; hence, the joint probability distribution for [f(z₁),...,f(z_N)] for any finite set of values of [z₁,...,z_N] is specified. Since the chosen random function is Gaussian, its name is therefore given as Gaussian process model.

It follows that the model is simply defined by its mean function, and its covariance function.