Loss function

Frequentist expected loss

We first define the expected loss in the frequentist context. It is obtained by taking the expected value with respect to the probability distribution, ${\displaystyle P_{\theta }}$, of the observed data, ${\displaystyle X}$. This is also referred to as the risk function^{[11][12][13][14]} of the decision rule ${\displaystyle \delta }$ and the parameter ${\displaystyle \theta }$. Here the decision rule depends on the outcome of ${\displaystyle X}$. The risk function is given by:

${\displaystyle R(\theta ,\delta )=\operatorname {E} _{\theta }L{\big (}\theta ,\delta (X){\big )}=\int _{X}L{\big (}\theta ,\delta (x){\big )}\,\mathrm {d} P_{\theta }(x).}$

Here, ${\displaystyle \theta }$ is a fixed but possibly unknown state of nature, ${\displaystyle X}$ is a vector of observations stochastically drawn from a population, ${\displaystyle \operatorname {E} _{\theta }}$ is the expectation over all population values of ${\displaystyle X}$, ${\displaystyle \mathrm {d} P_{\theta }}$ is a probability measure over the event space of ${\displaystyle X}$ (parametrized by ${\displaystyle \theta }$) and the integral is evaluated over the entire support of ${\displaystyle X}$.

Bayes Risk

In a Bayesian approach, the expectation is calculated using the prior distribution ${\displaystyle \pi ^{*}}$ of the parameter ${\displaystyle \theta }$:

${\displaystyle \rho (\pi ^{*},a)=\int _{\Theta }\int _{\mathbf {X}}L(\theta ,a({\mathbf {x}}))\,\mathrm {d} P({\mathbf {x}}\vert \theta )\,\mathrm {d} \pi ^{*}(\theta )=\int _{\mathbf {X}}\int _{\Theta }L(\theta ,a({\mathbf {x}}))\,\mathrm {d} \pi ^{*}(\theta \vert {\mathbf {x}})\,\mathrm {d} M({\mathbf {x}})}$

where ${\displaystyle M(\mathbf {x} )}$ is known as the predictive likelihood wherein ${\displaystyle \theta }$ has been "integrated out," ${\displaystyle \pi ^{*}(\theta |\mathbf {x} )}$ is the posterior distribution, and the order of integration has been changed. One then should choose the action ${\displaystyle a^{*}}$ which minimises this expected loss, which is referred to as Bayes Risk. In the latter equation, the integrand inside ${\displaystyle \mathrm {d} x}$ is known as the Posterior Risk, and minimising it with respect to decision ${\displaystyle a}$ also minimizes the overall Bayes Risk. This optimal decision, ${\displaystyle a^{*}}$ is known as the Bayes (decision) Rule - it minimises the average loss over all possible states of nature ${\displaystyle \theta }$, over all possible (probability-weighted) data outcomes. One advantage of the Bayesian approach is to that one need only choose the optimal action under the actual observed data to obtain a uniformly optimal one, whereas choosing the actual frequentist optimal decision rule as a function of all possible observations, is a much more difficult problem. Of equal importance though, the Bayes Rule reflects consideration of loss outcomes under different states of nature, ${\displaystyle \theta }$.

Examples in statistics

• For a scalar parameter ${\displaystyle \theta }$, a decision function whose output ${\displaystyle {\hat {\theta }}}$ is an estimate of ${\displaystyle \theta }$, and a quadratic loss function (squared error loss) $${\displaystyle L(\theta ,{\hat {\theta }})=(\theta -{\hat {\theta }})^{2},}$$ the risk function becomes the mean squared error of the estimate, $${\displaystyle R(\theta ,{\hat {\theta }})=\operatorname {E} _{\theta }\left[(\theta -{\hat {\theta }})^{2}\right].}$$An estimator found by minimizing the mean squared error estimates the posterior distribution's mean.
• In density estimation, the unknown parameter is probability density itself. The loss function is typically chosen to be a norm in an appropriate function space. For example, for ${\displaystyle L^{2}}$ norm, $${\displaystyle L(f,{\hat {f}})=\|f-{\hat {f}}\|_{2}^{2}\,,}$$ the risk function becomes the mean integrated squared error $${\displaystyle R(f,{\hat {f}})=\operatorname {E} \left(\|f-{\hat {f}}\|^{2}\right).\,}$$