Probabilistic Models and Inference (VO1 + PS1, Master)

See the UIBK course catalog entries for the lecture and the seminar for essential information.

This course will be organized similarly to its companion course Advanced Machine Learning (VO2 + PS1, Master). The online sessions will last about one hour and will encompass VO discussion and the PS; thus, attendance is mandatory for all PS registrants.

There is an online forum for students to ask and answer each other's questions, where also the instructors will answer questions.

Some Suggested Topics

Boltzmann Machines; Deep Belief Networks
Hidden Markov Models
Density Estimation
Sampling Techniques; MCMC
Latent variable models
Variational Inference (vs. MCMC; Variational Auto Encoder)

Suggest more and/or vote for any of these.

Texts

See the companion course.
DE Density Estimation for Statistics and Data Analysis by B.W. Silverman. We only refer to the first two chapters.

Agenda

Homework Assignments

Programming assignments are to be downloaded from JupyterHub, solved on your local machine, and reuploaded to JupyterHub, exactly like in the AdvML PS.

ProbMod-01-linear-regression is due 2020-11-11.
ProbMod-02-factor-graph is due 2021-01-13.
ProbMod-03-mean-field is due 2021-01-27.

Lectures

In addition to the resources given below, this course extensively uses Bishop's slides for PRML chapters 1 and 8.

The videos are in OLAT.

Topic	Video	Text	Notes	Ex.
2020-10-08Linear Regression
Introduction			intro
Linear Regression		PRML 1.1	PRML-01
2020-10-15Probability Theory
Probability Theory: Random Variables	8 5:13		PRML-01	+
Exercises: Why does a probability distribution have to sum to 1? What implications does this have on modeling real-world scenarios?
Probability Theory: Probability Densities	10 7:21	PRML 1.2.1	PRML-01	+
Exercises: Draw the pdf and cdf for a uniform probability distribution over the real numbers between 0 and 1. Draw a pdf and cdf for a nonuniform probability distribution over the real numbers between 0 and 1 where you are twice as likely to draw a number below 0.5 than above 0.5. What about numbers equal to 0.5? Can a pdf $p(x)$ exceed 1? Can a cdf $P(x)$ exceed 1?
Probability Theory: Expectations	12 9:24	PRML 1.2.2	PRML-01	+
Exercises: On Bishop Slide 25, what exactly is $f$? Why does conditional expectation use $p(x\|y)$ but not $f(x\|y)$? What is $\mathbb{E}[X]$ for $x$ drawn from the pdf $p(x) = 1$, $x \in [0,1]$? What is $\mathbb{E}[X]$ for $x$ drawn from the pdf $p(x) = 2x$, $x \in [0,1]$? Prove the identities involving expectations given in the exercise (Section 7.3 in the PDF notes). Try this without looking at the solutions.
Probability Theory: Covariance	14 14:19	PRML 1.2.2	PRML-01	+
Exercises: Prove the two scalar identities involving covariance and variance, respectively, given in the exercise (Section 7.4 in the PDF notes). Try this without looking at the solutions. Show that if two random variables are independent, then their covariance is zero. Explain the meaning of the coefficients of a 3×3 covariance matrix.
2020-10-22ML and MAP Estimation; Probabilistic Regression
Probability Theory: Normal Distribution	24 11:07	PRML 1.2.4	PRML-01	+
Exercises: Verify that the $D$-variate normal distribution contains the univariate distribution ($D = 1$) as a special case. What is the standard deviation?
Probability Theory: Maximum-Likelihood Estimation of the Paramters of a Normal Distribution	26 17:39	PRML 1.2.5	PRML-01	+
Exercises: What does i.i.d. mean, and how did we rely on this concept? What is a likelihood (function)? Explain the maximum likelihood (ML) principle. We maximized the likelihood function by maximizing its logarithm instead. Why did we do this? Why is this legitimate? We maximized the log likelihood function in the classical way by setting its partial derivatives in its parameters to zero, yielding a system of two simultaneous equations. But we did not seem to solve a system of equations. Why not?
Probability Theory: ML Estimate of Variance is Biased	28 4:38	PRML 1.2.4	PRML-01	+
Exercises: Explain informally why the ML estimator of the mean is unbiased but the ML estimator of the variance is biased. Why might it be interesting to estimate the parameters of a distribution from a sample? Can you think of practical applications?
Regression: Maximum-Likelihood Parameter Estimation	30 14:39	PRML 1.2.5	PRML-01	+
Exercises: Derive all three equations on Bishop Slide 34 (Maximum Likelihood). What is a generative model? For the purposes of prediction, what advantage does probabilistic estimation give us over non-probabilistic estimation using the same error function?
Regression: Maximum A Posteriori Parameter Estimation	32 12:13	PRML 1.2.5	PRML-01	+
Exercises: What is the key distinction between ML estimation and MAP estimation? In MAP estimation, why is the denominator of Bayes' Rule irrelevant? On Bishop Slide 36 (MAP: A Step towards Bayes), derive the error function from the posterior of $\mathbf{w}$.
2020-10-29Probabilistic Prediction; Classification
Regression: Bayesian Prediction	34 13:11	PRML 1.2.6	PRML-01	+
Exercises: Why do ML and MAP estimation underestimate the variance of the predictive distribution? How does Bayesian prediction fix this?
Probabilistic Prediction: ML and MAP Parameter Estimation; Bayesian Prediction	36 8:06	PRML 1.2.6	PRML-01	+
Exercises: Point out all occurrences of generative models and likelihood functions in Section 4.9 of the PDF notes. What is the core principle of Bayesian prediction? Why does prediction based on an MAP parameter estimate not constitute Bayesian prediction, even though it uses Bayes' Rule?
Classification: ML and MAP	40 11:24	PRML 1.5–1.5.1; 1.5.3	PRML-01	+
Exercises: For ML classification, how do we estimate the class-conditional densities? For MAP classification, how do we estimate the class priors? In MAP classification, what do we do about the marginal data density $p(x)$?
Classification: Generative and Discriminative Models; Reject Option	42 7:48	PRML 1.5.3–1.5.4	PRML-01	+
Exercises: In a 1-dimensional, 2-class classification problem, how many decision points can we obtain if we model the class-conditional densities by normal distributions? In a 2-dimensional, 3-class classification problem, what do the decision boundaries look like if we model the class-conditional densities by normal distributions with identical covariance matrix $\lambda I$? What is the distinction between classification using generative models, using a discriminative model, and using a discriminant function? In a general, 2-dimensional, 2-class classification problem, what may the reject region(s) look like?
2020-11-05Classification: Loss; Disciminative Models
Classification: Loss Matrices, Loss Functions	44 12:57	PRML 1.5.2	PRML-01	+
Exercises: Solve the exercise on Decision Theory in Section 7.6 of the PDF notes. Convert the loss function of our cancer example into an equivalent utility function.
Classification: High-Level Comparison of Probabilistic and Non-Probabilistic Paradigms	46 4:16		PRML-01	+
Exercises: In what application scenarios would you prefer to use classification based on a generative probabilistic model, a discriminative probabilistic model, or a (non-probabilistic) discriminant function, respectively?
Discriminative Models for Classification: Linear Discriminants	50 10:33	PRML 4.2–4.2.1	PRML-04	+
Exercises: Derive the coefficients for the linear discriminants a for 2 classes and for $K$ classes.
Discriminative Models for Classification: Nonlinear Discriminants; the Exponential Family	52 4:16	PRML 4.2.1	PRML-04	+
Exercises: Show that the three examples of members of the exponential family are in fact such.
Discriminative Models for Classification: Basis Functions	54 9:25	PRML 4.3.1, 4.3.2	PRML-04
Discriminative Models for Classification: Maximum-Likelihood	56 11:58	PRML 4.3.2	PRML-04	+
Exercises: Work out the derivatives of $\sigma(a)$ and of $E(\mathbf{w})$.
Discriminative Models for Classification: Logistic Regression for 2 Classes	58 13:28	PRML 4.3.3	PRML-04	+
Exercises: Make sure you understand all the math of IRLS for 2 classes.
Discriminative Models for Classification: Logistic Regression for K Classes	60 5:18	PRML 4.3.4	PRML-04	+
Exercises: Fill in the missing mathematical steps of IRLS for $K$ classes.
2020-11-12Bayesian Networks: Factorization; Conditional Independence
Bayesian Networks: Factorization and Corresponding Directed Acyclic Graph	100 11:00	PRML 8.1 (Intro)	PRML-08	+
Exercises: Characterize the topology of the Bayesian network representing a general joint probability distribution. Argue that a Bayesian network cannot contain directed cycles. Factorize joint probability distributions of various dimensions in various way, dropping various variables from conditions, and draw the corresponding Bayesian networks. Draw various DAGs, and write down the corresponding factorized joint probability distributions.
Bayesian Networks: Plate Notation; Modeling Regression	102 10:28	PRML 8.1.1	PRML-08	+
Exercises: Redraw Bishop-2006-PRML Fig. 8.5 without plate for $N = 3$. Draw a figure like Bishop-2006-PRML Fig. 8.8 using plate(s) for $N$ objects in the image, each with its own position and orientation.
Bayesian Networks: Counting Model Parameters	104 9:05	PRML 8.1.3		+
Exercises: For the Bayesian networks shown in Bishop-2006-PRML Figs. 8.11 and 8.12, determine the number of parameters of their (factorized) joint probability distributions, assuming that each $\mu$ and each $\mathrm{x}$ can take $K$ different values.
Bayesian Networks: Conditional Independence	106 4:49	PRML 8.2 (Intro)	PRML-08	+
Exercises: Do PRML Exercises 8.3 and 8.4 (also in the Notes).
Bayesian Networks: Conditional Independence: Tail-to-Tail and Head-to-Tail Connections	108 10:21	PRML 8.2.1
Bayesian Networks: Conditional Independence: Head-to-Head Connection	110 6:29	PRML 8.2.1	PRML-08	+
Exercises: Do PRML Exercise 8.10 (also in the Notes).
Bayesian Networks: Fuel Gauge Example; Explaining Away		PRML 8.2.1		+
Exercises: Make sure you understand all the mathematical reasoning, and verify the numerical results of Eqns. 8.30–8.33. Define in your own words the concept of explaining away. Do PRML Exercise 8.11.
2020-11-19BN: d-Separation and Markov Blanket; MRF: Conditional Indpendence Properties
Bayesian Networks: d-Separation	114 9:55	PRML 8.2.2	PRML-08	+
Exercises: Show that $a ⫫ b, c \mid d$ implies $a ⫫ b \mid d$. For the graph on Bishop Slide 3, determine the d-separation relations between various triples of nodes of your choice.
Bayesian Networks: d-Separation examples: IID variables; Naive Bayes classifier	116 14:35	PRML 8.2.2	PRML-08
Bayesian Networks: Metaphor of filtering by factorization or by d-separation	118 2:59	PRML 8.2.2		+
Exercises: Argue that adding an arc to Bayesian network always enlarges the family of joint probability distributions compatible with this Bayesian network.
Bayesian Networks: Markov Blanket	120 5:55	PRML 8.2.2		+
Exercises: Using the d-separation criterion, show that the conditional distribution for a node $x$ in a DAG, conditioned on all of the nodes in the Markov blanket, is independent of the remaining variables in the graph.
Markov Random Fields: Conditional Independence Properties		PRML 8.3.1		+
Exercises: Think of example applications that lend themselves to modeling with MRF. In PRML Fig. 8.27, what is the Markov blanket of each of the nodes?
2020-11-26MRF: Factorization; Relations to BN; Inference on Chains and Trees
Markov Random Fields: Factorization Properties		PRML 8.3.2, 8.3.3		+
Exercises: Show that the partition function $Z$ is needed for parameter estimation. One way to do this is to consider a two-pixel image and energy function as in the image-denoising example, setting all four variables to fixed values, and observing what happens if one minimizes the energy function over its parameters. Show that the partition function $Z$ is not needed to compute conditional probabilities within an MRF. In the image-denoising example, what happens if we set $h$, $\beta$ or $\eta$ to zero, respectively?
Graphical Models: Relations Between Directed and Undirected Models		PRML 8.3.4		+
Exercises: Show that there is no undirected graphical model that is a perfect map for the directed graphical model consisting of three nodes and two arcs in a head-to-head arrangement (PRML Fig. 8.35). Show that there is no directed graphical model that is a perfect map for the undirected graphical model consisting of four nodes and four arcs in a ring arrangement (PRML Fig. 8.36).
Markov Chain: Marginalization Using Distributive Law	128 9:58	PRML 8.4.1
Markov Chain: Marginalization Using Distributive Law: Example	130 8:16		PRML-08	+
Exercises: Show that observing (conditioning on) a variable amounts to replacing the sum over its values by the observed value.
Markov Chain: Marginalization Using Message Passing	132 5:08	PRML 8.4.1		+
Exercises: Show that $p(x_{n-1}, x_n) = \frac{1}{Z} \mu_{\alpha}(x_{n-1}) \psi_{n-1,n}(x_{n-1}, x_n) \mu_{\beta}(x_n)$ (PRML Eqn. 8.58, Ex. 8.15). Consider a Markov chain of $N=5$ nodes, in which nodes $x_3$ and $x_5$ are observed. Show that if our message-passing algorithm is applied to the evaluation of $p(x_2 \mid x_3, x_5)$, the result will be independent of $x_5$. (PRML Ex. 8.17)
Markov Chain: Marginalization Using Message Passing: Example	134 5:57		PRML-08
Markov Chain: Marginalization For All Variables	136 2:58	PRML 8.4.1
MRF Trees		PRML 8.4.2		+
Exercises: Show that a distribution represented by a directed tree can trivially be written as an equivalent distribution over the corresponding undirected tree. Also show that a distribution expressed as an undirected tree can, by suitable normalization of the clique potentials, be written as a directed tree. Calculate the number of distinct directed trees that can be constructed from a given undirected tree. (PRML Ex. 8.18)
2020-12-03Factor Graphs; Sum-Product Algorithm
Factor Graphs		PRML 8.4.3		+
Exercises: Convert the Bayesian network of the fuel-gauge example (PRML Fig. 8.21 and the accompanying text) into a factor graph. What are the factors?
Sum-Product Algorithm		PRML 8.4.4	PRML-08 Sect. 5, 6, 9.3	+
Exercises: In the case of discrete random variables, how exactly are factors and messages represented? In the case of discrete random variables, how exaclty does a variable node compute its outgoing message from its incoming messages (PRML Eqn. 8.69)? In the case of discrete random variables, how exaclty does a factor node compute its outgoing message from its incoming messages (PRML Eqn. 8.66)? Solve the fuel system example (PRML Sec. 8.2.1 pp. 376–378) using the Sum-Product algorithm.
2020-12-10Max-Product Algorithm; General Graphs; Loopy Belief Propagation
Max-Sum Algorithm		PRML 8.4.5	PRML-08 Sect. 7, 8, 9.4	+
Exercises: Do PRML Ex. 8.27. For the fuel system example (PRML Sec. 8.2.1 pp. 376–378), compute the most probable configuration using the Max-Sum algorithm.
Exact Inference in General Graphical Models		PRML 8.4.6
Loopy Belief Propagation		PRML 8.4.7
2020-12-17Kernel Density Estimation
Kernel Density Estimation: Motivation	400 7:49	DE 1	KDE Sect. 2
Kernel Density Estimation: Histogram	402 8:01	DE 2.2	KDE Sect. 2.6
Kernel Density Estimation: Naive Estimator	404 6:56	DE 2.3	KDE Sect. 2.7, 2.8
Kernel Density Estimation: Kernel Estimator	406 3:45	DE 2.4	KDE Sect. 3.1, 3.2
Kernel Density Estimation: Nearest-Neighbor Estimator	408 5:53	DE 2.5	KDE Sect. 3.3, 3.4
Kernel Density Estimation: Variable-Kernel Estimator; General Remarks	410 4:52	DE 2.6	KDE Sect. 3.5, 4
Kernel Density Estimation: Bounded Domains	412 6:50	DE 2.10	KDE Sect. 5
Kernel Density Estimation: Circular Domains	414 4:04		KDE Sect. 5
2021-01-07Hidden Markov Models
Hidden Markov Models			HMM	+
Exercises: See the notes.
2021-01-14Variational Inference
Variational Inference			variational-inference	+
Exercises: See the notes.

RSS Feed

Justus Piater, 2020/10/11 22:27

The preparation material for 2020-10-15 has been finalized.

Justus Piater, 2020/10/19 20:50

The preparation material for 2020-10-22 has been finalized.

Justus Piater, 2020/10/26 21:40, 2020/10/28 07:47

The preparation material for 2020-10-29 has been finalized.

Justus Piater, 2020/10/28 14:23

This page is now accessible from the internet; a VPN is no longer required.

Justus Piater, 2020/11/02 17:25

The first programming assignment is out.

Justus Piater, 2020/11/02 23:41

The preparation material for 2020-11-05 has been finalized.

Justus Piater, 2020/11/07 18:18

Assignment 1 Task 3 Opening Paragraph

Write a function that computes the predictive distribution expressed by the vectors $\hat{\mu}$ and $\hat{\sigma}^2$ for a set of test input values, received in the form of its design matrix $\Phi_{\textrm{test}}$, given the design matrix $\Phi$ and target values $\mathbf{t}$ of the training data, and the meta-parameter $\alpha$.

Justus Piater, 2020/11/08 11:48

The preparation material for 2020-11-12 has been finalized.

Justus Piater, 2020/11/12 16:59

Assignment 1 has been graded and feedback released.

Justus Piater, 2020/11/16 15:55

The preparation material for 2020-11-19 has been finalized.

Justus Piater, 2020/11/22 20:17

The preparation material for 2020-11-26 has been finalized.

Justus Piater, 2020/12/02 08:47

The preparation material for 2020-12-03 has been finalized.

Justus Piater, 2020/12/07 19:45

The preparation material for 2020-12-10 has been finalized.

Justus Piater, 2020/12/12 18:42

The preparation material for 2020-12-17 has been finalized.

Justus Piater, 2020/12/19 21:56

The second programming assignment is out.

Justus Piater, 2021/01/14 19:28

Oral exams will take place February 4 online. Please register by February 21.

Online exams are subject to the conditions Absolvierung der kommissionellen Abschlussprüfung bzw. Defensio. In addition:

You need to be able to write and draw in a way that I can see it on my screen. Thus, you need either a tablet (possibly separately connected to the video call), a webcam facing a sheet of paper or a board, or equivalent. Please arrange and test this beforehand!
I will occasionally take screenshots of the video call for my records (no video, no voice).

Justus Piater, 2021/01/15 19:46

The third (and last) programming assignment is out.

Sidebar

Table of Contents

Probabilistic Models and Inference (VO1 + PS1, Master)

Some Suggested Topics

Texts

Agenda

Homework Assignments

Lectures

RSS Feed

Assignment 1 Task 3 Opening Paragraph

User Tools

Site Tools

Sidebar

Table of Contents

Probabilistic Models and Inference (VO1 + PS1, Master)

Some Suggested Topics

Texts

Agenda

Homework Assignments

Lectures

RSS Feed

Assignment 1 Task 3 Opening Paragraph

Page Tools