Notes

Empirical Risk Minimization = Minimize loss L as measured (empirically) on the data.

$$ \min_{w,b} \sum_{i=1}^m L(f_{w,b}(x^{(i)}), y^(i)) $$

i.e. find parameters w,b that optimize this function.

Generally in regression, we have:

$$ $$

And in classification, we have:

$$ (s(y))_j = \frac{e^{y_j}}{\sum_i e^{y_i}} \\ D(s,l) = -\sum_i l_i \log s_i $$