\n",
"| | \n",
" \n",
" | \n",
"
\n",
"
\n",
"![](\"https://i.imgur.com/jtNMGca.png\" "\\"Bias")
\n",
"\n",
"\n",
"\n",
"#### **Interpretation**\n",
"\n",
"1. **Model Complexity vs. Error:**\n",
" - The training error decreases as model complexity increases because the model can better fit the training data.\n",
" - The test error decreases initially but starts to increase after a certain point, indicating overfitting. The test error curve shows a local minimum, reflecting the optimal model complexity where the tradeoff between bias and variance is balanced.\n",
"\n",
"2. **Bias-Variance Decomposition:**\n",
" - Bias squared decreases with increasing model complexity, as the model becomes more capable of capturing the underlying patterns.\n",
" - Variance increases with model complexity, as the model starts to fit the noise in the training data.\n",
" - The total error curve shows a local minimum at an optimal model complexity, balancing bias and variance. This minimum is marked with a vertical green dashed line.\n",
" - The intersection point of the bias and variance curves is not necessarily the point of minimum total error.\n",
"\n",
"In summary, the optimal model complexity is where the total error is minimized, which takes into account the contributions of bias, variance, and irreducible error. This point is typically where the model has the best balance between underfitting and overfitting."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qEuPgZgJxa16"
},
"source": [
"### Mean Squared Error \n",
"\n",
"From many *independent* imperfect measurements we can easily derive a better one. Here the accent goes on the concept of independence. We say that two events, $A$ and $B$, are independent if\n",
"\n",
"$$ \\text{P}\\left(A\\cap B\\right) = \\text{P}\\left(A\\right)\\cdot\\text{P}\\left(B\\right)$$\n",
"\n",
"Observation: While the above statement is correct, it is also deceivingly simple, and it's typically hard to use in practice.\n",
"\n",
"Now, assume we have $n$ independent imperfect, however unbiased, measurements of an object labeled as\n",
"\n",
"$$ x_1, x_2, \\dots x_n$$\n",
"\n",
"and we assume that there is an ideal mesurement $x^*$ and we want to find it. Gauss' idea is to create an objective function that weighs more heavily larger errors. This can be formalized by assuming that we want to compute a value $x$ that minimizes the sum of the squared errors/deviances:\n",
"\n",
"$$\\text{SSE}\\left(x\\right)\\overset{\\Delta}{=} (x-x_1)^2 + (x-x_2)^2 +\\dots (x-x_n)^2$$\n",
"\n",
"If we divide by the number of observations, $n$, we get\n",
"\n",
"$$\\text{MSE}\\left(x\\right)\\overset{\\Delta}{=} \\frac{(x-x_1)^2 + (x-x_2)^2 +\\dots (x-x_n)^2}{n}$$\n",
"\n",
"**Important:** When the number of observations is fixed, $n$ is just a constant.\n",
"\n",
"So, in this case, we want to determine\n",
"\n",
"$$ \\underset{x\\,\\in\\, \\mathbb{R}}{\\operatorname{argmin}} \\text{SSE}(x)$$\n",
"\n",
"that is to say we want the value of $x$ that yields the minimum of MSE.\n",
"\n",
"To solve the problem, we take the derivative of the MSE with respect to $x$ and set it equal to zero:\n",
"\n",
"$$ 2(x-x_1) + 2(x-x_2) + \\dots 2(x-x_n)=0.$$\n",
"\n",
"When you solve this equation, you get that\n",
"\n",
"$$ x = \\frac{x_1+x_2+\\dots x_n}{n}$$\n",
"\n",
"This highlights the idea that the average is the best representative of a sample."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0yigJBmt6FQS"
},
"source": [
"### The Gradient Descent in 2-D \n",
"\n",
"We want to minimize a cost function that depends on some coefficients. An example in 2-D is\n",
"\n",
"$$L(w_1,w_2): = \\frac{1}{n}\\sum (y_i - w_1\\cdot x_{i1} - w_2\\cdot x_{i2})^2$$\n",
"\n",
"Here we have a vector $$\\vec{w}:=(w_1,w_2)$$\n",
"\n",
"We can think of the vector $\\vec{w}$ having (in this case) two components and a perturbation of $\\vec{w}$ in some direction such as $\\vec{v}.$\n",
"\n",
"We consider the function $g(t):=L(\\vec{w}+t\\cdot \\vec{v})$ we get some important ideas:\n",
"\n",
"i. if $\\vec{w}$ is ideal for the cost then $t=0$ is min of the function $g$ and so $g'(0)=0.$\n",
"\n",
"ii. if $\\vec{w}$ is not minimizing the cost then we want to decrease the function $g$\n",
"\n",
"Here we have that $$g'(t):= \\nabla L \\cdot \\vec{v}$$ should be negative (because we want to decrease $g$) and we get a good outcome if\n",
"\n",
"$$\\vec{v}:= - \\nabla L$$\n",
"\n",
"This means that the coefficients should be updated in the negative direction of the gradient, such as:\n",
"\n",
"$$ w_{new} := \\vec{w}_{old} - \\text{lr}\\cdot\\nabla L$$\n",
"\n",
"where \"lr\" stands for \"learning rate.\"\n",
"\n",
"![](\"https://i.imgur.com/KzUuU3q.gif\" "\\"Gradient")
\n",
"\n",
"### The Loss Function (SSE) and its Gradient for n Observations \n",
"$$L(w_1,w_2):=\\sum_{i=1}^{n}(y_i-w_1\\cdot x_{i1} -w_2\\cdot x_{i2})^2$$\n",
"\n",
"$$\\frac{\\partial{L}}{\\partial w_1}=\\sum_{i=1}^{n} 2\\cdot (y_i-w_1\\cdot x_{i1} -w_2\\cdot x_{i2})\\cdot (-x_{i1})$$\n",
"\n",
"### A 2D Example with Partial Derivatives and One Observation \n",
"\n",
"If\n",
"\n",
"$$L(w_1,w_2) = (y-w_1\\cdot x_1-w_2\\cdot x_2)^2$$\n",
"\n",
"What are the partial derivatives of $L$ with respect to $w_1$ and $w_2$?\n",
"\n",
"$$\\frac{\\partial{L}}{\\partial{w_1}}= 2\\cdot(y-w_1\\cdot x_1-w_2\\cdot x_2)\\cdot (-x_1)$$\n",
"\n",
"$$\\frac{\\partial{L}}{\\partial{w_2}}= 2\\cdot(y-w_1\\cdot x_1-w_2\\cdot x_2)\\cdot (-x_2)$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L_HF1LHaA0Un"
},
"source": [
"### Code Applications \n",
"\n",
"---\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Xy_PgW957aX8",
"outputId": "84210f3f-5438-4977-8756-5e3b91beafcf"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running on CoLab\n",
"Mounted at /content/drive\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m522.0/522.0 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"import os\n",
"if 'google.colab' in str(get_ipython()):\n",
" print('Running on CoLab')\n",
" from google.colab import drive\n",
" drive.mount('/content/drive')\n",
" os.chdir('/content/drive/My Drive/Data Sets')\n",
" !pip install -q pygam\n",
"else:\n",
" print('Running locally')\n",
" os.chdir('../Data')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_tKGrayQ9R_o"
},
"outputs": [],
"source": [
"# Setup Libraries\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib as mpl\n",
"%matplotlib inline\n",
"%config InlineBackend.figure_format = 'retina'\n",
"import matplotlib.pyplot as plt\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "aTkw9baOHAo_"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "v1pF0itWJXSc"
},
"source": [
"#### Example with Functions and Classes"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yFF66qwBujBp"
},
"source": [
"Let's create some examples in Python:\n",
"\n",
"**Example 1** Create a function whose graph intersects the line $y=0.5$ exactly three times.\n",
"\n",
"*Solution:* We think of a simple possible design that accomodates the requirement such as the construct of a third degree polynomial:\n",
"\n",
"$$\n",
"p(x) = (x-1)\\cdot(x-2)\\cdot(x-3)+0.5\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "49Zlfbn9Upud"
},
"outputs": [],
"source": [
"def myfunction(x):\n",
" return (x-1)*(x-2)*(x-3)+0.5"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Gv3EtzSTU7nS",
"outputId": "7d5a7b88-8450-487f-f2e4-d6e54d15c557"
},
"outputs": [
{
"data": {
"text/plain": [
"0.14300000000000007"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myfunction(2.7)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "p0TpGhVR7yYq"
},
"outputs": [],
"source": [
"myfunction(t)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vc9p4jASdEwn"
},
"source": [
"**Example 2** Create a piecewise defined function with three different cases.\n",
"\n",
"*Solution:* Again, we can think of a simple example such as\n",
"\n",
"$$\n",
"f(x) = \\begin{cases}\n",
"x &\\text{if}\\,\\,\\,0