--- ![[Pasted image 20221219165934.png|center|350]] #### Mathematical Model of a Neuron - Single neuron model $y=f(\mathbf{w}^T\mathbf{x}+b)$can be grouped together in layers of m inputs and n outputs with - Input $\mathbf{x}\in \mathbb{R}^{m \times1}$ - Parametes or weigths $\mathbf{W}\in \mathbb{R}^{n \times m}$ - Bias $b$ - often included in the weights $\mathbf{w}=[\mathbf{w}^T,b]^T$ by using an extended input with an additional 1 $\mathbf{x}=[\mathbf{x}^T,1]^T$ - Pre-activation vector $\mathbf{z}= \mathbf{W}\mathbf{x}$ - Activation function $f:\mathbb{R}^{n \times 1} \rightarrow \mathbb{R}^{n \times 1}$ - Output vector $\mathbf{y}= f(\mathbf{z} )$ - How to learn Parameters ? - [[Forward and Backward Propagation]] --- #### Building Blocks if Differentiable Circuits - MLP Layers - Linear accumulation with nonlinear activations - [[Convolutional Neural Network Layers]] - [[Recurrent Neural Network Layers]] - [[Attention Mechanisms]] - [[Residual or Skip Connections in Deep Learning]] --- - **Overfitting** - NN have a lot of parameters $\rightarrow$ prone to Overfitting $\rightarrow$ fight off with a prior - **Regularization** - Regularize input data - **Early Stopping** - Stop when validation error starts rising again - **Input Noise Augmentation** - Add artificial noise - **[[Dropout Techniques in Deep Learning|Dropout]]** - Prune less relevant neurons and rescale activations for others