Member-only story
Key Properties of Activation Functions in Neural Networks
The Impact of Activation Function Choice in CNNs
As we have seen in the last article, activation functions are a fundamental concept in neural networks because they introduce non-linearity to the network, allowing the model to learn more complex relationships and capture intricate patterns within data.
In this blog, we will learn about generical but essential properties of activation functions, and how they impact the model performance. Knowing how these properties affect the model performance and convergence will help a researcher make better decisions when it's time to choose an activation function.
Differentiability and Gradients
Differentiability property refers to the ability of a function to have a derivative that is well-defined at every point in its domain. Gradients are used for optimization during backpropagation, and they provide information about the direction and magnitude of the steeped descent, allowing the model to adjust weights by minimizing loss. The way to obtain a gradient is by differentiation. Without a gradient, our models stop…
