Use tanh instead of exp in sigmoid function

less than 1 minute read

If the data is large, we encounter overflow in sigmoid function.

A simple trick to avoid overflow is to use tanh.

In your code,

y = 1 / (1 + numpy.exp(-x))

will become

y = numpy.tanh(x * 0.5) * 0.5 + 0.5


Reference 1
Reference 2

Updated: