Automatic differentiation with dual numbers
Differentiation is the heart of most machine learning, but how can we differentiate arbitrary functions? Perhaps the simplest accurate method is using dual numbers.
Now we want to ask: how does tweaking x = 3 change the output? In math-speak, what’s the derivative of the distance with respect to x?
The poor man’s way to answer this is numerical differentiation. We add a little bit to x, and see how much it changes the output:
We get that derivative = 0.5999999608263806. That’s 0.6. Well ... almost. The numerical error is due to our changeToInput = 0.00000001 not being infinitesimally small.
We’ll start by saying that ε, or epsilon, is a special number that’s infinitesimally small. More precisely: it’s not so small as to be zero, but it’s so small that when you square it, you get zero.
What is 42 + 7ε? Well, because ε is a different kind of number, we can’t simplify this expression, so we just leave it as 42 + 7ε.
