# Automatic differentiation with dual numbers

submited by
Style Pass
2024-04-03 16:00:05

Differentiation is the heart of most machine learning, but how can we differentiate arbitrary functions? Perhaps the simplest accurate method is using dual numbers.

Now we want to ask: how does tweaking x = 3 change the output? In math-speak, what’s the derivative of the distance with respect to x?

The poor man’s way to answer this is numerical differentiation. We add a little bit to x, and see how much it changes the output:

We get that derivative = 0.5999999608263806. That’s 0.6. Well ... almost. The numerical error is due to our changeToInput = 0.00000001 not being infinitesimally small.

We’ll start by saying that ε, or epsilon, is a special number that’s infinitesimally small. More precisely: it’s not so small as to be zero, but it’s so small that when you square it, you get zero.

What is 42 + 7ε? Well, because ε is a different kind of number, we can’t simplify this expression, so we just leave it as 42 + 7ε.