MRSSE | The ryg blog

submited by
Style Pass
2024-11-15 11:30:14

For BC6H encoding in Oodle Texture, we needed a sensible error metric to use in the encoder core. BC6H is HDR which is more challenging than LDR data since we need to handle vast differences in magnitude.

BC6H internally essentially treats the float16 bits as 16-bit integers (which is a semi-logarithmic mapping) and works with those integers throughout. It’s a bit more complicated than just grabbing the float bits, but not in a way that really matters here. The most straightforward approach is to do the same thing in the encoder, pretend we care about those 16-integers, and just work with squared errors etc. on those, which is easy to do and also quite fast.

This works reasonably well with white(-ish) light and not very oversaturated colors, but degrades horribly especially with bright, colored emissive light. My colleague Jon Olick wrote a post showing some examples back when we originally released Oodle Texture. The problem with working on log-space RGB data boils down to this: say we have some bright red glowing thing and the RGB tuples for a pixel (in whatever scaling we choose) are something like (2.34, 0.02, 0.01). Working with the logarithms per-channel in this scenario tells us that getting the value in the green channel off by 0.001 would be about as bad as getting the value in the red channel off by 0.1, so we end up producing large net brightness shifts to avoid imperceptible hue shifts, a bad choice.

We still would prefer something squared error-ish in the encoder core: the derivative of a squared error is linear, so minimizing a squared error by finding its critical points turns into solving a linear system. We find ourselves doing this hundreds of times for millions of pixels each, so a simple linear system is good.

Leave a Comment