Crisp tasks are more reasoning/system-2 based. Whether a response is good is typically precisely defined. Reasonable, knowledgeable people don’t dis

Musings on the Alignment Problem

submited by

Style Pass

2024-11-22 21:00:02

Crisp tasks are more reasoning/system-2 based. Whether a response is good is typically precisely defined. Reasonable, knowledgeable people don’t disagree about it.

Fuzzy tasks are more intuition/system-1 based. Whether a response is good is typically somewhat vaguely defined or could fall within a range. Reasonable, knowledgeable people may disagree about it.

Examples: distinguishing cats and dogs in pictures, recognizing strong go moves, poetry writing, assistant helpfulness ratings, …

In contrast to fuzzy logic, where this name is borrowed from, here crisp and fuzzy don’t refer to the output space: preferences comparisons have a small number of discrete allowed values, and image classification typically uses a discrete space. Crisp tasks like coding competitions have a output space that involves hundreds of discrete tokens, and this is the same output space of poetry writing, a fuzzy task.

There are many crisp tasks that we can evaluate very reliably, and thus it’s feasible to train on them extensively. In contrast, fuzzy tasks are often particularly difficult to evaluate reliably, and we currently don’t know how to do this at a superhuman level. 1