This post is part of a series on bad abstractions in machine learning. For context on why we are writing these, read Abstraction Enables Thought. Bad

Bad ML Abstractions I (Generative vs Discriminative Models)

submited by

Style Pass

2022-05-15 06:30:03

This post is part of a series on bad abstractions in machine learning. For context on why we are writing these, read Abstraction Enables Thought.

Bad Abstraction: There are two types of machine learning models. Discriminative models are trained to separate inputs into classes, while generative models learn a distribution from which they can draw new samples.

The first example is a quintessential generative modeling problem, while the second is uncontroversially discriminitive. But I’ve just described the same model twice.

Given the set of natural-language sentences in the first scenario, we can construct a dataset in the following way. First, match each of the 64 possible characters (26 lowercase, 26 uppercase, 10 digits, space, and period) to a label. Next, take every character-level prefix of every sentence, and label it with the subsequent character.

Note that the dataset we have constructed matches the specifications of the second scenario. Training the second model on this dataset is indistinguishable from training the first model.