An Unserious Person’s Take on Axiomatic Knowledge in the Era of Foundation Models

submited by

Style Pass

2024-11-18 20:00:09

TLDR; This post tries to explain why we started the work that led to Evo, which recently was on the cover of Science–thanks to a host of wonderful collaborators at Arc [Brian Hie, Patrick Hsu]. These are some rantings about things that interest me about axiomatic knowledge and foundation models, and why I think foundation models for science are philosophically interesting. Do not take it seriously, I am not a serious person.

I’ve exalted the axiomatic and classical mathematical ways of describing knowledge; they felt true, deep, and interesting. I grew up as a math undergrad, and I remain astonished at how so much math follows from so few axioms. It’s stunning, beautiful, and interesting. But in AI, I’ve observed that more closely adhering to our intellectual siblings wasn’t always a recipe for more success. Foundation models are a tool that provide a new way to store and access knowledge and data: messy, imperfect, but also pretty fun. A common refrain is that the new systems must look like the old generation of these knowledge systems: clean, predictable, formally reliable to be taken seriously… but I’m not so sure… that didn’t happen with web search: we didn’t make it return exact results like a database system, but it’s still incredibly useful. One path is to make foundation models reliable, and there is a lot of interest in that–awesome! This is a path we’re excited about too. However, another path that we're pursuing is to embrace the messy unknown of the pattern matching tool we call foundation models. Despite our less-than-axiomatic understanding of these models, could we still use them as tools to benefit different areas? I’ll explain why Eric and Michael looked at DNA foundation models, and then try to articulate what I’m excited about next–and dress it up as something of philosophical import:

I hope that thinking about this question leads to areas where foundation models break, we learn something new, and hopefully we question some of our own dogma…