A Heuristic Proof of Practical Aligned Superintelligence
In my previous post I showed that given a reasonable definition of alignment it cannot be the case that aligned AI doesn’t exist. The argument is really quite simple: if you can define it (and your definition isn’t impossible in-principle even by the best possible team of humans) then there must exist some boolean circuit/finite state machine that implements it.
A number of objections were made to that post but they all stemmed from people who either didn’t understand it or were quibbling with the definition of “intelligence” or thought that a non-practical existence proof is not worth anything. First we’ll review the following: a nonconstructive proof is still a proof.
For example the proof that there exists a rational number (fraction) q, which is equal to an irrational power of an irrational number. What is q? Which specific fraction? We don’t know. But we know that q exists. It’s easy to prove:
So let’s say that a as defined above is rational. Then we can say that our fraction q, is simply equal to a. Which specific fraction is it? What is the numerator? What is the denominator? I don’t know. But, by assumption it is equal to some fraction! OK. Now let’s suppose that a is not rational. Then a to the power b, which is equal to 2, is rational (2 is a fraction in the technical sense, it is 2÷1, as are all whole numbers). And we know that b (which is just the square root of two!) is irrational. So in that case our rational number q that is an irrational power of an irrational number is just 2.
