Rationality: From AI to Zombies

by Yudkowsky, Eliezer

Published: March 11, 2015

Read: December 30, 2019

★★★★★

Goodreads

Review

So, I have basically spent the last 4 years reading and rereading this amazing body of knowledge and finally reached the end. Truth, intelligence, cognitive biases, how do you make better decisions and believe true beliefs? The author is a genius AI ethics researcher that really, actually needed an accurate model of the world, information, and morality. In this pursuit, this body of knowledge has so many valuable ideas I have never seen before beautifully and lucidly explained. There is literally a before and after I started reading this as it fundamentally changed the way I view the world. Saving myself so many times from bad trains of thought and giving me foundational ideas to more accurately understand the world, myself, and others. I absolutely love this! However, I think I am particularly attracted to a life philosophy based solidly on truth and reason, so I doubt everyone will appreciate this as much as I do. Like a cult fanatic though, I hope you will convert to the truth! This is actually a series of >350 blog posts found here: https://www.lesswrong.com/rationality

Notes

Machine in the Ghost:

Amazing evolution works at all. 3 percent advantage takes like 780 to reach entire population. Forever. A grad student could do an hour what evolution took millennium. Don't give evolution more respect then it deserves.

Evolution requires many things like limited resources, heredity, death of generations. About competition of allele frequency not individuals. About relative fitness, so frodo savior gene of entire species is actually worse for individual relative fitness. Gender balance equal even though you need way less men then women because men are half genes of next generation. You could totally evolve to extinction. Seek group benefit over individual like a virus that kills so fast it doesn't spread. Group saving must confer it's benefit mostly to altrusist. With such a large probability space, most of evidence is from just suggesting a hypothesis.

We are adapter executors than fitness maximizors. Human adults asked about potential grief and highly correlated with reproductive potential in ancenstral environment. Correlation of .92.

HiEvolution cars about inclusive generic fitness not survival. Made brains and had them pressure the learning done. When it made brains, it had them figure out how to chase them instead of fitness. Not smart enough to refactor. Imagine if we started biological to long, imagine if we were this smart but only cared about inclusive generic fitness. 1000 delicious tastes that once correlated with survival. Evolution sharded into 1000 shards of desire. And now they one is boring and doesn't fit our taste.

Fragile purposes

When interacting witb others, we can predict outcomes by people's beliefs. #trust.

Aliens in funny suits cuz convergent evolution. Yea right, it's just good for media. Similar values and ideas too wow. To write, you must see your culture as just a special case. You don't know how other people mind exists, you just stimulate things in your mind. An Allen mind if would take so much time and knowledgeto understand.

Humans received constant source of optimization and took off in intelligence after writing. We could expect ai to be similar.

Intelligence is hitting a subspace of future realities(planning( or deisgns(inventions)

If you don't know if it will work, it probably won't. Emergent ai from just learning from website, prob won't work if you don't know why. Hitting a small target space is hard.

Killing someone is always locally bad. There are instrumental and terminal values. As a terminal values locally killing is always wrong and the net effect just means the other outcomes outweighed this heavy wrongness. Policies aren't one sided.

Terminal values, utility. Vs instrumental values and expected utility.

Part N Guide to Words

Syllogism like Socrates sis mortal. So he will die if he drinks hemlock. This is valid in all possible worlds and so doesn't tell us what world we live in. Mortal might have to certainly mean he will be mortal which means he will die if he drinks hemlock. So we are just confused by what mortal means if he didn't die. Logic is true whenever you go and do doesn't tell you where you are.

Disguised query. When people ask did it make a sound? They are asking if people heard it or if there were vibrations in the air. The word is it a sound is a standing for this.

Dictionaries are not legislators of language, just recorders of usage. Words are telepathy for communication, but it doesn't really matter we all agree about the word if we know what the other peoppe mean by the word. Then we can say different words, and translate with effort.

Instead of I can define a word anyway j want or without consquences, say definitions don't need words. Artistolean words are just sets of definitions .

Confusing if labels intersect, but not if they go down to definitions and play taboo. Can just keep playing taboo. Likewise Real conflicts might be hidden when people use the same word, but differ in expectations.

tabooing words allows: Rationalist ask why and what are the expectations. Drop down to lower levels of organization(reducstism), avoid cached thoughts, not losing sight of purposem.

The map is not the territory and your mind is so small of course it can't capture all the details. One thing to question the facts another thing tk question your ontological buckets of thought. Like q detective realize they are twins and then use a different word for her.  Things in the same bucket he categorical attributes and you lose details when you think of the the same.

Categories exert for on your ind. So there re consequences to define words.

When people argue by definition, if they actually are just trying to snuggle in Constitution. Wiggens are black haired. He's a wiggens because he has black hair therefore she has black hair, pretty empty right.

179, 180 chapters skipping

182 also skipped

Words are useful as a concept that helps with probability distributions. Raising that concept to our awareness should be intentional. Can't just make up words.

Check out the great summary of 37 problems with words

Bayes rules intuitive explanation

Hi

Lawful truth

If a match doesn't lit, we would all die because it would change everything down to the atom electrodynamic level and we wouldn't be able to breath because we use combustion to breath.

Reality more laced together than people think.

Universal law like Newton. Leaky categories aren't absolute. We hope moral law for dividing meat. Moral law need exceptions, but universe is rigid. Plato didn't invent science because idea you can generalize from contrived experiments is not obvious. Nothing unusual has ever happened.

3 sources of uncertainty though we in math world: we can live in math without knowing it, computing power. 3) if you on cubes and people on cubes, you can look down from above and know everything. But if you are person on cube, you need to know where you are. Initial conditions. We know physics though. If we are uncertainity about a phenomenal, that is a fact of our mind.

Should we hope for simplicity in rationality? Two scientist do experiment exact same results. One decided initially to test 100, another decided to test until effectiveness>70. Both test 100 and same results. Stats says the experiments tested definitely things because of exploration party. Bayes says no, they are the same. Bayes look for self consistent, not dependent on mind. Frequent ists vs Bayesian. But don't statistics just have a strictly bigger toolbox, reality is messy isn't it. Success is how close you are to battery way, computing hard though. Never more accurate than Bayes whatever you do. Longest regression had guardian distribution.

Proverb: scientist no smarter pride the lab. This should disturb us. Sheepherder can't count apples, only sheep. Economists bus lottery tickets. Experiments essential to make a map you have to look at territory with experiments. To find things out you have to look.

191

To map a territory, your have you undergo a casual process by which the map is correlated with the territory. If scientists believe in religion, then they don't really understand why just follow tradition of science.

Law 1 of them Dynamics: any particle can't create energy so no system can

Law 2: Phase state is preserved. There is therodynamic entorphy. And informational entrophy. If you know about the exact positions of atoms of a cup you can be Maxwell's demon. It works because the entrophy of the system is highly correlated with you so there is mutual information. If you have a system correlated with another you can do thermodynamic work. Our mind has the same process. If faith worked, we could use it make Maxwell's demon. Just because it is probabilistic, you should expect nearly impossible things to not happen.

Everything that can help you make decisions is just approximating Bayes. There must be real therodynamic work being done. Knowledge comes from a relationship between your mind and reality that is causal. So there must be some Bayesian evidence.

Reductionism 101

Beginner philosophers and old ones often make the mistake that they have to answer questions. Like do we have free will? Proving that it is a meaningless question is a different route. But why we like free will is an interesting Austin about cognitive science. Saying it was beneficial to ancestors doesn't answer the question. The mistakes we make can tell us so much about our mind

Wrong questions, what state of affairs in reality would correspond to a good answer. Why do things even exist? Dissolve the question to find how mental question is wrong in the beginning. Ignorance exists in the mind.

Why do I have free will? => Why do I thinki have free will? Actually has an answer. Psychological events are events.

Mind projection fallacy. Women is inherently sexy so alien takes her for ravashing. But, sexiness isn't property of the women but our mind. Probabilities in the mind not reality.

Against frequemtist who says probabilities are state of object that reflect the distribution of results. So argument if an unfair coin and you don't know how unfair, you would still say it's 50%. Not mixing lottery is unfair someone says. Unfair to who? Is your child a boy? Yes, 1/3 other is a boy. Is your eldest child a boy? Yes, 1/2 other is boy. Probabilities are states of partial information about things, only agents are uncertain.

Quotation is not the 912-914

Belief and truth is very different. If there was a word that meant believe falsely would have no first person present usage. Better to talk about beliefs as accuracy. Reality is perfectly normal, you think it is incredible means you need to change. Surprise is a measure of poor hypothesis. If y"ou can't understand why it how", too express indignation. If you actually don't understand, you are letting this indignation stop you from thinking about human psychology.

Saying I'm ignorant Dinesh make you knowledge, but it gives you another payment. Chaotic productivity schedule might be my stupidity.

Reductionism is obviously correct. Sure quantum could be supplanted, but brain make is never coming back. Just because our model of airplane calculation doesn't have quality doesn't mean there are only quarks. It often feels you are looking at reality, but you are looking at beliefs. Reductionism is disbelief that the higher level model exist. Difference between explain and explained away. Rainless exist, but gives in mines are explained away. I believe that I'm wearing sucks probably because I'm wearing socks. There's isn't a model of rainbows in reality doesn't mean you should abandon your model of rainbows. That which can be destroyed by the truth should be.

Anti reductionist think explain something makes it boring. ;(. But raindrops don't dance. God as human can make us feel and understand more though. Humans are like humans, we still got hate and love and murdee don't worry. We don't need a murder story to explain rainbows. Some day great stories are timeless and just retold. But humans aren't timeless, we can do better.

Joy in the merely real

What is the normal, mere. The real. That is setting us up for failure. Nothing is interested mysterious. Thrill of scientific Discovery is so high. But the exclusively is weird. The idea that someone else knows the answer somehow saddening. Why? Perhaps it because you worked harder and helped civilization. But, prob about status. But that means it's limited, Newton that selfish bastard who solved so much. But in the huge universe, someone probably already knows anything. So stop worrying what others know, if you don't know it's a mystery!

Don't reject dreams, just impossible one's. Bind emotions to reality without flinching away. That which is restored by the truth should be. Physics isn't weird you are. Directs your emotions into the universe

If you demands magic, magic won't help. Fantasy stories have things that don't exist, grass always greener on the other side of reality. Dragons would just be merely real. If you aren't happy now, you wouldn't be happy in the future as it also has a future to look to.

Imagine. Vibratory Telepathy, send invisible vibrations through the air allowing two users of this abilities to share thoughts and form very intense emotional relationships. Seer can chat complex paths through the world to achieve arbitrary goals. It competely destroys gambalance. All other lifeform are pawns to this aristocracy

Beauty of settled science: press science with contrevety is probably untrue. But in a science book you got highly confirmed science with practice problems and clear explanations.

Amazing breakthrough day. Journalists explain real breakthroughs that didn't happen yesterday that people can grasp. Reddit doesn't have just new things.

Is humanism a new religion? Perhaps similar brain reactions between ticket launch and religious hymen's. But not a new religion. No religion shaped hole. It would exist even if we didn't bake the original mistake.

Scarcity: when things become more desirable as they become less attainable. After phosphate detergents were banned, the residents rated it much more likely. Learning speech opposing coed donors bannedstidents opposed coed dorms without hearing the speech. Preservation of status and options motivated. Leaping on disappearing options. Dream more of possession rather than using it. When planning your life, ask what ongoing experiences you enjoy, rather than possessions or status changes you would enjoy.

Isn't awe universal? Why is religion sacredness so private and thus unstudyable. Retreat tp commitment. Mysterious became associatedd with saved. Religion is poisoned chalice with many bad thought patterns. Don't try to save face. Say you were completely wrong.

To spread science keep it secret: much more likely to sell information that is scarce. But science is left out in the open. To keep science secret, make it freely available. Why would I join your cult when physicists can do so much. If your allowed to learn, it must suck. Don't undervalued science just because it is not protected.

Physicalism 201: reductionism says we use multi level models for computing reasons, but physical reality has only a single level. Hand is idenityt of fingers, thumb, or palm. But conceptual possible that hand moves independtlh of fingers, palm, and thumb. Ai would be confused.

So we have billlard balls making up everything. But how do you get angry behavior from bollard balls. In the past that was a huge explaintory gap in the past. You need search trees and min max and utility functions.

In the past, you had heat and motion as separate ideas in science. Look someone could say we have two different words for it because they are different things sure our beliefs are different things, but maybe not reality. You can't just examine your beliefs to think about it, you have to look. Conceptual space is much larger than logical space. Water is not H20 conceivable, but not logical possible. If someone said we lied in science, heat and motion aren't different. Could I prove them wrong?

Old dualism is well old, we have more evidence now.

Animalism: humans have thoughts. But why wouldn't rivers also believe things. Not stupid idea, if it was no one would believe it. Moral behaviors from more complex tribal political arguments.

Aprior: moral means you violate it and your defecting. Rationality your brain is an engine and if you violate laws the engine doesn't run whether anyone is too. Philosophical reasoning different dates between aprior and evidence. But thoughts are in the world. Is Occam's a reasonable aprior? Did it work well in the past, well what if it stopped that is more complex but you gonna appeal to Occam's razor circular.... When pure thought tells you 1+1 you are in effect using your brain at evidence. Aprior thoughts you could know with equal validity by looking at someone's else brain engine. Regularized algorithms bias simple, but are more complex algs. We seem to live in a simple world, who knows why. Perhaps you can't argue anything to Occam's razor, if your mind doesn't implement modus pones how do you argue it into evidence. Rationality is more an engine rather than arguments.

Reductive reference: multi models useful, but reality uses quarks or whatever. Can still define a steam engine with just looking at heat movement. So am I wearing socks. If only quarks exist. Snow is white is true if and only if snow is white. You need to look. Words like a empirical determined ptr. Basically a promisory note. Patterns of hands, but they aren't all of reality.

Zombies. Exactly like humans, except not conscious  You aren't the one who thinks your thoughts, but hears them. Consciousness must be extra physical then, as you would have to know someone was conscious as an extra fact. But then it would have no effect on this world. A zombie world would have to be identical. Philosopher being a zombie would also have an internal marratvir about consciousness. A zombie master would be having a physical measurable effect on the world when he controls people. Another perspective of decarte, consciousness has additional properties beyond physical that don't have observable impact. But, without it we wouldn't feel conscious. Seems more consistent because you have one consciousness mysterious that invokes you to talk about consciousness. Instead, we got the immeasurable soul and then a separate cause that we think we are conscious. Need a reflexive coherent theory of mind because a self modifying ai would correct any belief that usually produces false beliefs. No way to differentiate between our universe and the zombie universe if no physical influence. This theory is dominated by Occam's razor of decarte dualism or the more likely we are again making the human mistake of saying something we don't understand is inherently mysterious. When we find out the physical reason we believe in consciousness, we will get insights we can't yet predict that might allow consciousness to exist in physics.

Zombie response: Seems fairly straight forward to define an AI that can map reality in purely physical ways. Ai would endure global rationality by enduring local. Modularity makes the problem more tractable. 2+2=5 only works with sheep, it's won't locally but it would make this discussion case wrong. A cheap self reflexive coherence.

Generalized antizombie principle: since we write about conscious, that should show consciousness has physical effects. We could say consciousness could be the cognitive process producing the feeling of consciousness. We have a switch that someone says when flipped removes your consciousness. Generalizing the arguments try 1: need in principle measurable impact(IPMI) on you to remove consciousness. Well no, switch has gravitonal pull that moves your atoms by neutron diameters. But, thermal noise is more so you shouldn't expect it to. You should expect to keep believing in consciousness for mostly the same reasons. While this less of a slam dunk and no absokutez the rules of probabilities and uncertainty are theorems and laws. Consciousness is not being requested to talking and consciousness, but it is one of several factors that would make you notice consciousness go out like a light when someone sneezes. Both substance finalists and reductionist could argue this.

Some argue religion shouldn't be taught with science because science deals withd ifferent things aprior. Probably just to not deal with the fact science proves religion wrong aposterior. What predictions would a no reductionist idea make, if a airplane didn't follow the mechanics of the quarks. What would you expect to see? How would that work? Supernatural is something that has mental entities that can't be reduced to more physical things. If our minds are made of quarks, then we can't imagine or compute things that can't be described by quarks. Our mind is quarks we think through quarks so quarks can calculate anything we can imagine. I don't understand, but somehow therefore we can't even imagine supernatural irreducible things. Reductionism is just against fundamentally complex things. If you believe in religion and are developing general aI, you are making a huge mistake.