If you ignored my arguments, and only focused on my conclusions, you would be forced to compare me to people who make claims that are “similar” in some relevant way, and then note how often people who made claims “like that” were correct. And you would find, before hearing my arguments, given your state of knowledge, that there was a higher chance that my argument would fail somewhere, simply because it sounded apocalyptic. And that is valid. However, if you’re actually inspecting my argument, the track record other people you think I share some trait with has nothing to do with this any more.
But that’s probably not what’s hanging you up. What’s probably hanging you up is that you identify, by-and-large, as a normal person with normal beliefs and concerns. And legitimate concern about extinction from poorly made artificial intelligence does not feel like a concern that befits a normal person like you. (As a matter of fact, plenty of people are concerned about this, but I get that it still doesn’t feel normal.)
I mentioned before how we have higher standards for evidence for things we don’t like. Not liking something doesn’t have to be about not liking the consequences. In fact, I don’t think it usually is. I think it’s usually that some piece of our identity is wrapped up believing X, and that’s why we don’t want to believe that X is wrong.
For instance, I think that among many of my friends, a large part of their identity requires that they believe that stricter gun restrictions in America will result in fewer deaths. I happen to believe they’re right. However, it used to be that a large part of my identity forced me to believe that, and so I gave more weight to points like “Guns kill people; ergo fewer guns means fewer people killed,” than to actual statistics that seem, on the surface, to contradict that. Had I continued to weight evidence in accordance with my desired liberal identity, and had the balance of evidence come out against stricter gun restrictions (which it could have!), I would have ended up wrong.
It is very dangerous to let your conclusions and predictions be decided in advance by a desired identity. I think this feature of human nature has already been responsible for billions of deaths, and I hope it’s not responsible for our extinction.
Extinction Risk from Artificial Intelligence 