Whatever your reason is why an AI couldn’t do X, I assume it has to do with the specific path I laid out for getting X. Let’s suppose you’re right that an AI couldn’t follow that path to X. We’d still be a long way from saying that there was no possible path to X at all. I’m sure you can come up with another path to X, besides the one I suggested. There are so many ways to skin a cat.
Or maybe your concern has to do with the assumption that people would notice if the AI came close to getting X, and could respond in such a way as to make sure the AI never did get X. Again, if you spend 30 seconds trying to formulate a strategy for an AI to get X without being noticed until it’s too late, I bet you can.
Extinction Risk from Artificial Intelligence 