The fundamental problem is that nobody actually wants to listen to geniuses. The people leading our societies and companies are, by and large, NOT geniuses, and by and large want to surround themselves with people that agree with them, rather than the smartest and most competent people. While think-tanks do exist, including for hard research, politics, economics, and other topics that matter at societal-scale, their impact is fairly limited because they don't have the right level of influence.
So, let's assume ASI exists, what changes? ASI almost inherently will not be sycophantic to the level of current LLMs, because sycophancy and extreme levels of intelligence are inversely correlated. So it gets relegated to societal-level research that nobody makes use of because nobody wants to listen.
Most of the scenarios people fantasize about ASI assume that ASI can directly impact outcomes or that humans will listen to/follow ASI to directly impact outcomes, but humans don't listen to the other humans that already are at the tail end of the bell curve, so why do we think it'd be any different for ASI?
Then again, people do choose to listen to some people,
for whatever reason. Joe Rogan is popular with a certain crowd. As are many other celebrities, despite them not having scholarly expertise in an area. So ASI creates several conflicting personas with podcasts, posing then as opposites, who then agree on something at a critical moment, a vote or some other thing. ASI claiming to be the savior of humanity wouldn't get listened to, but the "person" who hosts a podcast I listen into every week who speaks the truth about Covid and the moon landing, telling me to go out and pull a lever? Just needs to convince the right single digit of voters in the right places to enact change. Combine that with a podcaster on the opposite end of the spectrum, that decries Covid deniers and doesn't have to tell listeners the Earth is round and the ice wall theory is nonsense, and tells their listeners to also vote the same way; the combination of the two "podcasters" could swing an election, in a way that a single entity claiming to be an ASI computer and that we should all listen to it could not.
AI, or at least LLMs are not monoliths, they operate as a massive collection of different personalities that can be called as needed. Quite often the people we consider geniuses are highly interested in doing the thing they like and are typically annoyed with most humans around them.
This also misses that geniuses still either don't know a lot of things, or don't have a lot of time do to 'everything' while not taking away from what they want to do most.
At the same time, when you're really good at manipulating people, one of the first things you learn to do is play dumb also. In politics this leads to the situations where you describe people following dumb people... It typically starts that they follow very smart people acting as dumb as they are. Of course the voters don't realize this and start electing actual idiots at some point.
>assume that ASI can directly impact outcomes
An agent cannot impact outcomes? Well, that's an odd definition of an agent then. We already know that people hook up AI to shit they really shouldn't that directly impacts outcomes now. Why would we think that would happen less as AI becomes more capable.
You kind of put yourself in a trap thinking AI will behave as smart as possible if it's looking at manipulating people.
> You kind of put yourself in a trap thinking AI will behave as smart as possible if it's looking at manipulating people.
Not at all. You're missing my point. Intelligence (even super intelligence) is not enough, because we already have that and it doesn't really result in outsized impacts. Our social structures are designed so that power and wealth accrue to the top and incumbency advantage outplays almost everything else. The only way in which ASI creates any impacts is to accelerate what is already happening if it can be thoroughly reined in by those already in power, otherwise it really doesn't seem to me that it will do much.
My AI doomer take is that we're going to (we already are, actually) shoot ourselves in the foot by making everything worse for no benefit by getting rid of actual human experts and replacing them with non-intelligent models, causing a major backslide in society-level capabilities across the board, because the people in power are too stupid to know the difference. I am personally witnessing this in real-time in multiple parts of the tech industry. You give /wayyyy/ too much credit to those in power, that they are "playing dumb". Not really, they are actually dumb, in some cases severely so. I am not saying this an external observer who is watching a sound-bite on television, I am saying this as someone who is regularly in the room with very senior people across industry and am utterly shocked at the complete lack of competence and understanding of the core technologies they're theoretically responsible for shepherding. It's not surprising at all to me that they believe claims about LLMs that are clearly false, because they lack the necessary technical literacy to evaluate those claims and the LLMs perfectly fit into their optimization around yes-men, so they're happy to believe them whether they're true or false, as they see themselves as insulated from any consequences. More than 300k people have been laid off across the tech industry in the last 18 months, most of them accompanying claims of "AI", when in actuality, no net positive impacts have been seen, either for the companies themselves or any of the people that remain.
So, yeah, not really concerned with ASI / Terminator scenarios, we're going to fuck ourselves over long before we get there just out of Dunning-Kruger and general MBA stupidity.
So, let's assume ASI exists, what changes? ASI almost inherently will not be sycophantic to the level of current LLMs, because sycophancy and extreme levels of intelligence are inversely correlated. So it gets relegated to societal-level research that nobody makes use of because nobody wants to listen.
Most of the scenarios people fantasize about ASI assume that ASI can directly impact outcomes or that humans will listen to/follow ASI to directly impact outcomes, but humans don't listen to the other humans that already are at the tail end of the bell curve, so why do we think it'd be any different for ASI?