Make no mistake, every time I read a news story about all the time, effort, and investment being poured into building larger and larger generative AI models, LLMs in particular [like
today's post about data hunger], not to mention another technical paper, I wonder whether or not they are right and I am wrong. The fact the no one really knows, and so one must be open, weighs on me, though as far as I can tell, it doesn’t weigh on THEM at all. They’re sure. They have to be if they’re going to put all that money into it. Or is it the other way around, it’s the act of investing that makes them sure?
I’ve got explicit arguments and knowledge on my side. They’re by no means conclusive. But they are not nothing either. Nor am I alone in my skepticism. Yet, the roar of the true believers is a distraction.
If you don’t have any explicit arguments against the power of scaling, if you don’t already know a great deal about perception, cognition, and language, the lure of the crowd must be overwhelming.
