Rohit Krishnan asks that question at Strange Loop Canon. He starts with a story:
Robert Owen was a Welsh industrialist. Born in Newtown, Owen moved to Manchester when he was quite young, and quickly rose to prominence as a manager in the cotton industry.
He was self-educated, reading everything that he could get his hands on. He was particularly impressed by the enlightenment thinkers, from Locke’s theories on human development and the importance of education to Rousseau’s ideas on natural human goodness and the corrupting influence of society. They resonated with Owen’s belief in the perfectibility of human beings through environmental improvements.
His success at New Lanark Mills in Scotland, where he implemented progressive labor practices such as fair wages, reasonable working hours, and comprehensive education for workers' children, demonstrated his commitment to improving social conditions through rational management.
After he had his economic success he wanted to put his grand theories into play and create a true utopian society.
So he created New Harmony, in Indiana, aimed at demonstrating the feasibility of social reform through cooperative living and education. He brought about the smartest people he could find, artists and scientists and intellectuals, to make New Harmony incredible.
It worked for a while too, even though its story ends in decline. New Harmony faced economic difficulties, internal disagreements, and lack of enough practical skills among its members.
There are at least a couple interesting things here. One is the fact that New Harmony seemed to have a strong undercurrent around things we consider highly modern, around equality and collective living and education.
The second, which is arguably more interesting, is that they clearly believed in the perfectibility of man. Some combination of the christian belief with the American belief with the Enlightenment belief combined to say “we can be so much better”. They truly believed that with individual effort we could create a better society. And what's more, they felt they could demonstrate it, and did just that by gathering up followers and heading to the middle of America to show the world.
Krishnan tells a few more such stories along with this and that. I urge you to read the whole thing. He published three charts which I find useful. I'll put them here without comment (click on them to enlarge):
Here's his final paragraph:
The spirit of pioneering ambition is perhaps our best features as a species. We only seem to get it in glimpses as you glance through history, seemingly at random as if a capricious muse bestows it on us, outside our control. But this isn’t preordained, nor is it true. If audacious optimism and a focus on utopian outcomes is achievable, then not doing so is a fault of the spirit. It is worth asking why this form of optimism is no longer around.
I made a long comment, which I append below the asterisks.
* * * * *
First, thanks for the charts, Rohit, they're most useful.
Beyond that, yes, no, not really, we can't get there from here.
Let's look at your last paragraph. Your last sentence: "It is worth asking why this form of optimism is no longer around." YES. But I've got doubts about the first sentence of that paragraph: "The spirit of pioneering ambition is perhaps our best features as a species." I know what the words mean, and I have half-a-hunch what you mean by the sentence, and I'm not buying it.
I live in an intellectual world where evolutionary psychology is important. I don't buy it, but I take it seriously. In THAT world your sentence is most easily read as an assertion about our biological nature. I don't think that pioneering ambition is somehow in our genes. But, for example, I'm willing to believe that an exploratory urge is in our genes, and not only ours, but in many animal species. And I'm quite willing to believe that that urge enters into the construction of pioneering ambition.
What do I mean by that, construction? As a crude analogy, think of chess. On the one hand we have the rules of the game, the 8 by 8 board, the different pieces, and the legal moves. You have to know that in order to play the game at all. Think of that as analogous to our biological nature. In order to play even a halfway decent game of chess you need to learn some tactics and strategy; those things are defined over the basic rules, but cannot be reduced to them. That's cultural evolution.
And that's where we get pioneering ambition. That takes us to your next sentence: "We only seem to get it in glimpses as you glance through history, seemingly at random as if a capricious muse bestows it on us, outside our control." But it seems to crop up most strongly in the late 18th and early 19th centuries. I once asked my humanist colleagues about that: When did we start thinking about the future as a time and place when things would be different from what they are now, when they would be better, and made so by effort we can undertake now? That's what they told me, late 18th century. The American Revolution started in 1776, the French Revolution a decade later. Note that the American Constitution contained a procedure for making amendments because the framers believed that the future would be different in consequential ways.
Why then? You mention the Enlightenment. Yes, but it didn't come out of nowhere. There's a history there, a lot of cultural evolution. The same with the Industrial Revolution. How are the cognitive underpinnings of those developments constructed out of the "raw stuff" of biological human nature? Hardly anyone is even asking the question. But it's the sort of thing David Hays and I gave careful thought to in elaborating our theory of cultural ranks. That's just the barest beginning but it does provide a way of thinking about how something like pioneering ambition might be constructed.
Beyond that, well, it's complicated. Walt Disney was optimistic about the future and perhaps did more to promote the idea of a technology-fueled quasi-utopian future than any other single person. You can see his vision in a short promotional film he made in 1966, a few months before he died. In that film he lays out his vision for Experimental Prototype Community Of Tomorrow (EPCOT). What actually got built as EPCOT is far less interesting. After Walt died the world that had nurtured that vision fell apart, though not because Disney died. For all sorts of reasons, some of which you mention.
We can't go back. We need to figure out a new vision (or visions) for the future. One way I have of thinking about that is something I call Kisangani 2150. Why that name? It's derived from New York 2140, Kim Stanley Robinson's novel about the world in 2140. It's neither utopian or dystopian. But it's believable. My idea is to take the world, more or less, run it forward 10 years and center the story in Kisangani. Why Kisangani? Because it's deep in the Congo Basin, which is a very different part of the world from New York. Also, there's literary resonance. Kisangani is the location of the Inner Station in Joseph Conrad's Heart of Darkness. That, as you may recall, is where Kurtz went mad. And there's music, which I explain in the post where I first entertained the idea: Kisangani 2150, or Reconstructing Civilization on a New Model.
Basically, we have to rethink EVERYTHING. No time to waste.