Progressive ideology relies on the capacity of human beings to live fulfilled lives in a just and co-operative society. That people whose beliefs imply optimism seem to spend most of their time wallowing in pessimism is one reason that leftists sometimes lack personal credibility (another reason being that egalitarians so clearly enjoy being very well-off). But miserable idealists need to make a New Year resolution to look on the bright side. Pessimism is becoming an impediment to progressive politics. It is 50 years since J K Galbraith coined the phrase "private affluence and public squalor"; today, the dichotomy is between private hubris and public pessimism.
It is pessimism of a particular and pernicious kind. People are not generally negative about their own lives. In fact, we systematically exaggerate the control we have as individuals. As Malcolm Gladwell, among others, has shown, we tend to give our conscious minds credit for many reactions that are in fact instinctive. Other studies - of what we say has made us happy and what has actually increased our levels of contentment - show that we have a huge capacity to rationalise our life choices. When we are forced to make a choice between limited options, we are as likely to end up claiming the choice as our own as we would if it were unconstrained. And the more we like a future possibility in our lives, the more inclined we are to believe it will happen. The human mind is hard-wired to be personally Panglossian.
In contrast, we are unduly negative about the wider world. As a government adviser, I would bemoan what we in Whitehall called the perception gap. Time and again, opinion polls expose a dramatic disparity between what people say about their personal experiences and about the state of things in general. Take attitudes towards public services. In a recent poll, 81 per cent of respondents said that they were happy with their last visit to hospital. Yet when the same people were asked whether they thought the National Health Service was providing a good service nationally, only 47 per cent felt able to declare it was so, and most think the NHS is going to get worse.
This perception gap is not restricted to public services, as a recent BBC poll on families confirms. Some 93 per cent of respondents des cribed themselves as optimistic about their own family life, up 4 per cent from the previous time the survey was conducted, 40 years ago. Yet more people - 70 per cent, across race, class and gender - believe families are becoming less successful overall. While we apparently thrive in our own families of many shapes and forms, as social commentators we prefer to look back, misty-eyed, to the gendered certainties of our grandparents' generation.
What is true for families is true for neighbourhoods: we think ours is improving while community life is declining elsewhere. We tend to like the people we know from different ethnic backgrounds but are less sure about such people in general. We think our own prospects look OK but society is going to the dogs.
The media seem to be the most obvious cause of this phenomenon. Bad news makes more compelling headlines than good. Tabloids and locals feed off crime stories, middlebrow papers are dismayed at the chaos of the modern world and the alleged venality and ignorance of those in power, and left-leaning broadsheets enjoy telling us that global instability is endemic and envir onmental apocalypse inevitable. Mean while, the content of television programmes - from dramas to news bulletins - contributes to what the communication theorist George Gerbner called "mean world syndrome": people who regularly watch TV systematically overstate the level of criminality in society.
Yet it is too easy to blame the media; the job of commissioning editors is to give us what we want. We make our own contribution to social pessimism. In the burgeoning industry of reputation management, it is generally argued that people are much more likely to tell others about bad experiences of services than good ones (5:1 is the usual ratio). Academic research suggests that people tend to exaggerate in the direction of the general mood. Viewing our own lives positively but wider society negatively, we will tend to pass on and exaggerate evidence that supports these prejudices.
Evolutionary determinists may seek an explanation of our predilection for bad news in neurological hard-wiring; perhaps, for the survival of hunter-gatherers, warning is more important than celebrating. But it is in two of the mega-trends of modernity that more likely reasons for our social pessimism are to be found.
First, there has been the inexorable rise in individualism since the Enlightenment. As Richard Sennett brilliantly argued in The Fall of Public Man, aspects of modernity such as the power of consumer capitalism and the ubiquity of the idioms of psychotherapy have accelerated the process by which we see our authentic selves as revealed in the private and personal spheres, rather than the public and social.
Hand in hand with the rise of individualism, we have seen the decline of industrial and pre-industrial collectivist institutions, including the organised church, trade unions, political parties and municipal elites. Robert Putnam's work on social capital suggests this decline in collectivism reaches down into our social lives, with people choosing to spend less time with acquaintances and more with intimates. Putnam's more recent work controversially argues that trust levels are lower and loose social networking less common in more diverse communities.
This points to the second of modernity's mega- trends. Increasingly, we feel that we are the victims of processes set in train by human activity but no longer under anyone's control. Globalisation is the gravity of modern society: an unstoppable force that will knock us over if we try to defy it. The origins of the current credit squeeze in the US sub-prime mortgage market show a financial system that is beyond not only its managers' control, but even their capacity to chart.
Illegal immigration, terrorism and pandemics are seen as the inevitable flip side of cheap travel and consumer goods. Philosophers and policy-makers argue about how best to regulate emerging science and technology in genetics, nano technology and artificial intelligence. But can anything long delay the advance of knowledge - especially if it has commercial applications?
