Towards the end of the 19th century, and in the wake of Wagner’s achievement in Tristan und Isolde and Parsifal, the musical language which had been common property of Western composers since the Renaissance, underwent a crisis.

What we now know as tonality, which is the system of keys and scales, and the harmonic progressions, which had been accepted by audiences since at least the end of the Middle Ages, entered a kind of flux. Keys were no longer stable; dissonances began to resolve onto other dissonances (as in the Prelude to Tristan und Isolde), new harmonies began to insert themselves into the old sequences, and the scale expanded from eight notes to the twelve-note chromatic scale, using notes at random from other keys, and constructing sinuous melodic lines that seemed more adapted to dark and solitary emotions than to the cheerful day-light exuberance of choral song.

The crisis deepened during the first quarter of the 20th century, as a result of two striking innovations. The first was that of Debussy, anticipated by Liszt, who began to use the whole-tone scale (the scale without semitones). This scale, emphasizing each note equally, and being without a dominant, is directionless and lacks the dynamic tension of the traditional major and minor modalities. From the whole-tone scale new harmonies emerge – static, indolent, yet somehow not at rest. Debussy combined this scale with post-Wagnerian harmonies, in music which was guided entirely by his own sensitive ear, and by none of the rules of classical harmony, not even those followed and stretched by Wagner. Ravel followed suit, and in due course the French composers were to influence Bartók, Stravinsky and Janácek, all of whom borrowed the whole-tone language when they needed it, meanwhile inventing with the ear.

The second innovation, yet more subversive, was the introduction of entirely atonal melodies and harmonies by Schoenberg, who also, in his vocal setting Pierrot Lunaire, used Sprachgesang – a kind of insinuating sing-song, in which words are deftly stuck onto the musical line, rather than being sung to a melody of their own. It was impossible to dismiss Schoenberg’s innovations as the work of a second-rate composer trying to disguise his incompetence. In Gurrelieder, Verklärte Nacht, and Pelléas et Mélisande he had shown total mastery of tonality and of late romantic harmony, and these great works remain part of the repertoire today. But by the time of the Piano Pieces op. 11 he was writing music which to many people no longer made sense, with melodic lines that began and ended nowhere, and harmonies that seemed to bear no relation to the principal voice. At the same time, Schoenberg’s atonal pieces were meticulously composed, according to schemes that involved the intricate relation of phrases and thematic ideas.

In due course this meticulousness led to an obsession with structure and the quasi-mathematical idiom of twelve-tone serialism, in which the linear relations of tonal music were entirely replaced by a permutational grammar. The result, in the hands of a musical genius like Schoenberg, was intriguing, often (as in the unfinished opera Moses und Aron, and The Survivor from Warsaw) genuinely moving. Schoenberg’s pupils Alban Berg and Anton Webern developed the idiom, the one in a romantic and quasi-tonal direction, the other towards a refined pointillistic style that is uniquely evocative. But it should be remembered that all these experiments were begun at a time when Mahler was composing tonal symphonies, with great arched melodies in the high romantic tradition, and using modernist harmonies only as rhetorical gestures within a strongly diatonic style. And in England Vaughan Williams and Holst were working in a similar way, treating dissonances as by-ways within an all-including tonal logic.

A concert-goer in the early 1930s would have been faced with two completely different musics – one (Vaughan Williams, Holst, Sibelius, Walton, Strauss, Busoni) remaining within the bounds of the tonal language, the other (Schoenberg and his school) consciously departing from the old language, and often striking a deliberately defiant pose towards it. Somewhere in between those two musics hovered the great eclectic geniuses, Stravinsky, Bartók and Prokoviev. And meanwhile the polemics abounded, some dismissing the tonal idiom as reactionary and exhausted, some attacking the modernists as nonsensical and deliberately insulting to the good bourgeois audiences who paid for their self-indulgence.

As we know the contest between tonality and atonality continued throughout the 20th century. The first was popular, the second, on the whole, popular only with the elites. But it was the elites who controlled things, and who directed the state subsidies to the music that they preferred – or at least, that they pretended to prefer. From the time (1959) when the modernist critic Sir William Glock took over the musical direction of BBC’s Third Programme, only the second kind of contemporary music was broadcast over the airwaves in Britain. Composers like Vaughan Williams were marginalised, and experimental voices given an airing in proportion to their cacophonousness. During the 1950s there also grew up in Darmstadt a wholly new pedagogy of music, under the aegis of Karlheinz Stockhausen. Composition, as taught by Stockhausen, consisted in total randomness of inspiration combined with a meticulous mathematisation of the score, to produce music which makes little or no sense to the ear, but which fascinates the eye when spelled out on the page.

Stockhausen’s own works – bulbous monstrosities which make maximum demands on the listener’s attention and give next to nothing in return – received and still receive extensive, usually state-subsidised performances all across the world. His older Austrian contemporary, Gottfried von Einem, who was at the time writing powerful operas in a tonal idiom influenced by Stravinsky and Prokoviev, was in comparison ignored, not because his music is trivial, but because he was perceived to be out of touch with a musical culture determined to clear away the dangerous vestiges of the romantic worldview.

It is no longer accepted as proof of a low-brow musical sensibility to wish to reconnect with that the romantic tradition. It is now permissible to like Sibelius and Vaughan Williams, and to believe that they are superior to Stockhausen and Boulez. It is permissible to reject the notion that tonality was made irrelevant by the atonal school, and to recognise that some of the greatest works in the tonal tradition were composed in the middle of the 20th century: Rachmaninov’s Rhapsody on a Theme of Paganini, for example, Strauss’s Four Last Songs, Britten’s War Requiem, the later symphonies of Shostakovitch and Vaughan Williams, Aaron Copland’s Clarinet Concerto and Appalachian Spring. Some of these – the Rachmaninov and the Strauss – could be seen as extracting unexploited remainders from the tonal tradition. Others – Britten (and not in this work only) and Copland – could be seen as actively engaged in renewing the tonal tradition, drawing out new kinds of melodic line and novel harmonic sequences.

The idea put about by the Marxist critic Theodor Adorno, that tonality was by then nothing but the exhausted remainder of a dead tradition, was definitely disproven by post-war music. By the 1950s it was atonality and not tonality that was exhausted. The radical modernist idiom was kept alive by Darmstadt and the musical establishment, by the system of official patronage and by the fact that real musical education, which used to be a household requirement, had been effectively destroyed by the invention of broadcasting and recording. Without a musical education it is not easy to feel confident in saying, of Stockhausen, that the Emperor has no clothes.

Meanwhile the real modernist experiments – those which drew freely on the tonal tradition and on the eclectic spirit of Western civilisation – gained acceptance from the ordinary concert-goer. Works like the Turangalila Symphony of Messiaen, the remarkable Star-Child oratorio by George Crumb, and the triple concerto of Michael Tippett entered the repertoire without any need for the critical hype and institutional support enjoyed by Stockhausen and Boulez.

But there is another reason for the brief ascendancy of radical modernism, and one that bears heavily on the future of Western music. During the course of the 20th century a wholly new kind of popular music emerged. Nobody can say, in retrospect, that the waltzes and polkas of Strauss or the operettas of Léhar and Offenbach belong to another language and another culture from the symphonies of Brahms or the music dramas of Wagner. Strauss (father and son), Léhar, Offenbach are now counted in the ‘classical’ repertoire, just as much as Wagner, Brahms and the other Strauss. And the distinction between popular entertainment and high art is internal to their repertoire: the Overture to Der Fledermaus and the Hungarian Dances of Brahms surely stand side by side.

Only in the 20th century did popular and serious music finally divide, and the principal reason for this was jazz – itself an intrusion into Western culture from a place outside it. The origin of this remarkable idiom is veiled in obscurity, though it is evident that it absorbed, along the way, both the syncopated rhythms of African drum music, the blues notes that come from attempting to unite the pentatonic and the diatonic scales, and the chord grammar of the Negro spirituals. The jazz idiom showed a remarkable ability to develop, so that an entirely new harmonic language grew from it, and soon became the foundation of a new kind of popular song and dance. It was this quintessentially American idiom that most got up the nose of Adorno, and which served as his proof that tonality was destined to degenerate into short-breathed melodies and repetitious sequences. For that was all that Adorno found in the jazz of his day.

It is true that improvisation around a ‘jazz standard’ is a very different thing from the far-ranged musical thinking that we find in the concert-hall. A work that returns constantly to the same source for refreshment, and goes on ‘forever’ precisely because it goes on only for a moment is a very different thing from the symphony which develops thematic material, extracting a continuous musical narrative. But Ravel, Gershwin and Stravinsky showed how to incorporate jazz melodic lines and even jazz harmonic sequences into symphonic works that had some of the long-distance complexity of the classical tradition. Meanwhile there emerged a new form of popular music, on the edge of jazz, but reaching into the world of folk melody and light opera. This was the idiom of the Broadway Musical and the American Song Book. Brilliant musicians like Cole Porter, Hoagy Carmichael and Richard Rodgers became household names, with songs that our parents knew by heart, and which defined a new kind of taste.

This was music to be sung around the house, which normalised the emotions of ordinary people as they endeavoured to cope with the new world of machines, gadgets, social mobility, fast romance and easy divorce. Thus began the great fracture in the world of music between ‘pop’ and ‘classical’, in which it became ever more important for the critics to side with the classical tradition, and to find something that distinguished modern composers in that tradition from the ‘easy listening’ and ‘light music’ that filled the suburban bathroom.

For a while, therefore, there was an added motive for composers to take the path of radical modernism, and so to give proof that they belonged to the great tradition of serious musical thinking. A composer like Boulez or Fernyhough, ensconed in the madhouse of IRCAM in Paris, could be ‘bounded in a nutshell and count himself king of infinite space’, as Hamlet put it. Insulated from the vulgar world of musical enjoyment, sending out musical spells into the electronic ether, the composer began to live in a world of his own.

That it should be Boulez who received the accolades and not Maurice Duruflé or Henri Dutilleux is explained by the enormous publicity value of difficulty, when difficulty is subsidised by the state. The modernists had succeeded in persuading the official bodies that they were keeping alive the flame of high art amid the grotesqueries of an increasingly degenerate pop culture. And for a while, following the transformation of rhythm and blues into a universal idiom of song and dance by Chuck Berry, The Beatles and The Rolling Stones, it seemed as though they were right. What did this new popular music have to do even with the comparatively refined language and domestic charm of the Broadway musical, still less with the symphonic and operative traditions?

But then the whole thing collapsed. Impassable divides have an ability to survive in the old hierachical culture of Europe; but they don’t last for long in America. Composers like Steve Reich and Philip Glass had no desire to separate from their hippy friends, or to lose the one benefit that makes the life of a composer worthwhile, namely an audience. There emerged the new idiom of minimalism, in which the harmonic complexities of the modernists and those of the great jazz musicians like Monk, Tatum and Peterson were both rejected in favour of simple tonal triads, often repeated ad nauseam on mesmeric instruments like the marimbas.

The result, to my ear utterly empty and the best argument for Boulez that I have yet encountered, succeeded in entering the repertoire and gaining a young and enthusiastic audience. The explanation is simple. This music uses the devices of pop: regular and mechanical rhythm, fragmented and constantly repeated melodic lines, and a small repertoirs of chords constantly repeated. It has in effect joined the world of ‘easy listening’.

Whether Reich and Glass entitle us to talk of a new and ‘postmodern’ idiom in the world of serious music I doubt. For this is not serious music, but a kind of musical void. Listening to Glass’s opera Ekhnaton, for instance, you will be tempted to agree with Adorno, that the musical idiom (let’s not speak of the drama) is utterly exhausted. But then along came John Adams, whose mastery of orchestration and knowledge of real tonal harmony began to redeem the minimalist idiom, and to bring it properly into the concert hall. And other American composers followed suit – Torke, Del Tredici, Corigliano – writing ‘tonal music with attitude’, inserting advanced harmonic episodes into structures that make thematic and rhythmical sense. In Britain a new wave of tonal composers has also emerged, some of them – like James Macmillan, Oliver Knussen and David Matthews beginning as radical modernists – but all moving along the path mapped out by the great Benjamin Britten, the path out of the sterile desert where no melody ever grows, into a place of song and dance.

Such composers learned the lesson taught (however clumsily) by Reich and Glass, which is that music is nothing without an audience, and that the audience must be discovered among young people whose ears have been muddled by the ostinato rhythms and empty chord sequences of pop. To offer serious music to such an audience you must also attract their attention. And this cannot be done without melody and rhythm that connect with their own bodily perceptions.

Is it working? I don’t know. The catastrophic effect of modern pop – think of youjr own Death Metal Group Meshuggah, or the obscene Lady Gaga – is felt not only in the ears but in the soul of its devotees. And it is difficult to write fresh and tuneful music for a burnt out soul. Still, we have no other path to tread, unless it is the one that reaches back into that sound-proof bunker beneath the Centre Pompidou, where the spells are still being brewed and sent with undiminished malice across the airways of Europe.

The first mistake … detail from The Expulsion of Adam and Eve from Paradise, by John Martin. Photograph: Laing Art Gallery, Newcastle

One day in 1972, Village Voice journalist Ross Gelbspan attended a press conference. It was being held to promote a book called The Limits to Growth, which postulated that, because of increasing population and pollution and diminishing resources, our future world would be a place where no one would want to live. During the conference, Gelbspan was struck by the happy sight of one of the book’s co-authors, Donella Meadows. How heartening, he thought, that despite her book’s grim prognostications, she was pregnant. He went back to the office and typed up a story about how there was some hope amid the gloom, symbolised by Meadows’s swollen belly. His editors liked the story so much they put it on the front page.

There was only one problem: Meadows wasn’t pregnant. As I write this, I can feel blood rushing to my face in empathetic embarrassment. Even today, nearly 40 years after the error and almost a decade after Meadows’s death, Gelbspan is still mortified. At the time, he wanted to die. However, let’s snatch optimism from Gelbspan’s understandable anguish. As Aristotle wrote in the Ethics, it is not good to feel shame – since it is bad to have done something one should feel ashamed of – but to do something wrong and not feel shame is a sign of wickedness. In an increasingly shameless world, Gelbspan’s authentic distress is a sweet sign that not everything about us is going wrong.

In this lovely book about human mistakes the sickeningly young, forbiddingly clever and vexingly wise American journalist Kathryn Schulz doesn’t cite Aristotle, but he is a kindred spirit. Where Aristotle saw the value in a painful, ostensibly demeaning emotion, Schulz argues passionately for the value of error. The experience of being wrong, she argues, helps to make us better people, with richer lives.

We have all sensed the shame of being publicly wrong. I will always drag around, like Jacob Marley’s chains, the clutch of errors I committed nearly a quarter of a century ago in an article about junior doctors’ working hours. But journalists’ mistakes are the least significant in the harrowing catalogue of human wrongology: think of the surgeon who cut off the wrong leg, the pilot who pushed the wrong button, the judge who hanged an innocent man; of waking up next to Mr Wrong, or being Boris Johnson’s barber.

Or consider William Miller, who predicted that Armageddon would take place on the morning of 22 October 1844. Hundreds of thousands of devotees around the world were more than a little chagrined when the second coming of Christ foretold by Miller did not take place – not least because many of them had not bothered to harvest crops or even have breakfast, such was their conviction that the Rapture would occur before lunch. They experienced what came to be known as the Great Disappointment, while Miller, to his credit, produced a marvellous mea culpa in his tract “Apology and Defense”. “We expected the personal coming of Christ at that time,” he wrote. “And now to contend that we were not mistaken, is dishonest. We should never be ashamed to frankly confess all our errors.”

Few of us, when we err, feel the urge to confess thus. We prefer to be in denial than to confront the unacceptable truth that we were ever wrong; to be in the right is our natural home. This is why Keats’s negative capability – “when man is capable of being in uncertainties, Mysteries, doubts without any irritable reaching after fact & reason” – is so challenging to rational animals like us. To be in uncertainty is to be conceptually roofless. And to be in the wrong is even worse – not just a challenge, but a shattering experience of self-alienation. It is for that reason, no doubt, that according to Elisabeth Kübler-Ross’s five stages of grief, denial is a proper reaction to trauma.

Of course, one can go too far with this eulogy to obduracy. The refusal of South Africa’s former president Thabo Mbeki to admit that Aids is caused by HIV, and can therefore be controlled using antiretrovirals, contributed to the deaths of around 320,000 and made orphans of many others. And while Schulz is very good at exploring the manifold human mechanisms of denial, even she can’t explain why Hans Blix never managed to convince either Bush or Blair that their conviction about the existence of those weapons of mass destruction was wrong.

Denial is hard work. Sartre wrote that, in order to be self-deceiving, “I have to know this truth very precisely in order to hide it from myself the more carefully”. Tough gig: to be in denial we need to be both the trickster and the tricked.

What is most cherishable about this bumper book of other people’s booboos is its insistence that to experience error is, at its best, to find adventure – and even contentment. Schulz takes as her model Don Quixote, the knight-errant who was wrong about almost everything. “Countless studies have shown that people who suffer from depression have more accurate world views than non-depressed people,” she points out.

But Schulz is hardly counselling that we choose to be wrong. That would be a contradiction: error is like happiness, perhaps, in that we can only stumble across it rather than seek it out. Instead she proposes that once we find ourselves in the wrong, we should be optimistic. To be wrong, after all, is to depart from the facts into creativity, to become artists in our own lives. Error may feel like despair, but it is more akin to hope: “We get things wrong because we have an enduring confidence in our own minds; and we face up to that wrongness in the faith that, having learned something, we will get it right next time.” That may be no consolation to the patient who loses a healthy limb or the pilot whose mistake chucks a 747 into the Atlantic, but for those of us whose errors are less disastrous it’s a cheering perspective.

There is, if you apply Schulz’s sunny philosophy of wrongology, a parallel between the Guardian’s Corrections and Clarifications column and evolutionary theory: both are premised on the usefulness of error. When the Guardian launched its forum for nitpickers, journalists feared that public shaming over errors would be a poor exchange for the furtive pleasure of denial, and the delusion that one is incapable of being wrong. It proved otherwise: openness about committing mistakes, even if they are ones that make you want to lie face down in a shallow grave and cry for all eternity (I speak as a regular Corrs and Clars contributor), is essential for public trust and – that demanding monster – personal growth. Similarly, Schulz argues, “because of errors in the replication of genetic sequences, we wind up with variation among individual members of a species; and because of that variation, the species as a whole can adapt and survive. Such errors literally keep their hosts alive.”

She takes a similar upbeat moral from Thomas Kuhn, the great historian of science who noted that the periods between the breakdown of one system or theory and the entrenchment of another are always marked by an explosion of competing hypotheses. Each one is a response to the inadequacies of the previous orthodoxy. “By determining where we went astray in the past, in other words, we also determine where we will wind up in the future.” Being wrong, therefore, is important for scientific progress.

Schulz’s self-styled discipline of wrongology contains a great deal of lightly worn learning and careful reflection on psychoanalysis, philosophy, science and art. The book, if too long, is for the most part delightfully written. It reminds me of what we want clever Americans to be: positive, without a raincloud for a heart, and yet with the wit to make that optimism compelling.

There is a philosophy called “pessimistic meta-induction from the history of science”, which proposes that, since the most watertight of old theories have been disproved, we must assume today’s theories will be disproved, too. This holds for economics, medicine, education – any discipline you’d care to mention. It’s a good theory but not quite as appealing as Schulz’s rival account, which she calls “optimistic meta-induction from the history of everything”. This states that our capacity to err is inseparable from our imagination. “In the optimistic model of wrongness, error is not a sign that our past selves were failures and falsehoods,” Schulz writes. “Instead, it is one of those forces, like sap and sunlight, that imperceptibly helps another organic entity – us human beings – to grow up.” Schulz doesn’t relate whether she converted her colleague Ross Gelbspan to her optimistic meta-induction theory, but I hope she did. He’s surely suffered enough.