What the courtiers saw: the inside story of the great royal fightback

After Diana's death ten years ago, some doubted that the monarchy could survive. Matthew d'Ancona talks to the Palace officials who were there — some of them speaking publicly for the first time 1 t is almost exactly a decade since the death of Diana, Princess of Wales, transformed the country into what Private Eye would call a `cellotaph': grottos everywhere, great and small, full of cellophane-wrapped bunches of flowers, teddy bears, candles, the scenes of unrestrained emotion and group trauma the like of which had never been seen before. The mood of the week preceding the Princess's funeral shook the ancient institution of monarchy to its foundations. Ten years on, it is easy to forget how volatile and eerie those days seemed.

And yet the institution and the royal family have endured and prospered. Charles and Camilla, so vilified in the immediate aftermath of the Princess's death, are now happily married. Last month Diana's sons hosted a concert in their mother's memory at Wembley that was a model of amiable contentment; indeed, William and Harry have come to embody an appealing sense of continuity and change — traditional values in a modern setting, if you will. Their careers, love affairs and nightlife antics absorb the nation as did their mother's heartache, capacity for empathy and globalised glamour. Yet the collective hysteria and institutional instability have gone. How did the monarchy, that seemed so fragile in the first days of September 1997, pull it off?

In the past few months I have been making a programme for Radio 4 exploring this question, and have talked to a range of courtiers and Palace insiders, some off the record, some speaking on the record for the first time. A decade on, their memories of the shock and bafflement remain vivid. In her first broadcast interview, Penny RussellSmith, then deputy press secretary at the Palace, recalled to me the unnerving oddity of the mourning.

'My office window actually looked out on to the fence where I could see literally a wall of flowers being built, starting at ground level and approximating five or six feet by the end of the week. It was a memorable sight. I could see candles, I could see people's faces flickering there, looking obviously very sad, very reflective. And, in fact, there were so many flowers there that the ones that had been laid first had unfortunately begun to rot and so there was this very strange combination of the sort of rotting smell, with the smell of the freshly laid flowers that were constantly being laid on top. The other thing which I think was very noticeable was the silence.'

Decorum required stiff upper lips in public, but the mood within the royal citadel was anything but calm. As Mary Francis, who was assistant private secretary to the Queen, told me, the Queen and the Duke of Edinburgh 'read the newspapers like everybody else and nobody could not be stung by the ferocity of criticism. They were simply puzzled by people's response. They understood the affection for Princess Diana, but they were puzzled by the kind of hagiography that was happening. I heard the Queen and the Duke of Edinburgh subsequently sort of speculating and talking about that, just as my friends were.'

Yet the consensus breaks down when one asks why the monarchy survived what appeared to be a republican moment (even if it proved to be no such thing). In royal circles, there remains some confusion about what actually constitutes 'modernisation'. One senior official told me, in all seriousness, that the decision to allow women to serve at the royal table was evidence of the monarchy's readiness to change with the times.

Lord Luce, the Queen's former Lord Chamberlain, sounded a more pragmatic note in acknowledging that the legitimacy of the monarchy depends upon public consent as well as heredity. 'What you cannot do is to pull up the drawbridge and hide behind the moat,' he told me. 'You have got to remain in touch.'

Yes, indeed. But opinion divides seriously over when the 'Firm' started to take the necessary steps. Lord Luce is one of those who sees the change as steady and evolutionary from the late Eighties onwards, paying tribute to the work of his predecessor as Lord Chamberlain, Lord Airlie, in sorting out the royal finances, handling the Queen's decision to pay income tax, and the opening up of the Royal Collections. One very senior adviser even claimed to me that the week of the Princess's funeral showed just how well-prepared the Palace was for such tests.

But another camp — and one I found much more persuasive — admitted that modernisation was sluggish at best before the shock of Diana's death. The meetings of the Way Ahead Group, a steering committee of the royal family and their slightly embarrassed advisers, encapsulated the sense that there was a problem but also the absence of will to confront its deepest roots.

'It was always a precept of the royal family, and I've heard the Duke of Edinburgh say this more than once, that they're not there to win political approbation or political popularity,' Mary Francis told me. 'There had been a lot of talk about change in the organisation and in the institution in the years leading up to Princess Diana's death, but not a great deal had happened. After her death, subtly, and without saying very much about it, there was a general agreement that things should be done a bit differently, that rather more experiments should be made, that a less formal style should be adopted.'

Penny Russell-Smith, speaking in her last week as a Palace official in her ground floor office, made much the same point to me. 'Certainly, for the first year or so after the Princess's death, there was a lot of long hard thinking and self-criticism going on, to see what could and should be changed, and to make sure that what was changed was done for the right reasons.'

The chosen direction after the Princess's death was to make the monarchy just a little bit more Diana-like. Not so touchy-feely or emotionally intelligent, maybe, but certainly more open, more engaged, more systematic in its contact with the public. Among the reforms have been themed receptions (women in business, the media, Americans in Britain), a rigorously organised calendar of events (in the style of a New Labour `grid'), the wiring up of the Palace for television appearances. The Queen herself may not have changed — indeed, the whole point is that she doesn't. But the way in which she and her family go about their business most definitely has.

In 2002 the death of the Queen Mother presented the monarchy with another test of its durability — and one that, to the surprise of many, it passed with distinction. Indeed, if Diana's death marked the beginning of a process of worried introspection, then the Queen Mother's funeral and the wave of public mourning that preceded it formed the other bookend.

The monarchy had once again demonstrated both its resilience and its capacity to adapt, to cherry-pick the best of modernity. In the same year, the rock concert at Buckingham Palace to celebrate the Queen's Jubilee drew one million people to the Mall and the parks surrounding it. Lord Luce told me that 'some of us were quite nervous about this . . . but it showed that the monarchy was open-minded and prepared to look to new things and to reach out to a broader range of people.'

Yes: but why did it work? The unravelling of the royal marriages up to and including the death of the Princess had, in the words of Penny Russell-Smith, locked the Palace into the thankless task of 'damage control'. After Diana's death, the soap opera continued — but the royal family learnt how to adapt to the world of modern celebrity rather than to recoil from it. The mystique of the monarchy had gone for good: to adapt Bagehot, its magic was now drenched in daylight. The trick then, as so often in the history of British institutions, that long saga of Whiggish adaptation, was to make a virtue of necessity. In the world of Posh and Becks, the allure of the monarchy remains huge. To put it crudely: who throws better parties, or has better bling?

What we have witnessed, more subtly, is the rewriting of the contract between monarch and public. Lord Luce cites the example of bowing and curtsying — a custom once sacrosanct, now voluntary, but still observed by 99 out of 100 people. A culture of involuntary deference, in other words, has been replaced by a culture of voluntary etiquette and politeness. Respect for Queen and monarchy remains powerful. But, in the words of Lord Luce, 'it has to be earned in this age'.

Therein lies the inherent vulnerability of this institution, whose fate, by definition, depends upon the strengths and weaknesses of its incumbent. Mary Francis hinted to me that, as King, Charles would have to curb his inclination to speak openly and, some would say, self-indulgently on controversial issues.

'I think the challenge for him,' she said, 'will be to convey that the things that he wants to take an interest in are things that he is selecting because they're important to the nation as well as important to him, and that he's not doing them just because he has a personal enjoyment of them, and maintaining the difference between the public role and personal life is something that the Queen has done very effectively. But it will be a change and it will no doubt be a challenge.'

There was a time, of course, when some wondered whether there would still be a royal title for Charles to succeed to at all. What 1997 exposed was a fragility that may return. The moment came and went, but the monarchy, an ancient and cunning organism, absorbed the lessons of the trauma. It has evolved into something different rather than completely new, more at ease now, having swapped faded mystique for the most rarefied form of celebrity.

Are we as a nation just suckers for tradition? That is only part of the truth. In the past ten years deep, visceral instincts have fizzed in chemical reaction with the short memories of the modern age. The collective desire to support the monarchy is still very strong. And — in an era of attention deficit — if mistakes are corrected, they are quickly forgotten. So, in the end, the public were not rebels at all, but complicit with the monarchy in this process of selective amnesia and quiet restoration.

The twist in the tale is that this is the story of a joint effort, not a con trick. Diana's grip on our collective imagination is assured. And yet the monarchy's grip is tighter. The deeper lesson of the past ten years is that our national genius for memory is matched by a genius for forgetting.

A Royal Recovery, produced by Mark Savage, is broadcast on BBC Radio 4 at 9 a.m. on 21 August, and repeated on the same day at 9.30 p.m.