No great man, they say, is a hero to his valet. A bodyguard, who oversees his charge pitilessly night and day, might be expected to have a yet lower opinion. Walter Thompson, a protection officer from Scotland Yard, spent almost 18 years guarding Winston Churchill, from 1921 to 1945, with a break in the 1930s. His presence was so constant that, as he ruefully admitted, he became a "perpetual annoyance" to Churchill's wife, Clementine.

He was outside the door of every room her husband was in. He ate and slept at the Churchills' houses in London and at Chartwell in Kent, and then at Downing Street and in their Blitz-proof quarters in the annexe. He travelled with "the Old Man" so often that his own marriage was wrecked, protecting him from potential assassins, helping him through "the Black Dog" of his depressions, caring for him when he was ill and near death. In the war, as Thompson pointed out, he spent more time with Churchill than "any other human being", Clemmie included. She did not like it, sometimes making this clear by not feeding him.

Officialdom did not like it either, when it came to Thompson's memoirs. He had almost completed them by June 30, 1945. Churchill had just ceased to be prime minister., and Thompson was about to retire from the police. Downing Street and the police commissioner, Sir Harold Scott, agreed it would be "quite improper" for him to publish material "for some considerable time to come". Thompson's meagre pension, £353 12s a year, could be forfeit if he went ahead. He anyway had to supplement it — he collected rents for New Romney council and worked in pest control — and the threat was enough.

He became a minor celebrity, lecturing on his years with Churchill and writing more or less expurgated reminiscences, published in Britain and the United States. He died in 1979. His full, unpublished memoirs, running to 350,000 words, were found in the loft of a Somerset farmhouse by his great-niece Linda Stoker, and form the background to a new television series, and a book, Churchill's Bodyguard, to be published on November 7.

The intimacy that gives the memoirs their strength extends from the comic and endearing to the dark and tragic.

As Churchill built a sandcastle for his children on the beach at Frinton-on-Sea, Thompson witnessed one boot and sock being soaked by a mischievous wave. Churchill plunged the other boot into the sea. "Well, Thompson," he said. "We must treat them both alike." Thompson was with him, too, as he paced the garden while doctors fought in vain to save his fourth child, Marigold, "Duckadilly" to her father, from septicaemia. He was there when Lady Randolph, Churchill's mother, was dying. Thompson offered to give blood. It was too late, but Churchill embraced him, said he would not forget the gesture, and did not.

At times, history seemed to hold its breath. Driving back from Buckingham Palace in 1940, after the king had asked Churchill to form a government, Thompson congratulated him on the "enormous task" he had undertaken. "God alone knows how great it is," Churchill replied. "All I hope is that it is not too late. I am very much afraid it is, but we can only do our best." Tears welled in his eyes. He was 66.

Thompson thought he looked it — "he walked with a conspicuous stoop... most of his hair had gone... his cheeks were pouchy" — and at this supreme moment Thompson sensed his uncertainty. But he muttered and set his jaw, "never again at any time" to express such doubt.

He saw Churchill at his lowest points, a frustrated "kicker of wastepaper baskets" during his wilderness years in the 1930s, weeping over the concessions Roosevelt made to Stalin at Yalta in 1945. "Why, Thompson, did they allow the president, almost dying on his feet, to be there?" he asked. "All Europe will suffer from the decisions made at Yalta."

They were together, too, when the war was won. "Ah, the bloody beast is dead," Churchill said, "elated and with much emphasis", when he heard of Mussolini's fate. But when told Hitler was gone, he went to a window and looked out, remaining silent for some time. Thompson asked if he thought Hitler had committed suicide. "That is the way I should have expected him to have died," he said. "That is what I would have done under the same circumstances." On VE Day he sent Thompson back to get his cigars before greeting the crowds. "They expect it of me," he said, the showman to the fore.

Churchill was a handful from the off. He was an adrenaline junkie who took the same delight in danger as a statesman and world leader as he had as a young officer and war correspondent. Thompson got him as a temporary assignment when he was colonial secretary in February 1921. Colleagues warned him that Churchill was "a difficult man to guard and rather uncertain of mood". He first found him "almost priggish, lacking in consideration", thoughtless and self-centred. He asked to go elsewhere. His chief, with a faintly sadistic smile, told him: "Winston's asked for you to be with him permanently."

Their first trip together was to the Middle East. Jews were pitted against Arabs in Palestine, and nationalists demanded independence in Egypt. The party was accompanied by T E Lawrence, whose success in throwing the Turks out of Arabia had created the uneasy vacuum Churchill sought to fill. Lawrence warned the new bodyguard that Churchill's life would be "at risk from the instant we were on Arab soil".

As their liner came alongside in Alexandria, the clamour of the crowd on the dockside reached "a veritable crescendo of fury" as it caught sight of Churchill coming down the gangplank. To avoid riots, Thompson had to smuggle Churchill out of his hotel and onto King Fuad's royal train for the trip to Cairo.