On Wednesday the Conservative party in Kensington and Chelsea will choose a candidate to succeed me in parliament. It will be uncomfortable to become the lame-duck incumbent, and I shall be sad to hand over a coveted constituency which I enjoy enormously. However, I am convinced I am doing the right thing. I am out of sorts with much of my party, and things have reached a stage where I cannot be part of the answer and I don’t want to loiter and be part of the problem.

At the age of 11 I was interested in politics. In the election year of 1964 I helped to run a Labour party committee room in my parents’ house. I had a poster of Harold Wilson on my bedroom wall. Even seven years later when I reached Cambridge and one of my friends said he was going to a Conservative party meeting I was completely shocked.

But by the middle 1970s Labour was threadbare. Mrs Thatcher took over the Tories in 1975 with a gleam of revolution in her eye. For me it was alluring. Perhaps I’ve never changed: I have some left-of-centre views mingled with a zest for radicalism.

I worked for Margaret Thatcher during the 1979 election, briefing her each morning. Later I had jobs with Cecil Parkinson and the chancellor of the exchequer, Nigel Lawson. Cecil resigned over the pregnancy of Sara Keays, and I drove him back from the Blackpool conference through the throng of flashguns.

By the time I stood for parliament in 1984 I had seen high politics at close quarters. The by-election at Enfield Southgate was caused by the IRA’s murder of Sir Anthony Berry and, as we campaigned, the miners’ strike polarised the nation. I didn’t enter the Commons brimming with youthful optimism.

My enthusiasm for liberal economic reform had by now cast me as a figure of the right. Once that pigeonhole was built, it was difficult to escape. In fact, in my early ministerial career I felt somewhat to the left. I was working with Nicholas Scott (the MP for Chelsea was thought dangerously “wet”) on benefits for disabled people, and later as minister of transport I decided to keep open the Settle-Carlisle railway line, largely out of sentimentality.

Recently, looking back on the Thatcher years, with the benefit of hindsight my admiration is higher even than it was at the time. She inherited a defeated party and by 1979 led it to a substantial majority. She was a creature for her time. We too easily forget how despondent this country was, famous mainly for strikes and high taxes.

Many people detested her for being a warrior, but there were important battles to be won. Her determination, along with Ronald Reagan, to match Russian missile deployments in Europe played an extraordinary role in bringing the Soviet Union to an end. She needed more courage over the Falklands war than did Tony Blair over Iraq, because we had no certainty of winning. Across the whole field of economic and foreign policy, Blair has in many ways been her worthy disciple, but she was the pioneer and I don’t think Blair had it in him to be that.

I knew Margaret was difficult to deal with. She could be infuriatingly unreasonable and I often came away smarting from our encounters. But it was leadership of a kind. Whenever I had to meet her, I would revise madly like a nervous schoolboy and approach on tiptoes.

There was a magnificent paradox at the heart of Margaret Thatcher. In many ways she was very cautious: for example, she postponed the showdown with the National Union of Mineworkers until she was ready. But the rhetoric was stronger than the reality, and even when she was tacking this way and that she convinced people of her unshakeable purpose. All the while she was at No 10, every minister and civil servant knew what to do. The day she left, confusion set in.

The Conservative party has never got over deposing her. Long-established friendships were ended when colleagues disagreed about whether she should stay or go. The problem was compounded because her successor, John Major, wanted to be the opposite of her: to replace her supposed dogmatism with a pragmatic approach to issues such as Europe. At the time I pleaded with him to decide his policy and sack the half of the cabinet that disagreed with him. Now I believe that at least Major saw one thing more clearly than I did: that the party had to move on from Thatcherism.

I made two hideous errors in 1995. John Major told me personally of his decision to seek re-election as party leader. I was taken aback and I wished him well. Having told him that, I could not stand against him. But most people thought he would be mortally wounded in a first round of voting, and that the leadership would be decided in a second round once he had dropped out. I authorised preparations for that eventuality, including installing phone lines in a headquarters. Of course, when that was discovered, it caused my reputation lasting damage.

The second gaffe followed from that. I felt that if I could wow the party conference with a rabble-rousing speech I could climb back. I was secretary of state for defence and my speech prayed in aid the courage of the SAS. It was a terrible failure of judgment and taste.