Lord Malloch-Brown is to leave the Government in the next few weeks. The departure of the Foreign and Commonwealth Office minister — the latest in a series of figures to resign — is a serious blow for Mr Brown.

The former UN Deputy Secretary-General, who was appointed by Gordon Brown when he became Prime Minister in June 2007, insists that he had made clear he would only serve for “a couple of years”. “I remain completely loyal to the Prime Minister,” he said.

Mark Malloch-Brown was one of five non-politicians ennobled to serve in a “government of all the talents” when Mr Brown arrived in No 10. His departure means only three of the “goats”, the name given to the outsiders, will remain in post this autumn.

Lord Malloch-Brown, whose brief covers Africa, Asia and the UN, had first told the Prime Minister that he would like to leave at the summer recess before last month’s reshuffle but was persuaded to delay a final decision. Friends say that he has not lined up another job but has “compelling family reasons” for going.

His departure will be a greater blow to Mr Brown than that of Lord Carter of Barnes, whose intention to leave was revealed by The Times last month. The former public relations boss, who served briefly as Mr Brown’s communications and strategy director at No 10, did not remain in the Prime Minister’s inner circle when he left Downing Street and was made a minister.

By contrast Lord Malloch-Brown and Mr Brown have remained close.Lord Malloch-Brown’s appointment to the Government initially raised eyebrows because of his clashes with the Bush Administration while at the UN. Within weeks of starting work at the Foreign Office, he signalled a break with Mr Bush’s White House.

“It is very unlikely that the Brown-Bush relationship is going to go through the baptism of fire, and therefore be joined together at the hip, like the Blair-Bush relationship was,” he said in a newspaper interview.

He endured further public scrutiny when it emerged that he had been given use of a grace-and-favour flat used formerly by John Prescott.

Mr Brown has consistently backed the minister and has allowed him more licence than most members of the Government. In return, say Foreign Office insiders, Lord Malloch-Brown was instrumental in ensuring the success of the G20 Summit.

Lord Malloch-Brown said in a statement to The Times: “I have always said that I would not do this job for ever. And I have strong personal and family reasons for moving on at this time. I came into government as a professional, not a politician. My decision to step down at the end of July is not in any way a commentary on the political situation. I greatly admire the Prime Minister and continue to support him and his Government.

“I joined the Government at his invitation to help promote his international priorities. It has been a great privilege to do that.”

Senior Whitehall officials confirmed that it was expected that Lord Malloch-Brown would leave at the recess.

Last night No 10 said: “The Prime Minister is grateful for the outstanding work that Lord Malloch-Brown has done as Foreign Office minister.

“The Government has greatly benefited from his exceptional knowledge of Africa, the respect in which he is held by an extensive network of close contacts, and his passion for his work.

“His support in preparing the G20 London Summit ensured that the plans set out enjoyed the widest possible international support.”

Times Recommends

• Why Witherspoon is highlighting domestic violence
• No, MP does not stand for Mostly Plastered
• Don’t bottle out of your champagne moment