David Cameron was dragged into a race row last night after one of his frontbenchers made an unfortunate remark during a House of Lords debate. Lord Dixon-Smith, the Tory spokesman for communities and local government, referred to concerns about government housing legislation as the “nigger in the woodpile”.

The phrase described fugitive slaves who hid in piles of firewood as they fled persecution in the American Deep South in the mid-19th century. In November a Tory councillor in Bedfordshire resigned after using the same words.

Mr Cameron said last night that the remark – which is recorded in Hansard – was “not appropriate” but he refused to dismiss him. Instead Lord Dixon-Smith went twice to apologise to Lord Strathclyde, the Tory leader in the Lords. He told The Times afterwards that the remark had “slipped out without my thinking”.

He said that he had realised his mistake when in the chamber and apologised. “It was common parlance when I was younger, put it that way,” he said. He emphasised that he now considered the matter closed.

Lord Dixon-Smith, 73, a Eurosceptic farmer from Essex, has courted controversy before. In March he said that Britain “might have had rather fewer problems” if there had been less immigration.

The remark presents a fresh headache for Mr Cameron in the week that he urged a return to clear notions of right and wrong, suggesting that society had become too sensitive.

It comes less than a month after Boris Johnson, the Conservative Mayor of London, was forced to dismiss a senior adviser who suggested that members of the Caribbean community unhappy with his election could “go home”.

Lord Dixon-Smith’s remark was made during a debate on the Housing and Regeneration Bill, which merges some functions of the Housing Corporation and English Partnerships, creating a new Homes and Communities Agency. Lord Dixon-Smith was outlining his objections to the new agency in the debate, and was agreeing with the objections of Baroness Hamwee, the Liberal Democrat planning expert.

He said: “The Homes and Communities Agency is not a body to which we object in principle. As the minister has explained, it is an amalgamation of the Housing Corporation and English Partnerships. Of course, the nigger in the woodpile, as the noble Baroness, Lady Hamwee, has already pointed out, is that it still incorporates what I call the hangover of the new towns legislation. If it were not for that, we would have little difficulty with the foundation of this agency.”

Another Tory peer, Lord Brooke of Sutton Mandeville, intervened after Lord Dixon-Smith had finished and suggested that he reconsider his choice of words. “My Lords, before my noble friend sits down, he used a phrase about a woodpile,” Lord Brooke said. “If your Lordships’ House were happy, I think it would perhaps be helpful if the wording of the phrase were revised.”

Lord Dixon-Smith replied: “I apologise, my Lords. I left my brains behind. I apologise to the House.”

This prompted a wave of condemnation across Parliament. Keith Vaz, the Labour chairman of the Home Affairs Select Committee, said: “I regard this remark as racist, because it’s deeply offensive. It shows a lack of understanding and sensitivity to the ethnic community and seems to come from a throwback age when people used that kind of phrase as if it was normal. We will judge Mr Cameron on how he responds.”

Denis MacShane, the former Europe minister and antiracism campaigner, said: “Clearly, David Cameron should ask Lord Dixon-Smith to stand down. Cameron always condemns, but always covers up for the ugly face of a Tory Party that can’t stop itself saying out loud what it seems many of them think deep down.”

Lord Sawyer, a former general secretary of the Labour Party, said: “Language like this is absolutely unacceptable anywhere, including in the House.”

A Tory frontbencher said: “I am horrified that a member of my party should make a remark like this. I’m staggered that somebody on the front bench of our party could be so out of touch.”

A spokeswoman for Mr Cameron said that they had not heard about the remark or apology until contacted by The Times: “This was not an appropriate thing to say and it was absolutely right that he apologised to the House.”

Lord Dixon-Smith has been in Tory politics for more than 40 years. Educated at Oundle and Writtle Agricultural College, he served in the King’s Dragoon Guards from 1955-57, reaching the rank of second lieutenant, before turning to farming in 1958. He became a councillor in Essex in 1965, and was chairman of the country council from 1986 to 1989. He lost his seat in 1993 and was made a peer.

He has been on the front bench since 1997, responsible for local government, the environment, and, since 2007, the communities department. He is married with two children and seven grandchildren.

Times Recommends

• How I lost the gift of joy for a while
• Why Ann Widdecombe has got it wrong
• Alas, it’s the end of the road for petrolheads