As the presenter of Grand Designs, Kevin McCloud entertains millions of television viewers with his caustic observations on other people’s ambitious plans to build their dream homes.

Now the basis of his own visionary scheme to transform the quality of British affordable housing has been called into question after long delays and a split from his architects.

Building Design, the architecture magazine, describes his £19 million project to create a micro-eco-town in Swindon as “in chaos” after he sacked his award-winning architects Wright & Wright because they would not share more of the financial risk. The magazine claims that McCloud’s development has been mired in long delays. It says that his development company is yet to strike a deal with Swindon council over the land for the scheme and has already lost two development partners.

Speaking to The Times last night, McCloud denied the accusations.

“We are far from chaotic and we are not in a catastrophic position,” he said. “We have some of the most secure funding in the housing industry. I’m an easy target, aren’t I? My argument is: judge me when the project is finished. If my professional reputation is in tatters then, I will walk away quietly and become a taxi driver.” Next week McCloud presents a regeneration project in Castleford, the northern former mining town.

The scale of the challenge facing him there pales next to the mission he set himself two years ago. He founded Happiness Architecture Beauty (Hab) to achieve “contextual, sustainable, contemporary, enjoyable, sociable, affordable and profitable” homes because he was fed up with the poor quality of high-volume housebuilding.

Wright & Wright beat more than a hundred architectural firms to the Swindon commission as the industry rallied to McCloud’s vision. Hab praised its initial designs as a “tour de force”. However, despite being paid about £100,000 for work on various housing types, the firm has lost money on the project.

Hab dropped Wright & Wright after Westlea Housing Association joined as a development partner. McCloud said: “We just couldn’t get eye to eye over the contract, that’s the fundamental reason we parted company.”

Hab has now appointed Glenn Howells Architects, who were Stirling Prize nominees, as a replacement, after approaches to AHMM, Feilden Clegg Bradley Studios and Maccreanor Lavington, but has had to restart design work from scratch.

Sandy Wright, of Wright & Wright, accused McCloud’s partnership of lacking the “financial muscle” for its idealistic vision and said that an opportunity to change housing had “gone hugely awry”. He said that Hab’s terms had become “much tougher” after Westlea joined the partnership.

“They wanted us to do work at risk. There was no mention of this at the tender or in the interview or when they told us we had won. It was foisted on us after we had set up a team.

“Kevin has made his name by putting projects under scrutiny and turning it into television. It’s his turn to take the stage, and we are all watching. I agree with his mandate and his ideals, but we all need to be paid.”

Max Fordham, an engineer whose firm is still involved, agreed that Hab had been too idealistic. “You can’t do these things without properly organised funds,” he said.

Editor's choice

• Exclusive: Listen to AC/DC's Black Ice
• Burning Bush with W
• Boy George on sex, drugs and Amy Winehouse