The Shadow Chancellor was accused of mocking hundreds of thousands of people with learning difficulties after he joked that Gordon Brown was autistic.

George Osborne faced criticism from campaigners for people with autism and calls from friends of Mr Brown to apologise for his remark.

But Mr Osborne insisted he accepted that autism was a very serious condition and had made the remark during a discussion on how difficult he had found his dealings with Mr Brown.

The row came after a fringe event at the Tory conference in Bournemouth on Sunday at which Mr Osborne spoke about his childhood and recalled how his brothers used to call him “Knowledge” because of his memory. When Mary Ann Sieghart, The Times columnist who was interviewing him at the event, suggested to him that he might have been faintly autistic, Mr Osborne replied: “We’re not getting into Gordon Brown yet.”

His remark drew criticism from the National Autistic Society, whose director said that referring to autism as a means of conveying criticism of someone’s personal skills risked stigmatising the 535,000 people in Britain on the autistic spectrum.

Carol Evans, a director of the society, said: “Any pejorative use of terms relating to autism can cause deep distress and hurt to people affected by the condition. We as a charity are keen to raise awareness in order that these terms are not used lightly by commentators. To use such terms as a criticism of someone’s social skills only perpetuates the confusion that surrounds the condition.”

Nick Hornby, the best-selling writer whose 13-year-old son, Danny, has autism, joined the attack on Mr Osborne, saying that it was not acceptable to taunt an opponent as autistic.

Mr Hornby, who helped to found TreeHouse, an educational charity for children with autism, said: “George Osborne doesn’t seem to have noticed that most people over the age of 8 no longer use serious and distressing disabilities as a way of taunting people.

“(Mr) Osborne claims that when he was younger, he was nicknamed Knowledge — I’m not sure anyone will be calling him Knowledge again for quite a while.”

There was also a furious response from friends of the Chancellor. A source close to Mr Brown said: “This isn’t offensive to Gordon Brown, but it is grossly offensive to the thousands of people affected by autism and their families that their condition should be used by George Osborne as a term of political abuse and he should apologise.”

Mr Osborne tried to play down the row insisting he had not used the term autism himself.

Asked on Sky News if he regretted his remark, he replied: “Not at all. I didn’t say that actually.”

He added: “I was asked if I was slightly autistic.”

Mr Osborne said that he had been “merely” moving the conversation on.

Asked if he believed Mr Brown was slightly autistic, he replied: “No absolutely not. Autism is a very serious condition. Do I think that Chancellor Gordon Brown needs better relations with his colleagues, let alone with the Opposition front bench? Most certainly.”

At a fringe meeting yesterday hosted by The Times, Mr Osborne was again challenged by a journalist to apologise. He replied: “I think you will find I didn’t use the word autism or autistic at all.”

His original reference to Mr Brown came in an exchange in which he complained that the Chancellor had hung up on him during a telephone conversation about “pairing” for a Commons vote, and complained that discussions with the Treasury were impaired as he needed to be on speaking terms with his opposite number.

Mr Osborne complained: “We don’t have very good relations. I have discovered this is fairly common for shadow chancellors — there have been six before me. I have discovered it’s also common for members of the Cabinet.”

The row threatens to undo the progress made by the Conservatives under David Cameron in winning support from people involved in the field of special educational needs.

Mr Cameron has given a number of interviews explaining the impact of his experience in having a disabled son, Ivan, and how this made him look afresh at the importance of the National Health Service.

The Tory leader has also conducted high-profile campaigns against closures of special schools.

The party set up a policy commission on special educational needs, which has produced an interim report and will be undertaking further work in the autumn looking at the policy of “inclusion”, sending children with special needs to mainstream schools.

Times Recommends

• Maternity units' missing £330m funding
• All hands on deck to save the HMS London
• Harman struggles to shine positive light