From the windows of the Manhattan loft he has inhabited for more than 20 years, Robert Hughes, Australia’s most opinionated aesthete, looked out last week at the patch of sky where the World Trade Center used to stand. He was looking out of this window on September 11, 2001, when a hijacked aircraft flew low over his SoHo building and crashed into one of the towers. “There were a couple of BBC producers standing with their backs to the window — we were talking about a possible film on Goya,” Hughes recalled last week. “One of them said to me, ‘Bob, you’ve got a very strange expression on your face.’”

Five years later, the rest of New York continues to wallow in pained debate about how the victims of the attacks should be memorialised and what kind of buildings should be erected next to the void at Ground Zero. Not Hughes, though. He is happy that what he calls that “great ugly scaleless box of a thing” no longer disfigures the New York skyline. He never liked the Twin Towers, and he likes even less the endless “dickering” over 9/11 monuments and memorials. “Can you imagine how little would have happened to London after the Blitz if the same obsessive concentration on loss and sacrifice were placed upon every building that the Luftwaffe flattened?” he said.

He concedes that the initial shock of the towers’ collapse was “deeply discombobulating”. The post-9/11 view from his window was like “looking into a familiar face with a piece bashed out by some maniac with a baseball bat, which of course was the case”, he said. “But do I miss it aesthetically? Obviously not.”

This was fighting talk from a New York resident on the day most of the city was weepily observing the fifth anniversary of the worst terrorist attack on America. Yet Hughes has made a career out of the unflinching honesty of his opinions, as several prominent British artists know to their cost. As the vastly influential art critic for Time magazine for more than 30 years, Hughes was never much impressed with Brit Art celebrities such as Damien Hirst and Tracey Emin, and he is not about to change his mind now he is ageing and injured. Nor is he in any hurry to make peace with his biggest critics — his fellow Australians, some of whom seem to regard him as a traitorous exile devoted to “pissing” venom on his homeland.

Further aggro may well follow the publication of Things I Didn’t Know, the first volume of Hughes’s memoirs. The book opens with an embittered account of the head-on car crash that nearly claimed his life on a remote road in Western Australia in 1999. After a long legal wrangle over who had caused the crash, Hughes was eventually fined £1,000 and banned from driving in the state for three years.

Hughes, now 68, still walks with a severe limp and is in near-constant pain from the injuries he suffered. He has not forgiven the Australian reporters who condemned his behaviour during his trial. At one point, he incautiously described the occupants of the oncoming car as “lowlife scum”, and was then alleged to have described a prosecutor of Indian ancestry as a “curry-muncher” (a charge he has always denied). The media depicted him as an elitist bully; he accused them of making it all up.

On learning that Hughes would be returning to Australia in November to promote his memoirs, one Sydney columnist summed up the relationship: “He thinks we are an uncultured and ungrateful bunch of yobs (“Bullshit!” interjected Hughes, when I read him this passage); we think he’s a grumpy old bugger who should get over it.” “Well, I am a grumpy old bugger, but I don’t see any need to get over it,” snapped Hughes as he hobbled barefoot around his kitchen, making cups of coffee.

It may seem odd that so distinguished an Australian intellectual, so admired by much of the English-speaking world, should be regarded with such reserve by his countrymen. Yet Hughes believes he is not alone in suffering from what he calls “the strange cluster of fantasies that cling to the word ‘expatriate’ in Australia”. “There are at present north of 70,000 Australians living and earning their keep abroad,” he said. “As far as the Australian media is concerned (he pronounces it “ meejah”, with a contemptuous flourish), there are only four expats, and these are Germaine [Greer], Barry [Humphries], Clive [James] and Bob [Hughes].” According to him, the Oz media has concluded that “we four form a sort of cabal whose purpose is to denigrate Australia and piss on its fair name. It’s total rubbish”. He has seen Humphries twice in the past four years. As for James, whom he knew at university, “it’s a long time between drinks”.

It also rapidly became clear that any Greer-Hughes cabal might end in fisticuffs. “Germaine I basically can’t stand,” he said. “Oh, she’s never done me any harm, but her pretensions to being the grande dame of Aboriginehood and all that stuff are ludicrous. There’s no question about her intelligence, but the uses she makes of it — man.” He was most recently appalled by Greer’s attack on Steve Irwin, the Australian crocodile-hunter who was killed by a stingray. Hughes said he had barely heard of Irwin, and “certainly didn’t know he was regarded as this [Australian] hero... then Germaine weighs in with all this stuff about the torturer of reptiles and disturber of the natural order, and I mean, for Christ’s sake, who gives a flying f*** about that?” Hughes reveals in his new memoir that he became an art critic by accident. He was working as a cartoonist on a fortnightly Australian journal called The Observer, which was modelled on The Spectator of London. The journal’s previous art critic was found to have reviewed an exhibition of Blake drawings without having bothered to see it. The Observer’s editor, Donald Horne, promptly fired him and asked if there was anyone else on the staff who knew anything about art. Nobody spoke up. “You’re the cartoonist,” Horne then told Hughes. “You ought to know something about art. Good. Now you’re the f***ing art critic.”

Three decades and half a dozen bestselling books of criticism later, Hughes remains suspicious of much contemporary art — or, as he once put it in a BBC programme, The New Shock of the New: “the here-today, gone-tomorrow stuff. Message art that feels it has to have a punch line”. Now he has retired from Time, he is devoting himself to writing “long, not short”; books, rather than reviews. And he is concentrating on his own past rather than new artists. “But I can still tell a hawk from a handsaw,” he said. “Is some highly promoted Brit artist the new Donatello? I think it’s unlikely.”

Hughes has a broad theory that art in general doesn’t improve over time and doesn’t necessarily get worse, either. But he also feels that our time — right now — is not a great time for good art. “Suppose you come up with the name of a [contemporary] figurative painter whose work is as sublimely impressive as, let us say, Velazquez. I think you’d be really hard put to.”

He went on: “There are a couple of really great figurative painters around — there’s Antonio Lopez in Spain and, of course, Lucian Freud in England. But I don’t think you could say that either of these guys were the mirror equivalent of a Velazquez or a Rembrandt. There are times when art, the medium, just isn’t producing exceptional stuff.”