The suicide car bombers who have inflicted the recent terrible mass murders on Iraq are usually single young men. Lieutenant Dave Robison, a Texan doing guard duty at the heavily fortified entrance to the International Zone in Baghdad last week, told me: "We can sometimes spot them because they have purified themselves by shaving their beard and hair and their car is obviously very heavily loaded — often packed with artillery shells as well as high explosive."

So to avoid detection a few days ago, I was told, one man drove to his target to inflict his horrible crime with a woman and two children in his car. All of them were blown to tiny pieces, along with innocent bystanders.

Tactics like this, as well as the sustained attack on foreigners and Iraqis working for the reconstruction of Iraq, pose a serious threat to the US/UK-led coalition and its plans to bring democracy and the rule of law to the country.

It is often hard to be optimistic about Iraq. Security is worse than when I was last there in March. The snatching of hostages and their all too frequent murders are horrific. It is often difficult to see how anything good can come of it now.

But behind the headlines Iraq is not just a horror show. Inside the International Zone in Baghdad I went to the American headquarters, a jumble of partitioned offices spread through the vast halls of one of Saddam Hussein's grossly extravagant and vulgar palaces. Cables, screens, noticeboards, guards and metal detectors are everywhere. People are rushing back and forth beneath the garish marble and gilt, typing, shredding, meeting, talking. General David Petraeus, the American in charge of training Iraqi forces, said: "Iraq is more manageable on the inside than it seems in the media on the outside."

Petraeus also said, wryly, that working in Iraq is like being on a roller coaster. Now, despite appearances, he thinks that it is on the way up as more and more Iraqi police, National Guard and soldiers are being trained to take over from the coalition.

Will they be good enough, soon enough? That is the key question, but his cautious optimism is shared by General Sir Mike Jackson, Britain's chief of general staff, whom I accompanied to Iraq. At the end of his trip to Basra and Baghdad he said media claims that the war is being lost are "blatant hyperbole. Baghdad is not in flames. The insurgents do not represent the vast majority of Iraqis. They can be defeated. But it will be hard pounding".

I travelled first with General Jackson to the southern Iraqi town of Basra, where the 8,000-strong British contingent has its headquarters. There are 31 member countries of the American-led coalition but many of them are token or ineffectual or both. The British, fierce fighters and skilled in counter-insurgency, are absolutely vital.

Basra used to be one of the great trading cities of the Gulf. Now, after 30 years of Saddam's misrule, it is a pitiful slum. Still, since the invasion, economic activity has picked up here as everywhere else in Iraq. Merchants of washing machines, air-conditioners and satellite dishes are doing a roaring trade. Parked on one street is a white stretch limo that some enterprising driver imported from the Gulf and now hires out for weddings. If all goes well, he should have a successful car hire firm in a few years' time.

For that to happen the insurgents have to be stopped. The first and perhaps most dangerous are the Saddam supporters, known to the coalition as "former regime elements". They are based in the "Sunni triangle" north of Baghdad in towns such as Karbala and Falluja that are now off-limits to coalition forces and where the Americans have been hardest hit. They are well financed and organised.

Then there are the foreign Islamic extremists, loosely associated with Al-Qaeda. They are probably responsible for most of the suicide bombings in Iraq; their best known leader is Abu Musab al-Zarqawi, a Jordanian who last week issued another blood- curdling call to victory. He declared that the fight in Iraq was against "the tripartite satanic alliance of heresy and deceit . . . of Americans, Kurds and Shi'ites". He condemns the Shi'ites — the majority of the Iraqi people — as "the Sunnis' enemies, represented by the army of treachery, the party of Satan".

The Shi'ites have fighters, too. The largest grouping is around Moqtada al-Sadr, the intransigent young cleric who has a wide following among the undereducated Shi'ite youth who hated and feared Saddam but who also hate outsiders and are disillusioned by the slow pace of progress since the invasion.

Finally there are proliferating gangs of criminals for whom the lack of security is a fabulous boon. They have been behind many of the kidnappings of westerners in recent weeks. The victims are sold on as hostages: the lucky ones are bartered for large sums of money, the unlucky are often beheaded. Among those killed have been a group of Nepalese cooks. It is difficult to imagine a group more innocuous.