We met at midday beneath the clock of a busy mainline railway station. I was unaware that a counter-surveillance team had already made sure that I wasn’t being followed. Once the all-clear was given, I was taken to a vehicle with blacked-out windows. I couldn’t see out and had no idea where I was going.

My final destination was a safe house. Inside was a man who has to follow the same security procedures every time he meets his handler. The reason: he is a young Muslim working as an agent for MI5. I was meeting him for a BBC Radio 4 series investigating what the security services actually do and don’t do, an attempt to strip away the fiction that shrouds Britain’s spooks.

Since 9/11, Britain’s counterterrorist machinery has grown dramatically. But the perception in those early days was that the threat was from abroad rather than from within Britain. The seeds of extremism had been planted by radical preachers but the authorities had never understood how deep the roots had burrowed.

That became clear in spring 2004 when MI5 surveillance revealed that a group of young British men were buying fertiliser and talking about which targets to bomb in Britain. “It was the moment when the lights went on and you could see the state of the kitchen,” one counterterrorism official told me.

A huge surveillance investigation known as Operation Crevice soon began. Its scale was so large that other priority investigations had to be suspended. Surveillance teams were sent out to follow the suspects and their houses and cars were bugged.

By July 7, 2005 the home-grown threat was clearer, but the suicide bombings in London that day still came as a surprise.

“The fact these were British citizens did really bring the system up with a jolt and had serious implications for what we were trying to do,” said Sir Richard Mottram, who has just retired as chairman of the Joint Intelligence Committee.

After 7/7, MI5 faced a barrage of criticism when it emerged that during Crevice it had had multiple sightings of two of the bombers. But Peter Clarke, head of the Metropolitan police counterterrorism command, told me that collecting enough evidence to make sure any charges stuck had been vital.

“It was quite clear to me that if we were to make the arrests of, say, a large group of young British men and then not to be able to pursue it through the courts, my sense was that the criticism we would come under and the damage that would do to community confidence in our counterterrorist efforts could possibly be such that it could inhibit our ability to mount similar operations in the future,” he said.

Jonathan Evans, MI5’s director-general, last month talked of 2,000 people his organisation believed were a threat to national security – and possibly as many again whom they didn’t know about. In practice that means the days of having the luxury of running operations long and waiting to collect as much intelligence as possible may be disappearing.

“It’s going to get worse before it gets better,” said one counterterrorism official.

When an individual becomes a target, the intelligence agencies compile what are known as i2 charts, which look like a spider’s web emanating outwards from the central figure. Each line denotes a contact. Software allows investigators to look for overlaps or points of significance. Are calls to a pizza delivery number a sign of a hungry suspect or do the times of day suggest a contact?

This analysis involves a bigger role for GCHQ, the government’s listening centre. A growing number of its counterterrorist staff are based in London rather than at its headquarters in Cheltenham. Some are embedded in MI5 investigative teams.

MI6, the secret intelligence service, extends the counterterrorist machinery abroad. Its task is to build relationships with foreign intelligence services so that surveillance is maintained if a suspect goes overseas and to search out new investigative leads from abroad that MI5 can follow up back home.

Dealing with foreign partners can sometimes put Britain’s security officers in uncomfortable situations. A senior Pakistani intelligence officer told me that when his service picks up a suspect who is also of interest to the UK, British officials are allowed to watch the interrogation from a neighbouring room on video monitors. Occasionally they will also interrogate jointly.

It is unlikely that the Pakistanis would maltreat anyone seriously in front of their British counterparts, but there are times when MI5 and the British government will get information that might have been the result of torture.

“Intelligence doesn’t come like pots of jam with labels and health warnings,” argues Sir David Omand, security and intelligence co-ordinator 2002-5. “If you receive intelligence which looks as if it bears on the security of people living here, you have a duty to follow it up.”

The counterterrorist machinery has also spread out from London around the country, with a series of large regional MI5 stations opening to work closely with the police. In a few years 25% of MI5 will be based out of London.

Senior police officers point to a need to ensure that there is always the right balance between the number of people collecting intelligence in MI5 and those investigating it from the police end. Without that, they say, you risk having stockpiles of intelligence leads building up and “another MSK”, as one put it, referring to the 7/7 bomber Mohammad Sidique Khan, who was observed but not investigated.

“Our worst nightmare is intelligence coming in and getting lost in the system and then after an attack realising we had intelligence which hadn’t been developed,” a senior officer explained.

As a result there are now 1,400 police officers in the newly titled counterterrorism command at Scotland Yard as well as plans for 270 in each of the regional counterterrorist units. A joint project, Rich Picture, is designed to cast a wide intelligence net to pick up warning signs of radicalisation or unusual activity as early as possible.

The alleged airline plot in August 2006 caused another of the government’s periodic rethinks. For all the work on pursuing terrorists, the government realised that its strategy on preventing more taking their place was not working. After a bureaucratic tussle, the Home Office has become the centre for a new strategic hub for all counterterrorism policy: the Office of Security and CounterTerrorism.

The rethink also led to the creation of the nebulously titled Research, Information and Communications Unit (RICU). Officials deny this is in any way a propaganda department, although one conceded: “It does sound horribly cold war.”

Its task is to analyse the way in which key audiences at home and abroad (in other words, Muslims) react to messages from the government and to try to give more coherence to those messages to undermine the ideology of Al-Qaeda. In the counterterrorist jargon of the day, its job is to build and promote a counter-narrative to the single narrative that Al-Qaeda and its allies propagate. “You can do it without it looking like government propaganda, because if it looks too much like government propaganda then I don’t think people are going to listen, nor should they,” argues Mottram.

In practical terms, RICU is part of a broader shift away from the militaristic language of America’s war on terror and also a recognition of the role of foreign policy (in other words Iraq and Afghanistan) in radicalising individuals here in the UK.

New tools are on their way focusing on data mining and biometrics – although the spread of biometric identification is causing some headaches for intelligence services, which are used to sending their officers abroad undercover using aliases.

By 2011, MI5 will number some 4,000 staff. Are the mechanisms of accountability keeping pace? Every time the security service wants to do something like break into someone’s house to install a listening device or intercept their phone calls, it needs to get a warrant personally authorised by the home secretary. A commissioner also checks that proper procedures were followed. The number of warrants has risen inexorably in the past few years. But some are turned down, including one in the last few weeks.

Technology combined with a rapid growth in size is going to increase the capabilities of intelligence services in the coming years. The challenge will be to ensure that public confidence and accountability are not left behind.

The Real Spooks begins on BBC Radio 4 on Tuesday at 8pm

Explore UK News

• Crime News

• Education News

• Health News

• Science News

• Scotland News

Times Recommends

• Everyone will get vaccine against swine flu
• Artificial sperm and the race to build a baby
• BBC jets out stars to tout for prizes