The weeds pushing up between the brickwork at the jaded entrance to the Agios Kosmas Olympic sailing venue in Athens tell a story of neglect. Four years ago at this spot on the Aegean coast, Ben Ainslie became Great Britain’s most successful sailor by adding a gold medal in the Finn class to the one he won in Sydney in 2000 and his silver from Atlanta in 1996.

Today, the landscape is almost apocalyptic – like a scene from The Day After Tomorrow, the Hollywood global warming blockbuster released the same summer that the Olympics returned to their ancient home in Greece to a fanfare by supporters of the Olympic Movement.

The view at the taekwondo venue in nearby Faliro is no more uplifting. A few workmen potter around at the edges of the padlocked site, around which grim public underpasses are plastered with lurid graffiti and lined with stinking piles of rubbish.

Yannis Yalirakis, a civil engineer turned gatekeeper of the Olympic venues, does not appear to notice the bleakness. Charging around the old airport strip in a four-wheel drive, he sees – like most Greeks – only the positives. According to him and the politicians for whom he works, the sad picture presented by a tour of the desolate venues is misleading. Apart from the baseball diamond, which has been converted into a football pitch for a lower-division club, the basketball court that hosts a local team and the fencing hall, which stages trade shows, a casual observer could conclude, justifiably, that the Athens Olympics created a herd of white elephants.

Athens spent 20 per cent of its €11.9 billion (about £9.4 billion) Games budget on the sports venues, so this should be of significant concern to young Greek taxpayers such as Yalirakis, whose grandchildren will be paying for them. Four years on, it is clear that the city is undecided about how best to use the infrastructure it built for a 16-day sports event. But few Greeks are asking why.

Officials at Hellenic Olympic Properties (HOP), the state-owned company that is the landlord for 23 venues, claim that the situation is not as bad as it looks. Confidential papers seen by The Times appear to back them up. They show that the sailing venue is run by a Greek company under a 45-year lease signed along with payment of a €5 million deposit and a €14 million annual rent. The plan is to convert it into a marina and entertainment complex for up to 2,000 yachts.

Similarly, four companies tendered an interest in turning the taekwondo venue into a convention centre. Contracts have been signed to turn the Galatsi Olympic Hall, which hosted the table tennis and rhythmic gymnastics, into a shopping centre and the canoe/kayak venue into a water park, according to the documents.

However, politicians cannot deny that only one venue – the badminton hall – is up and running in its new guise: a 2,500-seat theatre that has hosted Swan Lake On Ice and Jesus Christ Superstar. They blame democracy. “All these processes take time. One cannot operate projects immediately,” Sofoklis Psilianos, the general secretary for Olympic utilisation at the Ministry of Culture, said.

From the moment Athens was awarded the Games, political bickering hampered construction, causing Olympic officials to fear that the venues would not be ready for the Games. The same happened with the legacy planning – or lack of it.

The explanation for the delay between the closing ceremony and the resurrection of the venues is that the Government changed a few months before the Games took place. But a postGames business plan, backed by legislation, should have been in place long before the election. Kostas Kartalis, the general secretary of the Olympic organising committee until the socialists lost power in March 2004, insists that there was a blueprint. “We had post-Games exploitation plans for the venues, but the following Government modified them,” he said. “It was a mistake.”

A sustainable development bill was finally passed in June 2005. Psilianos rejects the idea that the Olympic venues are a financial burden. He claims that they began yielding a net return for the city in 2007. This year HOP is forecast to generate revenues of €40 million, which would more than cover operating costs of €15 million.

Beyond the political buck-passing over the long-term use of venues, though, lies the fundamental acceptance that, for Athens, the 2004 Olympics were not really about the sport.

Athens was the smallest city to host a modern Games, an event that had become a logistical monster after the 9/11 terrorist attacks in the United States in 2001. Greece spent nearly €2 billion on security alone, an unprecedented amount on what may have been a kneejerk measure: security sources say that the flash command-and-control room did not work properly because it had not been tested before the opening day.

“The security cost was too much, but we could not afford not to have a secure Games,” Spyros Capralos, a former water polo player and secretary of state for the Olympic Games, said. “There was a lot of money thrown out the window, but overall the Games were successful. Cities bid for different reasons. For Athens, it was an issue of history and it was a good excuse to modernise our infrastructure.”

Success for Athens is measured not by how many more people are running triathlons but by how much more pleasant an experience a visit to the city is compared with a decade ago. The Games delivered a 21st-century transport system (road, rail and air) and telecommunications network, cleared the air of the worst pollution, cut unemployment and put Greece on the map – tourism jumped nearly 20 per cent in the subsequent two years. For the first time in years, visitors did not dash from the airport to Piraeus to catch a ferry to the islands; they hung around for a few days to visit the Acropolis, mull over the works of Plato and discover the culture of one of the world’s oldest cities.

“We transformed the Greek capital, not simply its image but from the ground up,” Dora Bakoyannis, the Greek Foreign Minister who was Mayor of Athens during the Games, said. “We upgraded infrastructure, brought colour back and made it cleaner and greener. The Olympics were the catalyst for a complete overhaul of the city. The benefits ultimately outweighed the costs.”

Athenians are fiercely proud of their contribution to the Olympic Movement. In the end, contrary to all expectations, the Games were well organised and the athletes and spectators enjoyed their experience. The city’s residents did not bear any additional direct taxes – as Londoners will for the 2012 Games – to fund the exercise, although the national budget deficit ballooned to 6.1 per cent of GDP, which is double the limit under European Union rules.

Politicians believe that the Games will make good on the public investment over the next 20 to 30 years. By way of example, Psilianos has pointed to the benefits of being the city to hand the Games to China: through trade and knowledge sharing, China Air found it necessary to run a direct flight between Athens and Beijing. China’s Cosco Pacific offered €500 million last month for a 35-year licence to manage two berths at Piraeus port. “There have been two big events in modern Greece – entering the European Union and hosting the Olympic Games,” he said. “We are now considered part of the club.”

Capralos, president of the Greek Stock Exchange, agrees. “Just after the Games it was easy to open doors, to get meetings with the outside world that would have been extremely difficult before,” he said. “It was a topic of discussion with people and there was positive momentum – that’s what business people felt and told me.”

Athens could have done many things better. Environmentalists regret that the Games did not fully live up to their “green” promises. George Vernicos, the president of Greenpeace Greece, said: “There were a lot of words, but we had to sacrifice many things because we were in a hurry and one of those was the environment. Security was the big issue, so the money and effort was mobilised there and not for the long term.”

Many involved in the Athens bid also regret the decision to convert most of the venues into commercial complexes unrelated to sport. But it was an inevitable consequence of construction without detailed planning for postGames use. “Legacy planning is a prerequisite now. We did not have the luxury to think what to do with them after the Games,” Capralos said.

Athens is still waiting for its Olympic Park to take shape for the next 20 years, but it is hard to find anyone who did not think that the whole experience was worth it. An independent Olympic economic impact report, imposed by the IOC on all host cities from 2004 onwards, will provide a more scientific answer in a couple of years, but until then Greeks are sticking with the touchy-feely approach.

Bakoyannis said: “I believe that most Athenians now look back on the Games with pride, but also with a sense of optimism for a brighter future. And, to be honest, that’s really what the Olympics are all about.”