From behind one of the camp dormitories a man jumped out, pink with rage. “This is private property!” he yelled in a harsh Saxon accent. “No pictures! Leave us alone — the past is the past!”

And so it was that we were expelled, not from the Garden of Eden but from the garden colony that was once an offshoot of the Buchenwald concentration camp.

About 4,000 inmates were worked, starved and beaten to death in Rehmsdorf, deep in eastern Germany. “We would see the dead in the morning after we woke up,” Mieciu Langer, a former prisoner, recalled. “Dozens of them scattered through the whole compound.”

Now the same camp has become a suburban idyll. The tidy, lush gardens are occupied not by the SS but by plastic gnomes. There are rose terraces, goldfish ponds, apple trees, swings and playhouses for the children — a case of Holocaust amnesia.

A dozen families live in the camp and they do not want to be disturbed. Conveniently, one of the barracks buildings has been converted into premises for a breeder of German shepherd dogs; every neighbour has bought at least one, trained to growl and snarl as soon as someone approaches the front gate.

“My Dad says he's going to report you to the police if you hang around any longer,” a boy who followed us around the camp said. “You don't belong here.”

The Rehmsdorf residents are, as the saying goes, in denial, unwilling to face criticism that they have made themselves comfortable on tainted soil. But they have committed no crime and are, if anything, only bit players in a much wider debate.

Across Germany and Central Europe there is a struggle to retain memory. Nazis are still being hunted, but only half-heartedly; they are very old, difficult to find, and even more difficult to bring to trial. The witnesses to their crimes are even more fragile and infirm. Memories have become jumbled.

All that remains are the places where atrocities took place and they, too, are often crumbling or have been restored in a way that diminishes their authenticity.

No sign points the way to the Rehmsdorf camp, just a barely visible memorial stone hidden by a shrub close to the railway station where inmates were herded into open cattle trucks. The railway line has been closed; the station put up for sale. Yapping dogs in an open-air cage make sure that no one lingers for long.

Eventually we found a camp resident willing to open his door and keep his kennel shut. Manfred Kriegel is a big, bluff man, hair tied back in a pony tail. He is a passionate country & western singer. The 65-year-old has had a chequered history: after various attempts to escape East Germany he was jailed by the Stasi and freed by the West. Now he lives in the house of the former camp commandant, the SS officer Rudolf Kenn.

Over the kitchen table the SS architects carved the slogan: “To be German means to show character.” Mr Kriegel has covered it up with a block of wood. He has also plastered over a huge blue skull with the chilling motto: “Führer, give us the order and we will follow you!”

It was painted by the masons from the Operation Todt building team that created the camp in 1944. “But the brown floor tiles, they're original,” Mr Kriegel said, leading us through the spacious house.

By the standards of eastern Germany it is a des res: high ceilings and french windows leading into the garden. Mr Kriegel led us through a corridor plastered with photographs of Manfred and the Rangers — his band — John Wayne, Elvis and a certificate from the Cheyenne tribe awarding him the name “Singing Bear”, and into the living room.

This is where Kenn had a Jewish inmate killed because he struck a false note on the piano while playing Oh, How Deceitful Are the Hearts of Women. Does Mr Kriegel sleep badly at night? “No”.

There was that time, though, when a Jewish survivor paid a visit. “He was completely overawed,” Mr Kriegel said. “Prisoners were usually banned from the commandant's house. Then his wife flipped out, shouting at me, asking by what right I was living here, that kind of thing. I almost had to throw her out.”

Mr Kriegel bought the house in the 1990s and if he'd had the money he would have bought the house next door, which used to be the canteen. “It could be really pretty, for the grandchildren.” The inmates' daily diet was two litres of watery soup, 125g of bread and some jam.

“It's not that the villagers were immune to the suffering of the prisoners,” Lothar Czossek, the village historian, said. “They were marched through the place and everyone could see what state they were in. One woman used to put some food on there,” Mr Czossek, 79, added, pointing to a windowsill, “And the vicar's children used to deliberately drop their lunchtime sandwiches.”

He holds up a wooden board that used to hang on the barbed wire fence: “Photographing or standing close to the camp is strictly forbidden. Offenders will be shot without warning.”

The camp was set up after Allied bomb attacks - “Your people” Mr Czossek said with a quick smile - damaged the strategically important Brabag refinery, which turned brown coal into petrol. It had to be rebuilt quickly, roads and railway track had to be cleared of rubble, cables fixed and bomb shelters constructed.

But there was no free labour available. Buchenwald was ordered to lend out its prisoners. The last RAF attack was in May 1944; by mid-June 5,000 concentration camp prisoners were housed in tents near the refinery. Some were seconded to build a more enduring camp near the railway station: 18 barrack buildings in all.

“The work was so hard, 12 hours a day with little food, and the pace imposed by the SS guards was so cruel that almost all inmates were completely drained after four weeks,” Mr Czossek said. “If they were still alive they were sent back to Buchenwald.”

Many did not survive the journey. New labour was constantly being sent, including the Hungarian Jew Imre Kertész who, in 2002, won the Nobel Prize for Literature. Rehmsdorf he remembers as a “small, impoverished, remote provincial concentration camp”.

After the war the watchtowers came down. Under the East German communists the camp was used to produce industrial fats and make camping furniture. The main street was named Street for the Victims of Fascism but no other attempt was made to preserve the memory of the inmates: overwhelmingly, Jews from Hungary, Poland and Latvia.

By the time that communist rule collapsed in 1989 Mr Czossek was about the only man in the community who knew what had happened. The compound was privatised, the old camp dormitories offered for sale as commercial premises, the SS guards' quarters as houses. Some dilapidated buildings were demolished and the camp slowly turned into a prime piece of real estate. SS building methods were more solid than the East German ones.

Suddenly Rehmsdorf camp — a place where people were worked to death — became a desirable place to bring up children.

Even Mr Czossek, who has encouraged the villagers for three decades to confront their past, has come to accept that there are worse fates for a concentration camp.

“They've made something nice out of these places and it must be better to have people living there than let the houses collapse,” he said. Working out of his kitchen, he has created a web of contacts among former inmates and set up a small museum. But what happens when Mr Czossek gives up? Will the active memory of the camp just trickle into the sand of Saxony-Anhalt?

As we made our way out of the compound, past the Appellplatz where prisoners had to line up to be counted at five o'clock every morning, we saw net curtains twitch and the backs of residents as they slipped out of sight.

The Rehmsdorfers may have a legal right to live in these buildings and to prettify them, but no amount of plastic gnomes can take away their personal unease. It is almost as if Rehmsdorf is a haunted village.

Lost lives

238,980 prisoners, from 30 countries, passed through Buchenwald 1937-45

43,045 died there

132 satellite camps and extension units linked to Buchenwald

21,000 prisoners liberated in 1945

1,000 children released

Sources: www.history.com; www.holocaustresearchproject.org

Times Recommends

• Thousands of Tamils dying in 'welfare camp'
• Jackson 'rebuilt his nose before he died'
• Renewed Iran protests crushed by militia