Lou Reed, Nico and Edie Sedgwick were there. Bob Dylan popped in, as did Truman Capote. Even Salvador DalÍ found time for a visit. And so did a lot of hangers-on.

It wasn’t always easy to keep track of the names who came and went from Andy Warhol’s Factory. Many had pseudonyms: Billy Linich, the hairdresser and self-styled Buddhist and black magician who first shaped its mood, later became known as Billy Name. And the downtown friends he brought were a parade of aliases: Rotten Rita, Mr Clean, and the Duchess - so-called because her father was head of the Hearst Corporation - but does everyone remember who the others were?

Things were a bit hazy: Billy’s crowd styled themselves “amphetamine rapture group” and their drug use led to several deaths. In October 1967 Freddie Herko, an aspiring dancer, put on Mozart’s Coronation Mass and danced naked out of the window. But they were also keen on anonymity: they were also known as “mole people” - homosexuals who kept their lives underground - and their sexuality lent the Factory its tone.

Of course, the Factory was more than a social scene: it was an artist’s atelier, film studio, experimental theatre and poetry workshop.

It also served as Warhol’s casting couch, and he had little sensitivity when it came to content. After Herko’s death he exclaimed: “Why didn’t he tell me he was going to do it? We could have gone down there and filmed it.”

And there wasn’t only one Factory: Warhol moved into the first of them, an old hat factory, in November 1963. It was in Midtown Manhattan, and far from most artists’ downtown studios, but it suited, and the years here were its best.

This was when Warhol himself made his best work, and the Factory’s visitors included Lou Reed and the Velvet Underground. Billy Name gave it its look: he silvered it entirely in tin foil. “It must have been the amphetamine,” Warhol said, “but it was the perfect time to think of silver.”

In 1968, however, the fun spilt over and people started to arrive with guns, so it moved downtown, to Union Square, where a new crowd arrived and it developed a more elegant tone. Then it moved across the street, to Broadway.

Finally, in the 1980s, it moved into a disused electricity-generating station. By that stage the mortgage was $60,000 a month, and Warhol was weary. “Getting rich isn’t as much fun as it used to be,” he said.

- Morgan Falconer is a New York-based writer and critic

Times Recommends

• Unite workers! You’re in Mr Brown’s pocket
• Obama is bashing on the brick wall of unlogic
• Want to understand the war? Ask the generals