It's that time of the year again in Hollywood, when network television executives succumb to a strange and destructive affliction, one that either leaves them speechless with frustration or leads to hair-pulling tantrums. The virus that brings on these symptoms is called "HBO envy", a bug that is at its most pernicious around mid-July, when the annual Emmy nominations are announced, and which doesn't even begin to fade until later this month when the winners are declared.

This summer the virus is raging out of control after Home Box Office (HBO), the cable and satellite division of the huge Time Warner media and entertainment conglomerate, once again rewrote the record book at the Academy of Television Arts and Sciences, racking up 124 Emmy nominations, 15 more than in 2003, overwhelming the much larger networks in the process. HBO's total was almost double that of its nearest challenger, NBC, which secured 65 nominations, and more than the other three big networks combined, CBS, ABC and Fox.

So comprehensive has HBO's dominance of the annual awards extravaganza become that competitors have all but abandoned the chase. In the late 1990s a group of rival TV and movie executives formed a pressure group, the Coalition for Emmy Fairness, in an effort to frustrate HBO's success, but were silenced in the subsequent flood of glittering plaudits for the cable channel. Since that brief bout of defiance, the other networks have been forced to grumble on the sidelines or seek consolation in the thought that they and HBO are really in a different business.

A glance at the list of HBO programmes up for Emmys this autumn reveals the simple reason for the premium channel's supremacy. The two HBO shows that won most nominations were Angels in America, the surreal, critically acclaimed miniseries about Aids victims in 1980s New York, starring Al Pacino and Meryl Streep, which won 21 nominations, and The Sopranos, the story of New Jersey mobsters, arguably less a weekly TV drama series than a modern cultural phenomenon. It got 20 nominations, a record for the show.

Not far behind is Sex and the City, the iconic series about four women's search for love and happiness in New York, which ended earlier this year; Carnivale, a strange tale of travelling circus folk and freaks set in the Depression-era Midwest; the comedy show Curb your Enthusiasm, created by Larry David of Seinfeld fame, and the made-for-TV movies Iron Jawed Angels, And Starring Pancho Villa as Himself, and Something the Lord Made. Out of HBO's stable of heavy-hitting shows, only the cult series about undertakers, Six Feet Under, failed to make this year's Emmy list.

HBO dominates the American TV business in a way no other programme-maker has, almost since the advent of the medium, and its influence on world television, Britain's in particular, is also growing by leaps and bounds. In 1972 it began service-transmitting an ice-hockey game to 365 subscribers in Pennsylvania; now HBO has cable or pay-for-view channels in Latin America, Brazil, Southeast Asia, the Far East, Romania, Hungary and the Czech Republic. It was also the first TV station to utilise satellite broadcasting, paving the way for CNN and BSkyB.

In Britain, HBO's reach is felt both directly and indirectly. Because of contractual obligations, HBO programmes such as The Sopranos, Sex and the City, and Six Feet Under are sold to British channels, principally the BBC and Channel 4, by the Hollywood studios that produced them, notably Warner and Paramount, and not by HBO. So the channel gains little except kudos from the deals. But there is an awful lot of kudos to be had. If it can no longer be said without fear of contradiction that Britain makes the best television in the world, then it's thanks in no small measure to programmes like The Sopranos.

Because of its commercial success and its deep pockets, HBO is playing a more direct role in British television by financing ambitious co-productions with British companies, mainly the BBC — projects that alone they could less easily afford to make and market. The list of joint ventures is growing and, judging by their critical and commercial success, the relationships have worked and will probably endure — in the case of the BBC, in a way that offers an opportunity to make world-class television in an economic way.

The Steven Spielberg-directed miniseries Band of Brothers, about US paratroopers in German-occupied Europe in 1944, starring Tom Hanks, was one of the first HBO-BBC co-productions. Its success ensured there would be more. In 2003 there were two more joint ventures, both about the second world war: The Gathering Storm, about the rehabilitation of Winston Churchill, starring Albert Finney, Vanessa Redgrave and Derek Jacobi, and Conspiracy, starring Kenneth Branagh, the chilling story of the 1942 Wannsee conference, when top Nazis decided on the Final Solution: the extermination of the Jews.

BBC Films and HBO Films have co-produced The Life and Death of Peter Sellers, which won rave reviews at Cannes this year, but the most ambitious enterprise so far is the historical series Rome, scheduled for screening on BBC2 and HBO America in 2005. The 12-part series, which may run for another two years if it is a success, is written by the LA-based British writer Bruno Heller. Through the eyes of two Roman soldiers and their families, it tells the story of the transition of Rome from a republic to an empire, following Julius Caesar's conquest of Gaul.

Currently being filmed at the Cinecitta studios outside Rome, with location shoots in the US and elsewhere in Europe, the series is costing an estimated $5m an episode. As the BBC faces an increasingly uncertain future, co-operative, symbiotic ventures of this sort make sense. The BBC gets to work with a partner who is prepared to shoulder most of the financial burden, while preserving considerable artistic influence, and HBO (whose CEO, Chris Albrecht, was won over to the idea of Rome after seeing the seminal series I, Claudius) can tap into the BBC's vast experience while enjoying the prestige of association with the world's most highly regarded media organisation. And both end up with great TV programmes.

If BBC-HBO co-operation points to a way of the future for the BBC, then it can also be said that the experience of HBO in the US carries important lessons for the future of television, in Britain and elsewhere. Those TV executives who tried to stem the HBO Emmy-winning tide in the 1990s were right: HBO is in a different business. While they are in the ratings war, HBO seeks out excellence and quality. And HBO's supreme achievement is showing that it is possible both to make first-rate TV and generate a profit.

Once regarded as the unloved, even unwanted child of the Time Warner empire, HBO has turned into a cash cow for the troubled megacorp at a time when it needs it most. This year it is expected to generate profits in the region of $900m out of a revenue approaching $3 billion, and that represents a growth rate of over 300% in the past decade. To put that into perspective, American network television's most profitable company, NBC, will likely turn in a profit of around $600m this year. And what makes HBO's bottom line a truly impressive statistic is that while NBC, like the other networks, is available in every American home with a TV set, HBO is only seen in a third of the total, around 40m homes.

HBO comes as an optional part of a package offered by the local provider. Basic cable packages sell for around $40 to $60, plus an extra $11 for HBO. The service is not cheap (after charges for the cable box and local taxes, a monthly bill can exceed $80); nor is cable universally available, owing to the cost of laying miles of the coaxial wire across rural America. HBO's audience is effectively limited to large urban centres and suffers, as do all cable channels, from a significant level of viewers who cancel their contracts each month (perhaps 4% to 6%), which makes the wooing of new viewers an ever-present urgency. Also, the channel is operating in a climate that some describe as critical for American television. But HBO is well placed to weather the changes.

An analysis by the Hollywood screenwriter and media pundit Bruce Feirstein suggests that the mass American TV audience is almost a thing of the past: "According to the latest Nielsen ratings (at the end of June), the nation's top-rated news program, NBC's Nightly News, had an audience of 8.9m people, roughly 3% of the population. Cable TV's top-rated The O'Reilly Factor reached 2m, less than 1%. The highest-rated entertainment show, CSI: Crime Scene Investigation, had 15.3m viewers, a little more than 5% of the population."

To put that in perspective, in 1983 more than 125m tuned in to the last episode of M*A*S*H, and in 1998, 76m watched the last Seinfeld show. In contrast, the finales earlier this year of Friends and Frasier won audiences of just 51m and 25m respectively. And Feirstein adds that since March, the Democratic White House contender John Kerry has spent more than $60m on TV adverts and yet, according to polls, some 40% of Americans still can't rate him as a candidate.

So where has America's TV audience gone?

Feirstein believes the loss can be partly explained by the fact that it is now scattered among the scores of channels now in existence. There are some 1,500 free airwave channels in the US, two-thirds of which are affiliated to the big five networks, NBC, CBS, ABC, Fox and PBS, but there are also a staggering 9,000 cable TV systems that Americans can tune into. But Feirstein and others also believe that America is in the midst of one of those elemental, technology-driven changes in the way people spend their leisure time, a change equivalent to the decline in radio audiences in the 1950s, or the challenge to Hollywood represented by the growth of TV. In this case, the new kid on the block is computer gaming. "Entire generations," he wrote recently, "from my four-year-old twins to 40-year-old men, are learning a different storytelling grammar and a new visual syntax. And when asked, they'll be the first to offer that it's more challenging, involving and entertaining than a TV sitcom."

The networks and the big cable channels have responded by dumping more and more sitcoms and turning to reality shows in an effort to win viewers back, beginning in 2000 with the import by ABC of Who Wants to Be a Millionaire?, which the struggling Disney-owned network flogged to death by screening five times a week. The other networks soon followed. Next to come was Survivor on CBS, in which volunteers have to survive nature and their competing ambitions in increasingly obscure locations. At the last count, Survivor has produced nine spin-offs. In 2001, Fox joined the race and screened American Idol, which soon became one of the most watched entertainment shows on TV. Last year the big reality-show hit was NBC's The Apprentice, starring Donald Trump, which averaged just under 21m viewers per show — a solid rating but minuscule compared with audiences of the 1970s and 80s. The show, in which aggressive neophyte junior executives vie for a job in Trump's empire, was, astonishingly, NBC's replacement for Friends. This is the environment in which HBO must not only survive but thrive. The dumbing-down of US television may well have been a British import, but the Americans have accelerated and refined it, giving British viewers a glimpse of their own future. HBO perhaps shows how they can avoid it.

HBO's president of entertainment, Carolyn Strauss, relishes this situation: "People with great programming thrive in the sort of circumstances described by Feirstein." Feirstein agrees: "Niche, specialist markets are the future for television, and HBO is already there." Strauss's predecessor, Chris Albrecht, who in 2002 became HBO's chairman and CEO, pioneered some of HBO's greatest hits (The Sopranos, Sex and the City). He has said HBO is "more than a place; it's an idea", and even claimed: "In certain cases, it's like the Medicis, like we're patrons of the arts." No surprise,then, that HBO's motto is "It's not TV, it's HBO".

Producing great drama that becomes iconic, hiring the best writers, directors and producers in Hollywood, screening comedies (like Da Ali G Show) that the networks would shudder at, and making miniseries and for-TV movies that would adorn and flatter Tinseltown at its very best is one thing; marketing them is another. The key to selling its programmes can be found in one word: buzz. Buzz means generating media acclamation and positive word-of-mouth chatter from viewers (what Americans call "the water-cooler effect") to create a unique aura around the entire HBO operation. That's where the Emmys are so important to the channel, and why each year it spends millions, far more than the networks, courting judges and pushing its nominees.

The ploy has worked with degrees of success. Some programmes have been beyond even that sort of help, like K Street, created by Steven Soderbergh and George Clooney, which bombed in its first season. And Carnivale has laboured to make an impact, while Six Feet Under struggled to achieve its present cult status. But generally it has worked, and there is no more spectacular example of it in action than The Sopranos.

Given that The Sopranos is available to only a third of the American TV audience, its success has been phenomenal. In its first season, 1999, the series scored 4.3m viewers, which more than doubled to 9.6m in 2000. Though the show had large screening gaps, public appetite for it has never flagged. As the Godfather films prove, America has a fascination with gangsters that The Sopranos skilfully tapped into, but it was the writing that made the show exceptional. One-liners like "I got the feds so far up my f***in' ass, I can taste Brylcreem" (from Tony Soprano's Uncle Junior) put the series on a different cultural plane. The 2004 premiere scored a record audience, 12.1m, which was beaten only by NBC's Law and Order: Criminal Intent, at 15m, while The Sopranos' finale in June was the second most watched programme that night in America, after the first game in the national basketball finals.

But all good things must come to an end. The Soprano writer David Chase has contracted to write just 10 more episodes, although they won't be screened in US homes until January 2006, and while there is talk of an HBO movie to follow, an era is definitely coming to an end. Not just that, but HBO's other hallmark series, Sex and the City, folded in February, also to record audiences.

It is in the context of the closing of this most successful chapter in HBO's history that the channel has turned to, of all things, a western as its showpiece series. But this western is unlike anything that has ever been seen on television. Critically acclaimed in the US and a hit with HBO viewers, Deadwood is coming to British TV screens next month courtesy of BSkyB.

Written by David Milch, a writer on Hill Street Blues and co-creator of NYPD Blue, the genesis of Deadwood is a case study in how HBO chooses its programmes. The story of how The Sopranos was rejected by every network before HBO took it up has entered Hollywood mythology, and one consequence is that talented writers and producers who used to go to HBO last with their ideas now go there first. Milch, one of the most accomplished screenwriters of his age, thought about selling his idea to NBC but instead took it to HBO. "What I actually proposed was a show set in Rome at the time of Nero," he explains. "It was about city cops in Rome, and I was interested in the idea of an environment in which there was order but no law, where essentially people served at the whim of the emperor but had to keep order. I wanted to see what principles of governance emerged." HBO was already developing Rome with the BBC, but Albrecht and Strauss were intrigued. "They asked me to think of another venue where I could engage the same things."

Milch's decision to place his dramatic study of lawless order in the Dakota Territory gold-mining camp of 1876 was, on the face of it, a risk. Westerns hadn't really featured on TV schedules since the mid-1970s, but Albrecht and Strauss had few doubts. "We try to take chances on how we tell stories," explains Strauss, "but to be able to take a classic American genre and put it in the hands of one of the most masterful of American storytellers, that didn't seem to be that risky to us."

Deadwood starts just after the Battle of Little Big Horn, in the midst of one of the least glorious chapters in American history, when General George Custer instigated a war with the Sioux Indians by exaggerating a discovery of gold in the black hills. A few weeks after his death at Little Big Horn, gold was found in substantial quantities — not where Custer said, but further north at a location called Deadwood. Within weeks, thousands flooded to the area: miners, prostitutes, conmen, card sharps, gunslingers, opium pedlars. It is in this primordial, self-governing, grimy and violent stew, filmed on a set the size of two football fields in Melody Ranch, west of LA, that Milch has placed as rich a cast of characters as has ever been assembled for a TV series. And nearly all of them are based on real historical characters, such as Wild Bill Hickok and Calamity Jane.

The unlikely star of the show is Ian McShane — best known in Britain for his portrayal of the roguish antiques dealer Lovejoy, but largely undiscovered in America — who plays Al Swearengen, the foul-mouthed, manipulative, owner of the Gem Saloon, part bar, part bordello, which dominates the mining camp.

It was Albrecht and Strauss who had the idea of casting McShane. He had greatly impressed Hollywood professionals with his portrayal of the ruthless gangster Teddy Bass in the film Sexy Beast, and once McShane heard that Milch was the writer and Walter Hill (Southern Comfort, Wild Bill) the director, he was on the next plane to LA. The choice of McShane was an inspired one. In July, McShane was voted best dramatic actor by the TV critics of America for the role. It will be a big upset if he does not figure prominently in next year's Emmy nominations. And he is clearly enjoying not just this late resurgence in his career, but also his role as the scheming bar owner. "Swearengen knows the lawless days are coming to an end," he says. "He's a primitive man living in a modern age, and longing to stay primitive but knowing that he has to change too." The conflict between Swearengen and Bullock as the town comes to terms with this change is the dramatic centrepiece of Deadwood.

McShane has also become a fan of HBO. "They leave you alone," he says. "People with vision, like Milch or David Chase in The Sopranos, or the guy who did The Wire, David Simon, they leave them alone... They give them the money to get on with it. I'm sure they have their say, but mainly the shows come out 95% as the creators envisaged."

Having talented executives who make clever programming and artistic decisions is only part of the explanation for HBO's extraordinary success. The other part has to do with the channel's business model. Freed from dependence on advertisers and liberated from the ratings system, HBO has a regular and large monthly income from subscribers that enables it to spend money on great television, which in turn enhances the standing of the channel. It reminds Ian McShane of the place where he cut his TV acting teeth: "With HBO, it's like what the BBC used to be but with better food." It's perhaps a place the BBC should think of returning to.