From the age of 10 I have been sharing my most intimate moments with some highly unsuitable men: a would-be bigamist with a penchant for dressing up in women’s clothes, a gun-running philanderer who isn’t averse to a little marital rape, a misogynist snob with more money than sense and a sociopath who likes to strangle puppies. Despite their failings as human beings these men have been there for me through teenage heartache, marital difficulties, new babies, new jobs. In fact I rarely go to bed without one of them.

I am talking, of course, about Messrs Rochester, Butler, Darcy and Heathcliff; a fictional quartet who have the power to reduce me to a quivering jelly of acquiescence with one sardonic arch of the eyebrow. Real men have their uses, but also their drawbacks. It is hard to imagine Rochester snoring or Darcy cutting his toenails in bed.

I am not alone in my serial infidelity with fictional men. Romantic fiction, which I define as a book that no man would be seen dead reading, accounts for something like half of the novels sold in this country. Every two minutes somewhere in Britain a woman is buying a Mills & Boon. But for all its commercial success, romantic fiction is pretty much ignored by the literary mavens. Other genres, like crime or sea stories, get serious attention, but the pile of books with pastel covers and winsome line drawings get barely a mention.

It is easy to understand why: if you are a man there is really nothing in these books to interest you and if you are a woman with any kind of literary pretensions, to admit that you are a reader of romances is the equivalent of saying you prefer books with larger print.

I’m not going to argue that romantic fiction deserves more attention because of its intrinsic literary merit: there are some really good writers who are ignored critically because of the “chick lit” label, but there are some really duff ones as well. No, the reason romantic fiction should be scrutinised more thoroughly is that it is the code to unlock what women really want.

We go to see movies together, and TV watching is a shared experience, but reading is a private pleasure. I pick up a romantic novel not so I can impress my friends with having read it, but because I know that while I am reading it I can leave the world behind. There will be a heroine that I sympathise with, a hero I want to share a sunset with, and I know that the outcome will be happy.

Personally I don’t want sex unless it is comically bad; what I want is 250 pages of delayed gratification with my lovable heroine and irresistible hero finally coming together in the last chapter. Like most women who read romantic fiction, I am only too painfully aware that fiction is precisely what it is, but I can carry my disbelief alongside my innermost conviction that living happily ever after is not just for fairytales.

One of the men behind Mills & Boon famously described his books as “better than Valium”, a remark as self-defeating as Gerard Ratner’s “total crap”. Reading romantic fiction is soothing but the effect is not to numb the pain so much as to reaffirm one’s belief in life. Women need the grown-up fairy stories of romantic fiction in order to make the random cruelty of everyday life bearable. And before men sneer at women who read romances, they should ask exactly why they need to read a book about the siege of Stalingrad or the SAS. Do they perhaps find facts less threatening than stories that deal with emotion?

But while romantic fiction is quite rightly written to a formula, it is hardly static as a genre. In the series I have made for the BBC, I argue that it is a barometer for women’s aspirations.

When I was growing up in the late 1970s and 1980s the trend was for heroines who conquered the boardroom as well as the bedroom. Emma Harte, the heroine of Barbara Taylor Bradford’s novel A Woman of Substance, is not only beautiful and desirable she also has the “most astute business brain in the country”. After a certain amount of trial and tribulation she finds not only love with one of the richest men in the world but builds a global retail empire herself.

Margaret Thatcher had just become prime minister at the time the book came out, and for a few years Emma Harte was one of the women we wanted to be: indomitable but tender, down to earth but naturally chic, size 10 by temperament as well as girth. A decade later Harte and her shoulder-padded counterparts were superseded by calorie-counting Bridget Jones, whose commercial aspirations extended to paying off her credit card.

But like Harte in the 1980s, Bridget Jones exactly matched the feminine zeitgeist. Suddenly, having a private jet seemed less appealing than a fridge full of chardonnay and a man of one’s own. The heroine for the noughties has not yet emerged, but romantic novelists, many of them former journalists, are adept at spotting what women want, so it is only a matter of time before a book and a heroine emerge that makes us all want to reinvent ourselves again.

And it goes without saying that the creator of this heroine will be a woman. Of course there are brilliant female characters created by men: Becky Sharp, Natasha Rostova, Madame Bovary, but I doubt whether many women identify with these women in the way that we carry Elizabeth Bennet, Jane Eyre or Scarlett O’Hara around in our heads.

If you are creating a character, it doesn’t matter what sex you are, just how talented a writer you are. But if you want to create a heroine, a fictional character who lives on and off the page, who becomes a benchmark for women’s aspirations, then you have to be female. I pity Emma Bovary and there are days when I recognise aspects of myself in her, but I certainly don’t want to be her in the way I want to be Elizabeth Bennet.

By the same token I don’t think a woman could have created James Bond, Biggles or even Peter Pan. Heroines, or heroes, are more than literary confections, they are the people we would like to see in the mirror. Movies are the exception to the male/female rule: many of the great heroines have been created by men, but of course they are played by women. Ilsa in Casablanca or the hooker in Pretty Woman are heroines because they are played by Ingrid Bergman or Julia Roberts.

I think that the reason so many men dismiss romantic fiction is that they find it rather threatening. Do women really want men who have to be tamed like Rochester, or humbled like Darcy? Surely most women would protest strongly at being swept up in what Mills & Boon calls a “punishing kiss”? If I had a son I would definitely prescribe a course in romantic fiction, much as in the old days young men were sent to brothels to learn how to please a woman. A careful study of the oeuvre would, I guarantee, give him an unparalleled insight into women’s mental geography which if skilfully used could be quite dangerous. It is lucky, then, that most men would rather have their toenails pulled out by trained badgers than buy a book with a pink cover. Our secrets are still safe.

Daisy Goodwin’s Reader, I Married Him begins tomorrow on BBC4 at 9pm