It is a truth universally acknowledged that one of the greatest mysteries of English literature is what caused Jane Austen to accept – then refuse – a proposal of marriage. The author was 26 when Harris Bigg-Wither, a rich landowner in Hampshire, asked for her hand. Austen said yes, but trundled up the drive in the family carriage the next day to tell him she had changed her mind.

What happened during the intervening night has kept generations of academics and Austenites intrigued. Did the creator of Elizabeth Bennet and Mr Darcy wrestle with her conscience and conclude that she did not care for him sufficiently and could not marry for money? Did she fear that a career as wife would preclude a career as a writer? Was she put off by his ridiculous name?

Cassandra, Austen’s sister and closest confidante, burnt her letters after she died so we will never know. The decision reverberated through the rest of her life and forms the basis of Miss Austen Regrets, a BBC drama to be aired tonight.

It portrays Austen in later life, riven by doubts about the choices she has made, worried about money and regarded by those around her with a mixture of admiration and fear. Gwyneth Hughes, its writer, is already girding her corsets against the uproar that her portrayal of Austen as chippy, sarcastic, flirty and rather too fond of a glass of wine, is likely to cause.

What she calls the “Janeites” – the legions of (mainly female) fans obsessed with Austen and all her works – are already complaining online that Olivia Williams, the actress who plays Austen, is too tall: just wait till they hear Austen comment sourly, in response to a friend praising Sense and Sensibility, that her writing about love “is like someone who can’t cook writing a recipe book”. Or complaining of having a hangover.

Yet Hughes insists that hers rather than the soft-focus, romantic heroine imagined by her readers is the truer portrait. So closely did she base the dialogue on material from Austen’s surviving letters that she almost feels they should share the writing credit. “If she was coming to tea this afternoon I’d warn you that she was rather prickly and difficult,” said Hughes. “She’d fix you with her beady eye and woe betide you if you said anything stupid. I think she probably was quite scary.

“To this day, it’s difficult for the cleverest girl in the room to pull a boyfriend and I think that explains Jane’s real problem: she was just too clever and too challenging. Everybody thinks of her as this rather shy, retiring country gentlewoman, but I don’t think she was like that at all.”

The Cambridge Introduction to Jane Austen notes that she shares with Shakespeare a rare crossover appeal, with both academic and popular status: the object of scholarly analysis and cult enthusiasm. Pride and Prejudice once topped a poll of “the books that changed your life” on Woman’s Hour on Radio 4.

Like Shakespeare, too, her work seems infinitely adaptable. Pride and Prejudice provided the inspiration for Bridget Jones’s Diary and the movie Bride and Prejudice gave it a Bollywood spin. The Venezuelan director Fina Torres is working on Sense and Sensibilidad, a Latino version of the story to be set in modern Los Angeles.

As Miss Austen Regrets and the recent movie Becoming Jane attest, the author’s own life provides fertile ground for romantic drama. Austen was born in December 1775, one of eight children born to a country clergyman in Hampshire, just beyond the realms of the gentry. Although the family had some wealthy connections, her father augmented his income by taking in boarders whom he prepared for university.

One can assume that it was a bookish household, but the expectation would have been that Austen and her sister Cassandra would support themselves – and assist the rest of the family – by marrying well; marriage being, in Austen’s words, “the pleasantest preservative from want”. As a teenager she seems to have thrown herself into the chase wholeheartedly. One of her neighbours, Mary Russell Mitford, described the young Jane as “the prettiest, silliest, most affected, husband-hunting butterfly ever”.

“Jane was very pretty, we know, with reddish cheeks, she was a born dancer, slim and able and agile and really able to get along in company of all kinds,” said Park Honan, emeritus professor of English and American literature at Leeds University and author of Jane Austen: Her Life. “She was spurred on to some extent by her flirtatious, widowed cousin Eliza, who had been married to a rakish young Frenchman who was guillotined in Paris very early in 1794 when Jane was 18. From that time on, Eliza was often with the Austens and flirted with Henry Austen, Jane’s brother, whom she later married.

“But Jane was already tremendously interested in writing and terribly intelligent even by her late teens – that was as much the centre of her life as the business of finding a husband.”

Her writings were initially intended to entertain the family, but by her early twenties she was working on the outline of three full-length novels. Like her heroines, her social life was a round of visits to neighbours and supper parties that often ended in dancing. She admired a number of men she encountered, noting of her brother’s Oxford friend Benjamin Portal that “his eyes are handsome as ever” and talking of the “beautiful, dark eyes” of Edward Taylor, the neighbour of another brother in Kent (she later wrote of being in a carriage near his house and stopping to linger “by the abode of him on whom I once fondly doted”).

She formed a stronger attachment to Tom Lefroy, the son of a neighbour, although a marriage between them would have been discouraged as neither had any money. How seriously one should regard any of this not clear, though. “I think what you find is that she trained herself to fall lightly in love to see what would happen and to measure the social and emotional result, partly because she was interested in fiction and human nature,” said Honan. “How are you going to find out about relations with the other sex if you don’t try something?”

Austen’s books show she was more than alive to the dangers inherent in remaining single. The Dashwood sisters in Sense and Sensibility have to rely on the largesse of vulgar relations after the early death of their father; Mrs Bennet’s clumsy attempts to find rich husbands for her daughters in Pride and Prejudice is spurred by the knowledge that after her husband’s death they will be penniless.

However, Austen scarcely envied the lot of her sisters-in-law and nieces, bearing babies year after year like “poor animals”. The demands of marriage and motherhood would surely have inhibited her creativity.

All this must have been playing on her mind when Bigg-Wither made his proposal in 1802. “By accepting him she would have become mistress of a large estate in the part of England she loved best. She would not only have a comfortable future but also be able to provide one for her sister and friends,” said Claire Tomalin, another Austen biographer.

Presumably it was the realisation of what the proposal meant to her family – to whom she was devoted – that prompted her to say yes instantly. How could she refuse him? Yet Tomalin believes that had she gone ahead with the marriage it would have been the end of her career as a writer: “She might have managed both childbearing and books – but it seems unlikely.”

Austen wrote later in Sense and Sensibility that “a woman of seven and twenty can never hope to feel or inspire affection again”. Miss Austen Regrets catches her as she is just about to hit 40 – one year before she was to die of a wasting disease. And while her sharp wit keeps her family and friends entertained, there is a bitter note to her musings on life and love.

Austen may have once advised her niece that “anything is to be preferred or endured rather than marrying without affection”, but the reality – dependant on others and making herself useful by sewing her brothers’ shirts and helping to look after her nephews and nieces – was humiliating.

It is a tribute to her lively character that at a time when men were scarce – Britain was at war for almost all her adult life – she caught the attention of so many men, but the right one never came along. “I think she didn’t marry because she didn’t get an offer from someone she liked well enough,” said Hughes. “The sad thing is you think of her as the great spinster of all time. She virtually invented the concept of ‘the one’ – she created Mr Darcy, but she never had a Mr Darcy of her own.”