When Sir David Scott was a boy, he never paid more than £25 for a painting. “I still cannot refrain from thinking sometimes of the wonderful things I could have got had I been able to afford £50,” the art collector once said.

Over almost a century of buying only what he loved - he never bought for investment - Sir David amassed, with his second wife, one of the most remarkable private collections in Britain. After his death in 1986, his widow, Valerie Finnis, continued to care for the works until she died in 2006.

The Scott collection, which goes on show in Edinburgh next week before a sale in London on November 19, is regarded as an unprecedented body of Victorian and 20th-century art, featuring many celebrated paintings that have been unavailable for generations.

Among the the masterpieces is the iconic Victorian painting by Sophie Anderson (1823-1903), No Walk Today, depicting a wistful girl in velvet and feathers looking out of a window.

Sir David, the grandson of the 5th Duke of Buccleuch, bought it for 14 guineas (£14.70p) in the 1920s. A supplementary obituary in The Times described how he was detained at work and missed the sale but found the picture unsold and bought it afterwards. It is expected to raise between £600,000 and £800,000 at auction.

Sir David and his wife, whom he married when he was 83, were noted gardeners as well as art lovers. They had no children. The proceeds of the sale, expected to raise £5 million, will go to the Finnis Scott Foundation, set up to benefit charitable causes in art and horticulture. The couple, who lived with their paintings hung around their home at Boughton, Northampton, were so enthusiastic that they asked visiting experts to write information and comments on the wallpaper next to the picture frames.

The career diplomat bought his first painting - a watercolour by Thomas Whitcombe of a naval battle - before the First World War and it became a catalyst for a lifetime of acquisition. The result is a 240-piece collection of a range and freshness that is said to be one of the last of its kind.

Sir David, who was born in 1887, focused primarily on narrative Victorian paintings because he loved the stories that they told. He remained an enthusiast for the genre at a time when its reputation was at a low ebb.

Grant Ford, senior director and head of Victorian art at Sotheby's, described the collection as extraordinary. He said: “Not in all my time at Sotheby's - a period of 22 years - has a collection of this quality come on to the auction market. The Scotts were collectors in the truest sense; they had an individual and discerning taste and they only ever bought paintings that they truly loved and understood and which said something special to them.”

As a young man working at the Foreign Office, Sir David would forgo lunch in a Pall Mall club with his contemporaries and instead cycle around the West End visiting the galleries and salerooms. His eye was unerring. He once said: “I have never bought a picture as an investment but solely because I liked it and wanted to own it and live with it. I cannot describe the exquisite enjoyment I get from looking at certain pictures . . . I want to go on looking at them, somehow to absorb them and make them part of my life.

“At intervals, when I needed money, I sold some disastrously - two Boudins, for instance, when I wanted money for going to Japan in 1936, for £136 each at Sotheby's - and again, when once more I went to Japan, a superb Salmon, bought for £13 in Edinburgh.”

The collection includes works by Sir John Everett Millaisand Rebecca Solomon. A painting by John Anster Fitzgerald that depicts a sleeping girl surrounded by dream scenes and grotesque figures is expected to sell for up to £600,000.

One particularly interesting work is Franz Xaver Winterhalter's La Siesta, which hung in Queen Victoria's dressing-room at Osbourne House on the Isle of Wight from 1841 to 1876. In her diary in 1841, the Queen wrote:

“Today I got from Paris a beautiful picture by Winterhalter which I had ordered.”

Sir David, whose first wife died in 1965, had only one child, a son who died fighting in the Second World War.

Times Recommends

• In Margot Fonteyn's footsteps
• Forget beautiful people, let's have wonky Brits
• Fundamentalist brew: faith and power