Afew weeks after I’d been given policy responsibility for our next steps on reform of the House of Lords, a colleague sidled up to me with a mischievous smile on his face. “Kick it into the long grass,” he said. “Only place for it.”

I’d had some sympathy for that view. I didn’t regard reform of the Lords as a priority. Removing the vast majority of the hereditary peers in 1999 had dramatically changed the nature of the second chamber. I felt there were other areas where we could better focus our efforts. That changed when Tony Blair handed me responsibility for Lords reform when I became leader of the Commons in 2006. The then prime minister described this as “a hospital pass” – the implication being that, like the recipient of a misdirected pass in rugby, I’d end up bloodied and bruised.

I convened a cross-party group of MPs and peers, plus representatives of the crossbenchers in the Lords and the bishops. My aim was to work towards free votes on an elected chamber, although avoiding the farce which took place in 2003. Peers had then voted for the status quo and MPs had rejected every option.

The more I examined the issue, the more I became convinced of the need for reform. All three main parties had committed themselves to reform in their 2005 manifestos. The Conservatives said they would “seek a cross-party consensus for a substantially elected House of Lords”, the Liberal Democrats promised to replace the Lords “with a predominantly elected second chamber” and Labour promised to make it “more representative”.

We agreed that members of the new chamber – I favour calling it the Senate – would be elected for long periods, with the change phased in over time and with elections staged so that the whole membership of the chamber did not change at the same time. We agreed that the second chamber must be a revising chamber, subordinate to the Commons. We agreed that any link with the peerage should go. A first green paper in February 2007 followed these talks. In the free votes in March, the Commons voted for a wholly or 80% elected chamber.

My first preference was for a second chamber which was 50% elected and 50% appointed. After that idea was roundly defeated, I voted for an 80-20 split. There was broad agreement between the parties on most elements of reform, with differences over the size of any new chamber (450 or 300) and over the voting system. These can be sensibly resolved.

Officials in my department are working on proposals which would set out the way forward towards the most radical shift in our parliamentary democracy for a century. The issue of Lords reform has tortured many before without a successful conclusion. Never before have we been so close to taking the final leap, although I do not underestimate the challenges ahead.

Why have I changed my view and become so convinced of the case for an elected second chamber? In part it has been the journey of the past three years, since Blair gave me that hospital pass. The arguments for maintaining an appointed second chamber are respectable, but I am afraid they do not pass muster in a 21st-century democracy.

Some claim that appointment is more democratic than election. I simply cannot accept that argument at all. There is a crucial role for a second chamber. But it is entirely incongruous that there is no direct link between the parliamentarians who sit in the Lords and the people who elect the Commons. It is as simple as that. I am not suggesting that an elected second chamber would be perfect but it is infinitely more acceptable than appointment.

Then there is the expenses scandal. The direct connection may not be obvious but there is a link. What has happened in relation to expenses is symptomatic of parliament having taken its eye off the ball. The system was allowed to grow without sufficient regard for how people would see it if they knew about it. When they came to know about it, they did not like what they saw – and quite right, too. We will soon be setting out proposals to address these issues. I believe that to some degree the same can be said of the Lords. Just as the public is outraged by the expenses issue, I think the public is no longer prepared to accept a second, unelected chamber.

There are many peers who do outstanding service. The Lords rightly has a reputation for the way it scrutinises the work of the Commons and often puts it right. Yet greater openness and transparency will mean it will become increasingly hard to sustain a model which eschews any form of democratic accountability.

This is not a zero sum game. There is no reason an elected second chamber cannot be even more effective than the Lords is at present. There is no reason it will not contain outstanding individuals of all parties and none. But it would also carry with it the crucial legitimacy that it was put there to do its job by the people.

Times Recommends

• Secret files show Thatcher's role in coup plot
• Veterans to boycott Shell after ban on poppy sales
• 'I survived so please let Baby RB live too'