Once again the Labour Party conference has two leader's speeches. It used to be Gordon Brown, setting out his “real Labour” alternative to Tony Blair. This time it was David Miliband's turn to whisper: “Things could be different under me.”

As the Foreign Secretary called for fatalism in the party to be replaced by hope and urged “leadership from a party dedicated to change”, it was not too difficult to decipher the code. This is not a man who is going to “stand by”, as he put it at a Progress Rally on Sunday night, and see new Labour destroyed on polling day.

It was not a spine-tingling speech, the delivery was rather wooden. But the content was carefully calibrated, with the help, incidentally, of some of the speechwriters who once worked for Mr Blair.

The young pretender sashayed from foreign to domestic affairs, emphasising that his party should be about the future not the past. He made the case that the Conservatives were “beatable” - an implicit criticism of Mr Brown's failure to make an impact against them in the polls.

Mr Miliband tried to scotch the suggestion that he is just the heir to Blair by saying that Labour should want to narrow the gap between rich and poor. He even started with his own “back story” (a classic leader's speech ploy), describing how his grandfather was refused the right to naturalise in Britain by a former MP for his constituency, South Shields.

It wasn't exactly disloyal - the Foreign Secretary turned around, a little clunkingly, to the Prime Minister at one point and praised his leadership on international aid - but it was crafted to demonstrate his availability for the top job. Mr Miliband has no intention of challenging Mr Brown directly, but he is cheeking him.

The boy David is pelting Goliath Gordon with pebbles and it remains to be seen whether the Great Clunking Fist will be able to resist snatching the catapult and smashing it over his head.

Mr Brown cannot be happy. He thought he had slapped down this puppy over the summer, when he sent out the Downing Street attack dogs to accuse him of “self-serving” disloyalty, but he is popping up all over the place, just as the Prime Minister's own leadership is under fire. Not only did Mr Miliband co-operate with two in-depth profile pieces last weekend (including one in The Times magazine), he has also agreed to speak at eight fringe meetings in four days.

All over Manchester Mr Miliband has been “setting out his vision”, followed by photographers looking for evidence of a plot. He has reached out to the Left, calling for “checks and balances that discipline the abuse of market power” in the City; he has distanced himself from the war in Iraq, admitting that Britain and America have failed to “win the peace”.

Leaders have “got to inspire, you've got to instruct, you've got to motivate, you've got to discipline”, he told the Daily Mirror - a pretty clear description of all the things Mr Brown is failing to do.

Alan Johnson's endorsement of the Mili-bandwagon has encouraged the sense of momentum - and removed one of the Foreign Secretary's most powerful rivals. There is speculation that they could form a dream ticket, intellectual rigour with a populist touch. When the Health Secretary went the bar of the conference hotel during the weekend, John Prescott shouted out to him: “Oi, Alan, where's David?”

In private, ministers discuss who would be the stars of a Miliband Cabinet, Labour donors contemplate whether they would be willing to contribute to his campaign. When Ed Miliband went to the Democratic convention in the United States last month he was rather surprised to be asked whether he felt awkward about having to choose between his brother and his boss. At the moment Ed Miliband is loyal to the Prime Minister - despite knowing his faults (when he was Mr Brown's special adviser he once deleted his number from Mr Brown's mobile phone because he had had enough of being called at all hours of the day and night.)

Last year the Labour conference was all about the election - something that was never discussed on the platform, this year it is all about the leadership - but the matter is being studiously avoided in the hall. The economic meltdown has secured the Prime Minister's position for now - even his harshest critics admit it would look “extraordinarily self-indulgent” for the Labour Party to swap leaders when there is a global financial crisis. But there is a general view that it is only a temporary stay of execution for Mr Brown. “This changes everything,” one Cabinet member told me, then paused before adding: “Until next week”.

Ministers are gaining in confidence as the leader's authority fades - the diminutive Hazel Blears laughed off the idea that she is the plotters' Miss Big by promising to buy a Blofeld-style cat to stroke.

The Blairites are planning to assert themselves more in the coming months. “It's about rights and responsibilities,” one senior minister told me. “We have a responsibility to be loyal to Gordon but we also have the right to say what we think about the direction of the party.”

The threat of a reshuffle has simply hardened resolve. “You've got to stand up to bullies,” a Cabinet member said. “Loyalty?” adds another, “that's a bit rich from him.”

Senior figures believe it is no longer a matter of if but when. The majority of the Cabinet are now divided between those who want the Prime Minister to go in the next few weeks, and those who think he should be allowed to stay on until the local and European elections next June.

Mr Miliband is more impatient than some. But will he act? Today, when he stands to address the conference for the real leader's speech, the Prime Minister will get the chance to respond.

Times Recommends

• Can I be complimentary, my dear Watson?
• Hardliners realise Britain is a soft touch
• The jaw-dropping truth about Gordon