The singer hoped to be awarded retrospective planning permission for the 17-foot extension, which he built in 2006, after Runnymede Council last year told him he had breached the rules.
Now the council has dismissed his appeal against its decision. Sir Cliff now has until March 9 to demolish the structure.
Dave Farrell, Sir Cliff's planning consultant, called the demolition order 'a bit silly'. 'We're disappointed but it's just one of those things,' he added.
Sir Cliff knocked down a bungalow on his Wentworth estate in Virginia Water to build a large mansion and planning rules allowed his new home 30 per cent more area than the original house.
The conservatory was built in 2006, but was deemed to make the new property too big. It was noticed by officials when Sir Cliff Richard put the house up for sale last year for £1.35 million.
Though he applied retrospectively for planning permission, he was turned down.