Prince Andrew is reportedly "proud of his status" and believes he should not "downgrade" to Frogmore Cottage, according to former BBC royal reporter Michael Cole.
King Charles has requested that Prince Andrew relocate from Royal Lodge to the smaller Frogmore Cottage, but he cannot legally force him to move due to a 75-year lease on the property.
Speaking to GB News, Cole said: "Almost everybody would actually like to live there [Frogmore Cottage] and it would be beautiful compared with probably the houses they do live in.
"But Prince Andrew, who is very, very proud of his status and asserts it whenever he feels he has to, will not want to downgrade.
"I think what is at the back of the King's mind, it's quite understandable, is that he would wish his oldest son the Prince and Princess of Wales and their three children, they'd have more space there if they were able to live at Royal Lodge and it would be more appropriate for them.
"At the moment they're living in Adelaide Cottage, which is a quite nice place on the other side of Windsor Great Park."
He added: "The Royal Lodge is a Georgian House that is three storeys high and set in the middle of Windsor Great Park, surrounded by trees and beautiful gardens."
"Prince Andrew still lives there and strangely enough his ex-wife also lives there.
"The King wants his younger brother to move to Frogmore Cottage, which was done up at great expense for the Duke and Duchess of Sussex, but they decided that Canada and then America were more homely for them.
"It's a problem because Prince Andrew has a lease, He has a 75-year repairing lease, which means he has to look after the place.
"Now The King can't move him out legally, but he's letting his feelings know by removing his security.
"He's let the security people who provide the security for Royal Lodge know that their contract will not be renewed when it expires.
"Now Prince Andrew is going to look at this very, very unhappy way."
It has now been reported by The Sun that the King has not renewed his younger brother's private security.
A source told the outlet "Everyone is speculating this means the Duke will have to leave the Royal Lodge, because what other reason could there be to take his security away?"
They added: "It isn’t a secret that the King wants him out."
The presenter left the ITV show in October 2023
It comes as Prince Harry is set to celebrate his 40th birthday
Adam Sandler and Netflix share first glimpse from the set of upcoming sequel
James Middleton shares heartfelt anniversary post after Princess Kate's chemotherapy treatment
Meghan Markle is reportedly "planning a party" for Prince Harry
Jennifer Lopez files for divorce from Ben Affleck last month