Speaking on the podcast, he said: “I don’t think he was (the favourite).
“I think it was a more nuanced relationship than that.
“He was the one that she worried about, he wasn’t a sort of golden boy who could do wrong, he was potentially where trouble lay.
“She recognised that.”
The late Queen was believed to be close to her son Andrew as a result of a number of shared interests, including horse-riding.
At the moment, the former prince is under investigation for misconduct in public office.
Andrew has always denied any wrongdoing or accusations against him.