Religious tolerance

When Muhammad (saw) conquered Makkah why didn't he just remove the idols instead of destroying them? I understand that the Kaabah is a place of worship for Muslims but removal would have been a valid option too. Doesn't that conflict with no compulsion in religion and freedom of non-muslims? P.S I'm a believing and practicing Muslim engaged in a debate with an ex-muslim