Can the grave of the Prophet ever be worshipped?

The hadith : > O Allah, do not turn my grave into an idol that is worshiped. Musnad ahmad and Muwaata imam Malik. Some people derive from this that the Prophet's grave would never be worshipped as his duas are all accepted unless stated otherwise and hence it was impossible for their to be shirk besides his grave. So basically no one can go their and do shirk their by making dua to him or even by literally making sujud of worship to him as this is not possible. What is the rely to this doubt?