Qāḍī ʿIyāḍ said:

“Imām Mālik felt it reprehensible for someone to say, ‘We have visited the grave of the Prophet ﷺ.’ Some explained this by arguing that the Prophet ﷺ cursed women who habitually visit graves. That explanation, however, is contradicted by his statement, ‘I had prohibited you from visiting graves; now go ahead and visit them.’

Others contended that the visitor is superior to the visited, but this is refuted by the report that the inhabitants of Paradise will visit their Lord, Exalted is He.

The more correct view is that Mālik objected only to linking the visit specifically to the Prophet’s grave. Had one said instead, ‘We have visited the Prophet ﷺ,’ he would not have disapproved, for the Prophet ﷺ said, ‘O Allah, do not let my grave become an idol that is worshipped. Allah’s wrath is severe against a people who took the graves of their prophets as places of worship.’ Mālik therefore avoided wording that focuses on the grave itself and resembles the practice of those earlier peoples, so as to block any means leading to such excesses.”

This passage appears in Qāḍī ʿIyāḍ’s al-Shifāʾ (2/82–83), quoted here in abridged form.

I add: In popular usage many people now employ the phrase “We have visited” when referring to visits to the graves of prophets and righteous individuals more than they use the expression “visiting graves” for such occasions.