You could do this a number of ways. Based exactly on your question, you would set up a task report, filter on milestoneID is not blank, and switch over to a matrix grouping where the project name is to the left and the milestone name is above.
I prefer and recommend a project report using the mile...