Is there a separate report to see the performance of the recommendation engine in Web Personalization/RTP?

Yes. There is a daily/weekly Recommended Assets email report that can be configured to be sent to users.

Recommended Assets Report

This email report includes all the pieces of content and count of clicks on the Content Recommendation Bar.

To customize which automated email report the user receives, see: Email Reports - Marketo Docs - Marketo User Manual .