Don't use the Activity Map plugin, use the Activity Map data in Workspace, then you will be able to see all link usage over time and not just what is visible...
Of course, this also relies on having a good Activity Map implementation (actually setting region names instead of letting Adobe choose the region name based on developer set id which are largely unreadable and likely to change).
I use this:
https://experienceleague.adobe.com/docs/analytics/analyze/activity-map/link-tracking/activitymap-link-tracking-methodology.html?lang=en
Though instead of using "lpos" I use "data-lpos" so that it conforms better with HTML standards.
I work with our developers to break the page into the regions that I want/need granularity on. So that when Activity Map data is collected, I get regions such as "main menu", "header", "footer", "sub-nav", etc... I even have them create dynamic naming for content blocks based on the heading for the area "More News" becomes "more-news" and "Recommended Content for [user name]" becomes region "recommended-content" (removing the user name for both privacy and to allow that block to roll up for all users)
Then if I need to look at how the main menu or the quick links, or whatever is performing, I can segment on Activity Map Region = "main-menu", pull in the Activity Map links and use the Activity Map Link Instances as my metric.
You can go even further into this if you want, you can override the Activity Map Links with additional data (such as the link and the position, etc)... and while you cannot create classifications on Activity Map directly, you can use processing rules to copy Activity Map data into another dimension (like a prop) and create classifications on it to split out the link from the position data so that you can correlate behavioural changes over time.