Time spent per visit (seconds) and Average Time Spent on site (seconds) are almost the same (when looking at something like a Daily breakdown)... but since time spent is based on calculating the difference between time stamps from page A to page B, and one of them has extra logic, you are right that these can show up very differently.
Now in general, either of these should work, but here is information about each metric:
https://experienceleague.adobe.com/docs/analytics/components/metrics/time-spent-per-visit.html?lang=en
https://experienceleague.adobe.com/docs/analytics/components/metrics/average-time-on-site.html?lang=en
Time spent per visit (seconds) has some extra logic to calculate the time against each visit
ie. [Total seconds spent] divided by ([Visits] minus [Bounces])
If I look at the two metrics side by side, broken down by Day, and then again broken down by Page... Time spent per visit (seconds) seems to be closer to having the same average for the month? So based on that, my leaning would be to use this metric..
Since Time Spent is all based on page view data, I really don't understand why Average Time Spent on site (seconds) comes out to about half the average as Time spent per visit (seconds) when broken down by page... so I share your concerns about trust....
Also, I was helping someone a few months ago, and we found that Time spent per visit (seconds) was more stable when using it against custom date ranges... so again, that's a good indicator that this metric is probably the better of the two to use.
https://experienceleaguecommunities.adobe.com/t5/adobe-analytics-questions/custom-date-range-doesn-t-work-with-average-time-on-site/m-p/586753
Since we also have another tracking system on our sites that has heartbeat tracking, I generally tell people to use it over Adobe's data, but IF I were to pull a report in Adobe, I think Time spent per visit (seconds) is the one that I would use.