You should separate the question of upgrading to AEM 6.4 from the performance discussion. The question of support should be justification enough to upgrade.
Regarding the question if you should use Oak-Solr (powering JCR queries by Solr) or not: From my point of view I haven't seen valid reasons to prefer Oak-Solr over the embedded lucene. You can improve the search performance in many ways even with the embedded Lucene engine. I would recommend only to use oak-solr when you have done extensive performance tests; I am not aware of any recommendation when to use it (thus you need to develop that experience on your own).
If you don't consider oak-solr, but if you want to have a site-search (end users are searching your site), then I would always prefer Solr over JCR queries (no matter if backed by Lucene or Oak-Solr). You only need to find out which option is best suited for you to get your content into the solr.
Jörg