I basically want to create a custom index where in I want to run a query on entire repository ("/content" for example) and take only nodes (pages/contents) between two different dates.
Right now, its traversing the entire repository, can a custom index on date reduce, if so how can I do that ?
Use Oak Utilities : Index Definition Generator to generate appropriate index per your query
Tweak the index, if required per Jackrabbit Oak – Lucene Index
Use 'nrt' for same or otherwise manually reindex based on your use case. - async = ['async', 'nrt']
Try to go through the docs to understand how it works - Jackrabbit Oak – Indexing
how about new contents which got created, do they automatically be added in index or a reindex is needed everytime ?
In addition to Gaurav response - for those reading this thread and wanting to know more about Indexing - please watch this GEMS session - Best Practices for Indexing and Queries in AEM