This feature has shipped:https://experienceleague.adobe.com/en/docs/experience-platform/query/data-distiller-audiences/overview

This feature is being released at the end of this month in September. 2024 The audience schema for Real-Time Customer Profile has undergone changes where the previous schema is being deprecated. We will be publishing the new schema which we believe is lot simpler. This is the code for the old schema...

The lookup can be a dataset that you import into AEP. Use Data Distiller with Profile Snapshot and you should be good to go. You can generate a set of searches dataset on schedule for verification.

You should read this document I published: https://data-distiller.all-stuff-data.com/unit-5-audiences-and-segmentation/aud-101-build-segment-overlaps-using-profile-attribute-snapshot-datasets Profile snapshot datasets is the answer you are looking for and you can use SQL on it. 

Key AEP concepts that you need to know: 1. When you enable a dataset for Profile, Profile Store and Identity Store (contains the identity graph) monitor for new batches of data from the moment you enabled it. If you disable it, it stops ingesting new data.  2. Deleting a dataset will delete all data...

Remember failed batches of data are not mastered in the dataset in the data lake. They are quarantined and kept separately for inspection. The Catalog API is the best way to get those or better take a look at the monitoring dashboard.

3. Performance guardrail (Soft limit) - When exceeding performance guardrails, you may experience performance degradation and latency. Can adobe define the standard limits for concurrent batch queries, when one can expect performance degradation and latency for 'n' number of concurrent batch queries...

2. As mentioned in the document Maximum execution time is 24 hours? Also Does query service have restrictions on the number of records/size of data query can deal with? If yes, documentation link pls? If not,  is there a adobe's recommendation on the data size? There are no restrictions on the size ...

1. 'N' number of concurrent batch queries are supported as mentioned in the document above. Lets assume each record has 10-15 attributes and this is the only custom batch query running in the platform, If query has to handle 4 million records each day, what's the execution time normally taken by que...

Query Service supports CREATE and INSERT INTO operations on data lake and the accelerated store (the reporting data layer).  Data Lake on the platform today supports an append-only model. This is going to change in the future and Query Service will start supporting UPSERT-style features.  The Accele...