Restricting the number of rows imported from source connector in AEP

Hi @arpan-garg ,

Although Adobe Experience Platform (AEP) provides a rich set of features to ingest and process data, it may not offer a direct out-of-the-box solution to limit the number of records ingested from a specific source connector that I know off.

However, you can use the following workarounds to achieve this:

Create a View in Redshift:
You can create a view in your AWS Redshift that limits the number of records from the original table. To do this, execute the following SQL query in Redshift:
sql
CREATE VIEW limited_records_view AS
SELECT * FROM original_table
LIMIT 10000;

Then, in AEP, configure the Redshift source connector to pull data from the limited_records_view instead of the original table. This way, you will only ingest the limited number of records.

Use a custom query in your source connector:

You can specify a query that limits the number of records to be fetched. The query should look like this:

sql
SELECT * FROM original_table
LIMIT 10000

Pre-process the data using AWS Glue or Data Pipeline:
You can use AWS services like AWS Glue or Data Pipeline to create a separate table with the limited number of records from the original table. After creating the new table, configure the AEP dataset to ingest data from this limited records table.
While none of these methods are strictly out-of-the-box, they do provide a way to limit the data ingested by Adobe.

Hope this helps.

Regards,
Madhan