Restricting the number of rows imported from source connector in AEP

[arpan-garg]

Hello,

I was wondering if there is a way to restrict the data pulled from the source connector. Currently, I have a table in AWS Redshift with approximately 250,000 profile records. However, my development sandbox license only allows for 10,000 records. Therefore, I would like to import only 10,000 records to test if everything is functioning properly. Unfortunately, I haven't found an option to limit the data during the connection process. One potential solution could be to create a new table and transfer 10,000 records from the original table, then link this new table to the AEP dataset. I'm curious if there's an out-of-the-box solution for this.

[adobechat · Accepted solution]

Hi @arpan-garg ,

Although Adobe Experience Platform (AEP) provides a rich set of features to ingest and process data, it may not offer a direct out-of-the-box solution to limit the number of records ingested from a specific source connector that I know off.

However, you can use the following workarounds to achieve this:

Create a View in Redshift:
You can create a view in your AWS Redshift that limits the number of records from the original table. To do this, execute the following SQL query in Redshift:
sql
CREATE VIEW limited_records_view AS
SELECT * FROM original_table
LIMIT 10000;

Then, in AEP, configure the Redshift source connector to pull data from the limited_records_view instead of the original table. This way, you will only ingest the limited number of records.

Use a custom query in your source connector:

You can specify a query that limits the number of records to be fetched. The query should look like this:

sql
SELECT * FROM original_table
LIMIT 10000

Pre-process the data using AWS Glue or Data Pipeline:
You can use AWS services like AWS Glue or Data Pipeline to create a separate table with the limited number of records from the original table. After creating the new table, configure the AEP dataset to ingest data from this limited records table.
While none of these methods are strictly out-of-the-box, they do provide a way to limit the data ingested by Adobe.

Hope this helps.

Regards,
Madhan

[arpan-garg]

Hi Madhan,

Thanks for your input, it was very helpful.

Regarding the redshift connector, I have come to the understanding that there is no option available to pass a custom query, which rules out that option. However, I can create a new view and use it to ingest the data. I was just curious to know if there is an out-of-the-box solution available.

Once again, thank you for your suggestion, it was greatly appreciated.

Best regards,

Arpan