Hi All
My requirement is every day i will get new file in S3 and need to ingest into my dataset and previous batch need to be deleted.
For this i have used below curl's
To get the batch details
curl -X POST https://platform.adobe.io/data/foundation/import/batches \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'Content-Type: application/json' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
-d '{
"datasetId": "{DATASET_ID}",
"inputFormat": {
"format": "csv"
}
}'
Batch deletion
curl -X POST https://platform.adobe.io/data/foundation/import/batches/{BATCH_ID}?action=REVERT \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
While testing it in postman i am able to view the deletion happen but if i query the dataset i am able to view the records
Can any one suggest me any alternative way.
Thanks in advance.
Thanks and regards
Gowtham Raj
Solved! Go to Solution.
Hello @gowthamr123 ,
If I understood correctly, you followed these steps:
1. Deleted the batch using the "Delete a Batch" API, and I assume you received a 200 response.
2.Then you ran a query to check the batch status.
At this point, the status should show as "deleted."
However, if your data is still visible the next day, it seems the batch was deleted, but the data itself was not removed from the Data Lake or Profile Service.
From the process you've described, it looks like the ingestion and deletion APIs are only managing the batch in the Batch Data Collection.
The batch deletion API does not indicate that it will delete data from the Profile Service or Data Lake.
If you want to remove data from datasets, I recommend using the Data Hygiene API, which is designed for this purpose:
https://experienceleague.adobe.com/en/docs/experience-platform/data-lifecycle/api/overview
Kind regards,
Parvesh
Views
Replies
Total Likes
Hello @gowthamr123 ,
If I understood correctly, you followed these steps:
1. Deleted the batch using the "Delete a Batch" API, and I assume you received a 200 response.
2.Then you ran a query to check the batch status.
At this point, the status should show as "deleted."
However, if your data is still visible the next day, it seems the batch was deleted, but the data itself was not removed from the Data Lake or Profile Service.
From the process you've described, it looks like the ingestion and deletion APIs are only managing the batch in the Batch Data Collection.
The batch deletion API does not indicate that it will delete data from the Profile Service or Data Lake.
If you want to remove data from datasets, I recommend using the Data Hygiene API, which is designed for this purpose:
https://experienceleague.adobe.com/en/docs/experience-platform/data-lifecycle/api/overview
Kind regards,
Parvesh
Views
Replies
Total Likes
Views
Likes
Replies
Views
Likes
Replies