Inject data form Json file

[Saurabh_G]

Hello Experts,

Can one help me on how to inject data form json format file to AEP.

Kindly provide process steps or supporting documents.

Thanks

[AmitRaghuwanshi · Accepted solution]

Hi @Saurabh_G ,

Here are a few links that might help use AEP APIs.

1. AEP Authentication Help Documentation
2. AEP Postman Collections
3. Adobe Developer Console

Before you test the data ingestion, you should use the postman collection as mentioned in point 2 (there will be a explicit node for Data Ingestion with all required API endpoints. I have also added a screenshot for your reference) and also create the developer console API project as per point 3 and extract the credentials (API key, client secret, and JWT token which are required to call API).

 

Once you have configured the collection and JWT token, you can follow the following steps,

• Step 1. You should first create a Dataset using the AEP web interface and this will require a Schema name.
• Step 2. Now you create a Batch using API. 
  Url: https://platform.adobe.io/data/foundation/import/batches
  This requires Dataset id from above step 1.
  Body: {
  "datasetId" : "datasetid", 
  "inputFormat": {
  "format": "json"
  }}
• Step 3. Once batch is created, you can upload small files below 256 MB using API. .
  This requires dataset id and batch id from steps 1 & 2.
  Filepath will be an adobe path.
  Url: https://platform.adobe.io/data/foundation/import/batches/:batchId/datasets/:datasetId/files/:filePat...
  Attach: JSON file of data in binary format.
• Step 4. After all, files are uploaded. Call the Finish uploading File using API.
  This requires Batch id from step 2 and action param values as complete.
  https://platform.adobe.io/data/foundation/import/batches/:batchId?action=COMPLETE
• Step 5: After ingestion, you can go to Dataset definition using the AEP web interface and can search for the Batch ID created in Step 2 and can see the number of records ingested and also status of batch ingestion.

Hope this helps.

With Regards,

Amit

[Avinash_Gupta_]

@Saurabh_G Adobe Experience Platform Data Ingestion represents the multiple methods by which Platform ingests data from these sources, as well as how that data is persisted within the Data Lake for use by downstream Platform services. There are various ways :-

Streaming ingestion : Streaming ingestion for Adobe Experience Platform provides users a method to send data from client and server-side devices to Experience Platform in real time.

Batch ingestion: Batch ingestion allows you to ingest data into Experience Platform as batch files. Batches are units of data that consist of one or more files to be ingested as a single unit. Once ingested, batches provide metadata that describes the number of records successfully ingested, as well as any failed records and associated error messages.

Sources : Experience Platform allows you to set up source connections to various data providers. These connections enable you to authenticate to your external data sources, set times for ingestion runs, and manage ingestion throughput.

JSON file ingestion is supported from almost any of the above method. Kindly go through the AEP ingestion docs to understand the flow and based on your case decide your ingestion method.

https://experienceleague.adobe.com/docs/experience-platform/ingestion/home.html?lang=en 

[Saurabh_G]

Hi @Avinash_Gupta_ 

thanks for details but I am aware of above details, but I am not sure how to import data from JSON file

[Avinash_Gupta_]

@Saurabh_G 

JSON file ingestion is supported from almost all the ingestion methods. You would have to decide it based on your user case.

For example :- you can use streaming ingestion HTTP API to ingest the data using JSON file. Here is the AEP documentation that provides you all the steps:-

https://experienceleague.adobe.com/docs/experience-platform/sources/ui-tutorials/create/streaming/ht... 

Hope this helps!

[AmitRaghuwanshi · Accepted solution]

Hi @Saurabh_G ,

Here are a few links that might help use AEP APIs.

1. AEP Authentication Help Documentation
2. AEP Postman Collections
3. Adobe Developer Console

Before you test the data ingestion, you should use the postman collection as mentioned in point 2 (there will be a explicit node for Data Ingestion with all required API endpoints. I have also added a screenshot for your reference) and also create the developer console API project as per point 3 and extract the credentials (API key, client secret, and JWT token which are required to call API).

 

Once you have configured the collection and JWT token, you can follow the following steps,

• Step 1. You should first create a Dataset using the AEP web interface and this will require a Schema name.
• Step 2. Now you create a Batch using API. 
  Url: https://platform.adobe.io/data/foundation/import/batches
  This requires Dataset id from above step 1.
  Body: {
  "datasetId" : "datasetid", 
  "inputFormat": {
  "format": "json"
  }}
• Step 3. Once batch is created, you can upload small files below 256 MB using API. .
  This requires dataset id and batch id from steps 1 & 2.
  Filepath will be an adobe path.
  Url: https://platform.adobe.io/data/foundation/import/batches/:batchId/datasets/:datasetId/files/:filePat...
  Attach: JSON file of data in binary format.
• Step 4. After all, files are uploaded. Call the Finish uploading File using API.
  This requires Batch id from step 2 and action param values as complete.
  https://platform.adobe.io/data/foundation/import/batches/:batchId?action=COMPLETE
• Step 5: After ingestion, you can go to Dataset definition using the AEP web interface and can search for the Batch ID created in Step 2 and can see the number of records ingested and also status of batch ingestion.

Hope this helps.

With Regards,

Amit

[Manoj_Kumar_ · Accepted solution]

Hello @Saurabh_G 

One way to do this is via the API as other have suggested. But if you are looking at a one-time import then open the dataset. You can just drag and drop your JSON file to upload data to your dataset.

[soukumar]

Hi Manoj,

Do we have any way in AEP where we can set the mapping for json/parquet files while uploading like we do for csv files ?

[santoshranghar]

Hello Experts,

I Want to upload data in dataset using JSON. Could you please share the JSON structure which need to follow to upload it successfully.

JSON contains three key like person_id, product_id,product_name. Below is screenshot for your reference.

Regards,

Santosh