Import in Delta mode

[Frankie_d]

Hi All, 

I came across this best practice v7 document noting that 'regular data imports must be done in delta mode' to import only modified or new data sets to ACC instead of the entire table. Would anyone know if there is an additional document or guide to importing data in delta mode or can anyone explain how to do this? 

Cheers
Frankie 

[Marcel_Szimonisz · Accepted solution]

Hello @Frankie_d,

It is more of methodology than some kind of mode you can turn on inside adobe campaign

"Delta import is a data integration method that focuses on bringing only the new or modified data from a source system into a target system. It's particularly valuable when dealing with large datasets where transferring the entire dataset each time would be inefficient and time-consuming."

Delta import can be done with many ways:

1. CSV/Excel Import: This is a straightforward method where data is prepared in CSV (Comma-Separated Values) or Excel files and then uploaded directly into the system. Many applications support this method.

2. Database-Level Import: You can use SQL commands and tools to import data directly into a database. This involves writing SQL scripts or using database import/export features.

3. Integration Software: Integration platforms like Apache Nifi, Apache Camel, Talend, or MuleSoft can facilitate data import between systems. They provide connectors and workflows to automate data transfers.

4. ETL (Extract, Transform, Load) Tools: ETL tools like Apache NiFi, Apache Spark, Talend, Informatica, and Microsoft SSIS are designed for data integration tasks. They can extract data from various sources, transform it, and load it into the target system.

5. APIs (Application Programming Interfaces): Many systems offer APIs that allow programmatically importing data. You can develop custom scripts or applications to use these APIs for data transfer.

6. FTP (File Transfer Protocol): If your data is in files, you can use FTP or SFTP to transfer files securely between systems. Many integration platforms also support FTP-based transfers.

7. Web Services: If the source and target systems support web services (e.g., RESTful APIs or SOAP), you can use these to import data.

8. Message Queues: For real-time or near-real-time data import, message queues like RabbitMQ, Apache Kafka, or AWS SQS can be used to transmit data between systems.

9. Custom Scripts: For unique or one-time data import tasks, custom scripts or programs can be developed in programming languages like Python, Java, or PowerShell.

10. Bulk Loading: Some databases and data warehouses offer bulk loading options specifically designed for efficient large-scale data imports.

11. Cloud Data Import Services: Cloud platforms like AWS, Azure, and Google Cloud offer services and tools for importing data into their ecosystems.

12. Data Migration Tools: Specialized data migration tools and platforms are available for transferring data between different database systems or cloud platforms.

13. Middleware: Middleware solutions can facilitate data integration between different systems by providing a layer of abstraction and transformation capabilities.

14. Manual Entry: In some cases, manual data entry is still a valid method, especially for small-scale data import tasks.

15. Change Data Capture (CDC): CDC tools capture and replicate changes made to source data in real time, facilitating continuous data import into a target system.

16. Data Sync Tools: These tools help synchronize data between systems, ensuring consistency in real time or on a scheduled basis

Marcel Szimonisz

MarTech Consultant
for more tips visit my blog
https://www.martechnotes.com/

[SD_11]

Hello @Frankie_d,

You can create a recurring import workflow to import your delta file.

Create a workflow as below and modify the workflow based on your business requirement.

For full configuration of this workflow, follow the setup guidelines from the link below:
https://experienceleague.adobe.com/docs/campaign-classic/using/automating-with-workflows/use-cases/d...

Regards,

Sanchari

@ParthaSarathy