Last Friday, we connected XDM, Schemas via data stream via this post. Reiterating, XDM is a framework on which schemas would be built for data ingestion. Actual data flow to schemas through DataStreams. Schemas are rules on which the dataset would store data in form of tables. This Friday we would see more of Data ingestion from sources and getting the DataStreams.
Source: A source is a general term for any input connector in Platform. Sources in Experience Platform are Adobe applications, advertising, Cloud storage, CRMs, Customer Success, Databases, payments, streaming and protocols.
Source connector: Source connectors (also known as sources) help users easily ingest data from multiple sources, allowing the structuring, labelling and enhancement of data using Experience Platform services. Data can be ingested from a variety of sources such as cloud-based storage, third party software, and CRM systems
Data ingestion: Data ingestion is the process of adding data from a source to Experience Platform. Data can be ingested into Platform in several ways including streaming, batches, or added via source connectors.
Let’s relate all of them in “what to do”
Use Experience Platform to centralize data and collect from disparate sources. This data is further going to AAM, analytics and modelling to give insights about the customer behaviour.
Learn about the various connectors from here.
Learn about the API tutorial about the source connectors here.
Learn about data ingestion from here.
When and why
Data is ingested either by streaming or batch depending on your requirements. Streaming data ingestion is implemented through Launch and batch data ingestion could be done even simply by using an Adobe Experience Platform Workflow to take a CSV-file, map it against an XDM-schema and then ingest it into Adobe Experience Platform. So after you figure what kind of customer preferences you would like know and personalize their experience, you would create schemas to take snapshot at the particular data . Next thing would be configuring data sources for ingestion.
Once schemas are defined, data connector are bundled properly in data streams, you would be able to see the data flowing the profile and events.
Golden Nugget from @ankitasodhi
"There is some expected latency for each source connector. So, plan your use-case considering those delays.
Details on the expected latency for each connector is mentioned the product documentation.”
Found some interesting things using the streaming, HTTP API, connector:
Thanks @jkm-disco for great observations.
Apart from connection, there is need to setup XDM schemas and data streams for data flow. Account profiles enable you to unify account information from multiple sources. This unified view of an account brings together data from across your many marketing channels and the diverse systems that your organization is currently using to store customer account information.
In observation 4, does it mean during 'sending data to specific destination' , dataflow name is superficial?
It would be great if you add observation 3 into ideas so that PM team can have look into it.
With regards to observation 4, when constructing the XDM for HTTP API (stream) ingestion, there are two fields that seem to be optional and don't serve any purpose:
2. There is a field in the request body schema called SOURCE_NAME which is optional, but it seems this value can be set to anything without impacting the ability to ingest the data.
For observation 3, I'll create an idea for it.