Conditionally include partial data from a source/dataflow into Profile

[Devyendar]

When we profile enable a Schema and Dataset from Marketo, all the data ingested into it is included in profile. But we have data quality issues in Marketo due to which do not want to include all the data into Profile, only certain data matching few qualified campaigns should be included into profile. Marketo doesn't support row level filtering for Individual Records data. Is there any easier way to achieve this?

[Karuppiah_Sakthivel]

Hi,

When a Marketo schema and dataset are Profile-enabled, all ingested records automatically flow into the Real-Time Customer Profile — there’s no way to apply row-level filters within the Marketo connector.

To include only qualified records (e.g, specific campaigns), the easiest approach is to:

1. Ingest all Marketo data into a non-Profile dataset (your raw landing zone).
2. Create a second, Profile-enabled dataset with the same schema.
3. Use Query Service (or Data Distiller) to filter and copy only the qualified rows into the Profile-enabled dataset.
4. Schedule this query so your Profile dataset stays up to date

[Devyendar]

@Karuppiah_Sakthivel We are using the Query approach for the time being, we want to move away for this, as Queries make Marketo data work for only Batch Segmentation (as it is considered as Batch dataset). The original Marketo Connector based data can be used for Streaming Segments and real-time use cases. These use cases are impacted using Query ingestion and query execution on large volume of data can not adversely impact Query performance/failures.

[Karuppiah_Sakthivel]

Good point — using Query Service does make the dataset batch-only, so we lose streaming segmentation and real-time updates from the Marketo connector. It’s a solid workaround for data quality control, but ideally Adobe should let us apply filters directly on the Marketo source to keep streaming use cases intact.

[Devyendar]

@Karuppiah_Sakthivel that's right, Adobe does allow filtering to be applied for Marketo Activity data based on eventType but similar filtering is missing on Marketo Person data.

[itsMeTechy]

You can have the target schema designed in such a way that any records with data quality wont go through and errored out. And in the data flow you have partial ingestion and have a higher number like 50% or more. By this you can restrict data with bad quality not ingested and rest going through. This is a nasty work around but does work.

you can also leverage data prep to find a record is bad record or not. for ex: if email is not in good quality, and email is the identity for Marketo data, you can check against a regex, if the incoming email is not satisfying the regex, then make it null, and mark email attribute as mandatory attribute in the target schema, or you can hard code the email to hard coded email, which will allow the records to come through and will be available in data lake, but since it is hard coded all such bad email records will be under 1 profile. which will also help you to analyze the reason for data quality.