AEP to have direct integration with Databricks

[#1]

Description:
Enable a native integration between Adobe Experience Platform (AEP) and Databricks to allow seamless data exchange for advanced analytics, machine learning, and large-scale data processing.

Why is this feature important to you:
Databricks is widely used for data science, modeling, and big data workflows. Many organizations already use both AEP and Databricks but face challenges moving data between the two. A native integration would eliminate friction, reduce reliance on custom pipelines, and unlock more value from both platforms by enabling real-time insights, model scoring, and enrichment of profiles in AEP.

How would you like the feature to work:
Users should be able to configure a connection to a Databricks workspace directly from the AEP UI. They could export datasets from AEP to Databricks, run transformations or machine learning models, and then write enriched results (e.g., scores, segments, predictions) back to AEP. Ideally, this would support scheduled or triggered jobs and respect identity namespaces and schema alignment.

Current Behaviour:
Today, connecting AEP to Databricks requires custom ETL workflows, cloud storage intermediaries (like S3 or Azure Data Lake), or APIs. This adds complexity, latency, and cost, and often involves multiple teams. There’s no streamlined, supported connector between the two platforms, limiting agility for data science and activation use cases.