Update dataflow mapping before source file arrive with new field

[MV1]

Hi @members, 

I wonder what's the best solution to update a dataflow, here is the background: We have a existing scheduled dataflow, which will ingest data every day, and there are mapping, dataset, schema associate with this dataflow.

If we need to add a new field to the schema, and would like to use this dataflow as well, what changes we need to make? Please correct me if I'm wrong:

1. Update the schema with new field

2. Update the mapping with new field

I know we patch the dataflow with the new field in mapping through API but is there any option through UI to update the dataflow by mapping new field going to start flowing next day? If I do it from UI, I have scheduled dataset so when we wait for source file to arrive it ingest the data without new field and then I have to update dataflow and ingest same file again to have new field in datalake.

Thanks

[supratim320 · Accepted solution]

Hi @MV1 ,

I think you are right. If implemented from UI dataflow mapping, you will have to wait for the change to be introduced in the file and then only you will see that in the preview mode and then map the field in mapping stage.

I would recommend (if possible), to add the changes in schema and add the column in the upstream system's file (even if as a placeholder) beforehand. Otherwise, during change window, you will have to coordinate work with upstream system and make changes in that precise window.

You can give it a try (though I am not hopeful): make changes in schema, dataflow mapping, migrate into higher environment and then see if it breaks (ideally it should not work - so be mindful of backing out then)