Replying to my own question!
We were able to get the process working.
- Every step after the base source connection must be done via APIs.
- When we set up source connections (file) to the base source (s3 bucket) we had to specify the exact file, not just a folder.
- In the source connection API the data block needs to contain "format" : "delimited" for tabular data
- We had to create a custom mapping set to represent the data as it would be once it was decrypted
- We did this with a small test file and next will load a larger sample of real data
