Get data from 3rd endpoint and generate csv file

[neeliv]

Hi,

I am trying to fetch data from 3rd party endpoint and load it into one of data schemas in ACS. Using just External API activity is not an option because each time the workflow is run I need to have delta of records so the ones that appeared from the last workflow has been running.

The mentioned 3rd party endpoint has the option to put filters on query strings, so for instance it can looks like this:

var datenow = Date.now();
lasthour = now.setHours(now.getHours() - 1);

(date_modified >= lasthour) AND (date_modified < datenow)

To achieve this goal I've tried to use Adobe IO in the middle and I found that there is one similar use case described here: https://medium.com/adobetech/adobe-campaign-content-recommendations-in-email-b51ced771d7f

My script for Adobe IO looks as follows:

async function main () {
    try {
      //Declare all the options for GET request along with query parameters
      const fetch = require('node-fetch');
      const response = await fetch('https://endpoint.com', {
      	method: 'get',
        headers: {
          "Subscription-Key": "XXX",
          "TENANT": "YYY"
        }
      });
      const data = await response.json();

      console.log(data);

      //Generate current timestamps
      var datenow = Date.now();
      var lasthour = now.setHours(now.getHours() - 1);

      //Get the returned json and transform it into csv file
      if (data) {
        var json = data
        var fields = Object.keys(json[0])
        var replacer = function(key, value) { return value === null ? '' : value }
        var csv = json.map(function(row){
          return fields.map(function(fieldName){
            return JSON.stringify(row[fieldName], replacer)
          }).join(',')
        })
        csv.unshift(fields.join(',')) // add header column
         csv = csv.join('\r\n');

        return { message: csv }
      }

    } catch (error) {
    // log any server errors
    //logger.error(error);
    // return with 500
    return {
      statusCode: 500
    };
  }
}

Based on it I created the action and then api using wsk. However it doesn't work, it returns either 500 http code or 400 - depends which --response-type I'm using. I've tried to create api with all response types listed in documentation (html, http, json, text, svg).

I'd appriciate any help to make it running.

Thanks,

Dominik