Handling requests in scale | Community
Skip to main content
A
appuputti
Level 2
April 17, 2019
Solved

Handling requests in scale

  • April 17, 2019
  • 3 replies
  • 2428 views

Hi all,

I have a situation where I am required to send millions of requests to internal Rest API. Now after some investigation it seems that HTTPClientRequest is not the way to go. I am looking alternative options without involving some middle-server to drop files etc.

One approach i was thinking is to use execCommand and have my dispatch script written in python or similar in server. Have anybody done anything similar? What is the best way to get the data to the script? Is there a way somehow to open memory buffer/pipe through execCommand and send the details to the script or what?

Regards,

Ari

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by Jonathon_wodnicki

    Hi,

    If you can, try to use files for data exchange.

    If the only option is API calls, GNU parallel with curl and a local file is pretty good for simple request parallelization, example usage here:

    Thanks,

    -Jon

    3 replies

    Jonathon_wodnicki
    Community Advisor
    Jonathon_wodnickiCommunity AdvisorAccepted solution
    Community Advisor
    April 17, 2019

    Hi,

    If you can, try to use files for data exchange.

    If the only option is API calls, GNU parallel with curl and a local file is pretty good for simple request parallelization, example usage here:

    Thanks,

    -Jon

      A
      appuputtiAuthor
      Level 2
      April 26, 2019

      Hi Jon,

      Thanks for your response. I have some follow up questions regarding this. Let me first give some more context.

      We need to send push notifications from Adobe and we have our own centralized notification hub that directs the requests to correct provider (iOS & Android). According to Adobe official documentation, one should use HTTPClientRequest for this. However it is only libcurl and therefore is sending the requests as sequential which in our case is not possible as there will be millions of requests. Also I've heard that this method is not recommended to anybody hence i think it should be removed from documentation.

      Our notification service is rest API and now we need to figure out the best way to provide the requests.

      Currently I am investigating two possible options:

      1. Send the requests from the Adobe campaign server using parallel curl calls or python script with execCommand from workflow.

      2. Possibly build a middleware and serve the audience as file from adobe and send the API requests from the middle server.

      Option number 1 is currently more attractive for us as it requires less engineering.

      Do you know under which privileges execCommand is executed through campaign workflow because when i run simple example script directly from the command line it works fine with the authentication we are using but when execute the same script through workflow with execCommand it seems to send it as user because the authentication doesn't work. As a reference, when send the same post request from javascript box the authentication works fine.

      Thanks,

      Ari

      Jonathon_wodnicki
      Community Advisor
      Jonathon_wodnickiCommunity Advisor
      Community Advisor
      May 1, 2019

      Hi,

      The execCommand() call is executed by wftask, under 'neolane' user, unless someone went off-road on the install. You can try sudo'ing to neolane and running curl by hand there to test. And you're mostly correct, HTTPClientRequest is a buggy subset of libcurl, though it does support parallelization by setting the async flag on execute() and calling wait().

      If you're using an external REST service to send push messages, you may be better served skipping all this entirely, and instead writing a simple js lib with a single HTTPClientRequest connection, then extending the mobile app external account to add your provider and its config. For reference on how to write the js lib, see nms:androidPushConnector.js, specifically its global interface: processDeliveryPart(), getStatus(), getMessages(). The return xml values are undocumented and have to be derived empirically- this is what I spent most of my time on when making similar. When it's all wired up, push messages will be managed by MTA and everything will function normally and in line with the platform.

      Thanks,

      -Jon

        Terms & ConditionsAccessibility statement

        Loading footer...