Curl command timeout while creating the bigger package

[tnik]

Hi ,

 I am using curl command to build the package (eg 250 MB) , but it times-out when the package is big and throws 503 error , but the package keeps building in the background and finally i can see a proper build package in the package manager ,

Can we increase the timeout of curl command and also is it possible to print building logs in terminal  as we see in the package manager 

I have tried to increase the curl timeout using -m flag , but its not working

Any workaround for the same? 

Thanks,

[pulkitvashisth]

Hi @tnik 
Can you try the ootb curl params for your issue :
Like https://everything.curl.dev/usingcurl/timeouts.html

You can use below params with your cUrl request and then check.
<cUrl command to build> --max-time 600 --retry 5

600 = 600 seconds
Let me know if this works.

[tnik]

hi @pulkitvashisth , 

i tried this flag it does not hold the connection for 600 seconds , it fails after every 60 seconds and go to next try till the retry count exhaust 

Thanks

[tnik]

the retry just keep on increasing the build count set by the package manager API

[aanchal-sikka]

@tnik 

1. Use a request to create and initiate your task
2. And then poll the system until this job is completed, then return the result.

This is an asynchronous pattern, and it’s much more scalable when it comes to the amount of processing you can do in there.

Of course you cannot use a single curl command anymore, but now you need to write a program to execute this logic

https://cqdump.joerghoh.de/2023/11/19/if-you-have-curl-every-problem-looks-like-a-request/

[tnik]

Hi @aanchal-sikka ,

have you tried this in the past , actually my requirement is to create the multiple package of site pages which would be roughly less 300 MB and move them from one cloud instances to another across the org  via automation , be it CI/CD scripts or programmatically at AEM level and then install them.

but not sure if this kind of content migration thing between two cloud instances in different org using automated package approach  can be done via workflows or sling jobs

[aanchal-sikka · Accepted solution]

@tnik 

I have worked on Curl commands to copy content from one Cloud to another. But, I had assured that filters are used wisely to keep the package sizes small.

I had used Curl for complete process:

- Create & Build

- Download 

- Upload & Install

So, I see 2 choices

- Create short, but multiple packages

- Check for package status. Before moving to next command.

-- Please check if preview command helps. I haven't tried it

  https://experienceleague.adobe.com/en/docs/experience-manager-65/content/sites/administering/operati... 

-- I guess download fails until build succeeds. Keep a short delay before you download. And if it fails, try download after some time.

I had written the script more than an year back. So, unable to recollect all the details.