That error message in-and-of itself isn't helpful.
First validate the command you're using executes on command-line (removing AEM from the picture altogether)
Then, assuming you get the desired output from ImageMagick, apply that to the Command Line step in your Workflow.
Also FYI -- AEM 6.3 is fast approaching end-of-core-support within the next month (April 2020).