You can write custom process step for the rejection and keep it after Reject Participant step something like below:

Here is the sample code you can write in process step and configure it.
Please note that you need to choose appropriate route based on your workflow model structure.
List<Route> backRoutes = session.getBackRoutes(item);
Route rte = backRoutes.get(0);
session.complete(item, rte);
Note: Please make sure to disable Handler advance checkbox for the process step.

Hope this helps.