I was working around the following workflow. Once getting in the Author Admin Approval OR split reject, it's not taking the next process step. Similarly in the approve branch, steps after the approve/reject participants of inner OR split are also behaving the same way.
The logic of OR split works usual way as described by @Arun_Patidar. And, as I see your workflow, for approve/reject - you've used participant step. So, it seems like once the operation is done by "Author Admin Approval" - it's going to either Approve or Reject based on the selection and next step would be to go to Approver or Rejector inbox to proceed to next steps of workflow. But, if you're would like let's say. "Directly receive email" after rejection then in that case, you need don't need to have "Reject" step in workflow. Instead of that, directly add "Admin Reject Email to Workflow Initiator" under OR split and terminate the workflow programatically. The same goes for Inner OR split as well.