I was working around the following workflow. Once getting in the Author Admin Approval OR split reject, it's not taking the next process step. Similarly in the approve branch, steps after the approve/reject participants of inner OR split are also behaving the same way.
The logic of OR split works usual way as described by @Arun_Patidar. And, as I see your workflow, for approve/reject - you've used participant step. So, it seems like once the operation is done by "Author Admin Approval" - it's going to either Approve or Reject based on the selection and next step would be to go to Approver or Rejector inbox to proceed to next steps of workflow. But, if you're would like let's say. "Directly receive email" after rejection then in that case, you need don't need to have "Reject" step in workflow. Instead of that, directly add "Admin Reject Email to Workflow Initiator" under OR split and terminate the workflow programatically.
The same goes for Inner OR split as well.
Please check for scripted decision
If you are using manual/user based decision then user has to select next path from inbox/notification console.
The decision path is manual or automated.
If you choose user to take decision then you have to go to inbox and choose the next path.
To automate your steps based on approval or rejection using OR Split, You have to make sure your handler advance checkbox is checked for each OOTB/Custom process steps.
and one more thing... I don't think the auto advancer step is required after the participant step.