In my current custom workflow referenced items are not getting published. I want to have a functionality similar to OOTB publish workflow where it gives a popup in publish wizard and asks if we want to activate referenced items as well. Can you please help me as to how can I implement the same behaviour for custom workflow?
The popup is not part of the workflow itself, but part of the "activation" logic.
Workflows run async and are not directly connected to a browser window, thus they cannot render HTML and send it as response to a user.
If you have a custom workflow, you can either replicate the activation feature (most likely to can reuse the complete logic and UI then), start the reference search first and as a final step put all that information into your custom workflow. This allows a limited amount of interactive work, but it's more complicated than a simple workflow, which just runs in the background and has no feedback channel.
Or you just implement a workflow step (as usual) and use the ReferenceSearch class  to find all other objects in the repo which are referencing your page.
Thanks much Joerg for detailed explanation and solutions.
2nd solution would be appropriate for my custom workflow. However, for our requirement, we need to make use of com.day.cq.dam.commons.util.AssetReferenceSearch API tosearch the assets which are referenced in particular page we want to activate.