We finally figured out the issue so thought I will share in case others face the same.
We use a client side rendering framework for components. So from AEM, a data-model with update request is passed to the front-end to render the component on page load or component update.
In the affected components, we had a CQ listener of REFRESH_PARENT set on edit.
When we were targeting a component in an experience, 2 calls were being made by AEM. One for experience related update and another one (caused by REFRESH_PARENT) for default. This was causing AEM to believe that we were trying to update the default view in an experience and hence all component edit toolbar options were getting removed.
When we changed it to REFRESH_SELF, no default call was made, and or toolbar rendered fine.