Disable drag and drop components for Layout Container

We have a column control component which includes a responsive grid - wcm/foundation/components/responsivegrid. The hierarchy goes like :

- Column Control

     - Layout Container

          - Parsys

     - Layout Container

          - Parsys

The layout container surrounding the parsys doesn't allow us to add components from the toolbar. (Correct implementation)

But it allows us to drag and drop components from the Toggle Side Panel which should ideally not be the case. The component gets added between column 1 and column 2. On page refresh, we are not able to see the component which gives a bad authoring experience.

So is there a way we can restrict the drag drop functionality to the layout container?

Thanks,

Bhavika