There is no configuration OOTB to show/hide different modes.
You can do it though using javascript.
1. Create a servlet to check current user group and return false if user is not part of layout group, otherwise true.
2. Create a clientlibs 'cq.authoring.dialog' with javscript, in javscript call above servlet and based on response hide layout option
Example :
$( document ).ready(function() {
$.get("demo_servlet.json", function(data, status){
//console.log("Data: " + data + "\nStatus: " + status);
if(data === 'false'){
$('#selectlayer-popover .editor-GlobalBar-layerSwitcherPopoverContent .coral3-ButtonList-item[data-layer="Layouting"]').hide();
}
});
});
Arun Patidar

