Hi,
Steps:
1. Applying the deny permission to the workflow model path "/var/workflow/models/<custom-workflow-folder-name>/<custom-workflow-name>" or "/var/workflow/models/<custom-workflow-name>" depending on where you have created the workflow for a particular group.
2. Assign the group created to those set of user whom you which to restrict.
Now if you log in from a specific user, you will not be getting the option for selecting this model.
Second, even if there is any customization that the workflow can be triggered by the click of the button on the AEM UI, then you will be getting the error in the backend, that the workflow can not be started due to insufficient permissions.
Thanks