For my project, I would like that *only* the custom form widgets we have created (in the apps folder) are displayed to authors. I don't want authors adding the OOTB textbox/dropdown etc by accident as we have customized these.
I have a custom template and a custom etc/design - I thought I could do this like I would with a aem site page, by removing "group:Adaptive" from the components array on the design, but this does not remove the options from the author.
Additions to my custom design are added as expected (suggesting that the design is indeed being used). As a test, I even remove "group:Adaptive from all locations where it appears out of the box (3 locations) using the below search:
SELECT * FROM [nt:unstructured] AS node
WHERE ISDESCENDANTNODE(node, "/")
AND CONTAINS([components], "group:Adaptive Form")
Even after removing these 3 references, the "Adaptive Form" group is show when authoring the form.
Where is this defined?
How can I prevent this default behaviour from occurring? I don't want to remove/change the group of the OOTB components
OOTB Groups for AEM Forms can help with security in terms of who is permitted to access certain authoring capabilities:
You can control access by adding users, or restricting users based on these groups. Please refer to the hardening and security guide  for more details with respect to AEM Forms. If there is something missing, or incorrect, please let us know.
You may be able to customize your groups via the AEM security user admin page, i.e. /useradmin. See screenshot for the default permissions of the forms-users group as an example.
I found an alternative. If you take a copy of /libs/fd/af/layouts/gridFluidLayout and make the forms reference the copy.
You can remove the "updatecomponentlist" listener from the cq:editconfig. This prevents the OOTB components being added to the dialog.
If you are restricting the user from accessing any property under Apps you might actually be restricting the same user to perform some generic operation like creating the template, client libs as well. I would suggest that you create a group and a dummy user and start with more restrictive ACL over the apps. You may take reference from the OOTB groups shared by Kevin earlier but you might have to change the ACL post that even.