You need to search for Category Parameters (specific occurrences of a custom field). This isn't a top level object and cannot be searched directly, so you have to search the Parameter object and use an EXISTS filter that specifies the specific occurrences where the parameter exists on a certain custom form and section. EXISTS only work in the unsupported API.
https://<domain>.my.workfront.com/attask/api/unsupported/param/search?EXISTS:a:$$EXISTSMOD=EXISTS&EXISTS:a:$$OBJCODE=CTGYPA&EXISTS:a:parameterID=FIELD:ID&EXISTS:a:categoryID=<categoryID>&EXISTS:a:categoryID_Mod=eq&EXISTS:a:parameterGroupID=<sectionID>&EXISTS:a:parameterGroupID_Mod=eq
If you like my content, please take a moment to view and vote on my Idea Requests: https://tinyurl.com/4rbpr7hf