It seems not to be possible to display Document-specific custom forms on the Details screen of the document. Users have to navigate to the Custom Forms option on the left-hand nav bar and then select the form. Even then, it appears only on that section (Custom Forms). All other objects (can't promise I've checked every one) allow the display of Custom Forms on the object details screen and Documents should be the same.
