We encountered a similar issue, even though we weren't using SDI. When we tried to access a URL like "/content/aem-demo/xyz.html," it correctly redirected to our custom 404 page as expected. However, when we tried to access the same URL without the ".html" extension, it unexpectedly redirected to a 403 (Forbidden) page without the header and footer.
You can find a detailed explanation of why we experienced the 403 error in the accepted answer on StackOverflow. https://stackoverflow.com/questions/48166989/why-aem-returns-403-for-requests-without-extensions
To resolve this issue, we implemented a rewrite rule that added the ".html" extension to the URL in such cases. After applying the rewrite rule, when we accessed a URL without the ".html" extension, it now redirected to .html page and displayed the expected 404 page for a not-found page, as desired.
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)/$ $1.html [R=301,L] Redirect URLs that end with a trailing slash to URLs without the trailing slash and append ".html".