Caching secured content in dispatcher

I have a need to cache assets in dispatcher and at the same time impose authentication / authorization being done from publisher. I went through the documentation - https://docs.adobe.com/docs/en/dispatcher/permissions-cache.html and tried it. 

Code updates: 

Created a authentication servlet – DocumentsAuthcheckerServlet.java that is accessible as /bin/myappname/servlets/docpermissioncheck from dispatcher.  

Dispatcher config updates: 

Added the necessary rules as defined in the documentation link, for the document path in question.

Expected result:

Now the expectation is that when the end user requests documents under the specific folder defined in auth-checker module, 

1. The dispatcher should forward request to authentication servlet for validation
2. The servlet should respond as ALLOWED or DENIED status
3. Dispatcher should cache and deliver – if allowed
4. Dispatcher should cache but deny document – if response was deny from servlet  

Actual Observation: 

The steps 1 and 2 works as expected. But the dispatcher finally delivers the document regardless the response was 401 or 200 status. Logs below for detailed information. 

Dispatcher log:  

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Found farm publishfarm for calix-dev.adobecqms.net

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] checking [/content/dam/..../xxxxx.pdf]

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Creating new socket: x.x.x.x:4503

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Authorization checker: connected to backend 0 (x.x.x.x:4503)

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Authorization checker: backend answered with status code: 401

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] incomplete request, no socket reuse

[Sat Oct 03 14:00:07 2015] [I] [10230(140177125636064)] authorization checker denied access: /mnt/var/www/html/content/dam/......./xxxxxxx.pdf

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] cache-action for [/content/dam/....../XXXXXXXXX.pdf]: NONE

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Creating new socket: x.x.x.x:4503

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Connected to backend 0 (x.x.x.x:4503)

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: user-agent

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: accept-encoding

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: accept-language

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: accept

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: host

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: if-none-match

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: cookie

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: Via

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: X-Forwarded-For

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] Adding request header: Server-Agent

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] response.status = 200

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] response.headers[Date] = "Sat, 03 Oct 2015 18:00:07 GMT"

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] response.headers[Last-Modified] = "Sun, 30 Aug 2015 00:09:42 GMT"

[Sat Oct 03 14:00:07 2015] [D] [10230(140177125636064)] response.headers[Content-Type] = "application/pdf"

[Sat Oct 03 14:00:07 2015] [I] [10230(140177125636064)] "GET /content/dam/........./xxxxx.pdf" 200 445171 62ms

I am not sure where did the dispatcher configuration go wrong. But please suggest if similar situation is seen before. 

Additional information: 

The authentication check is not being done using javax.jcr.session's checkPermission() method. Instead, we have custom implementation that verifies the DAM - custom metadata attributes and ultimately sends response as SlingHttpServletResponse.SC_FORBIDDEN or SlingHttpServletResponse.SC_OK.