LCCS Recording causes 404 error with HTTP HEAD

[Former Community Member]

Hello everbody!

I'm trying to proceed with session recording, but I'm not receiving the file. I'm getting the following error instead:

access_log:

209.34.68.17 - - [28/Jul/2011:12:38:41 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5Ftime2011628124236510%2Ezip HTTP/1.1" 404 -

error_log:

[Thu Jul 28 12:38:41 2011] [error] [client 209.34.68.17] File does not exist: /mnt/disk2/lccs-recordings/na2-sdk-e72fda66-f06e-4702-b0b2-7c0c9aa51e35_x002F_audiocast_x002F_time2011628124236510.zip

Any suggestions?

Thanks,

Renato A. Ferreira

[Former Community Member]

OK, Once i've touched the file before stop recording the server sent a PUT request. It's appear for me like that old TFTP implementations. Is there any reason for it?

But to it finally work I needed to remove the basic authentication from my server configuration. The authetication using http://<username>:<password>@hostname/ url format does no worked. As we can see in the "-  -" after ip address in access_log, the system did not send the user on requests.

[nzezelj]

Hi Renato,

Thanks for using the recording feature!  I am not sure I really follow what happened but I am glad you resolved your issues.  404 probably meant your recording didn't make it to the repository.  As far as authentication is concerned, it is only needed if you setup your WebDAV that way. 

Thanks,

Nikola

[Former Community Member]

nezezelj, I will try to explain:

1) The 404 problem happens because the LCCS recorder only upload the zip file to my repository if the file already existis on directory and it sends a HEAD request to check it before try a PUT request. Once I understood it and touched (that unix command, you know!) the file before start recording, then recorder started to try PUT the file (overwriting an existing one), but it's still weird. I saw this same behaviour last time with that old TFTP much time ago.

2) I'm sure that I properly configured the repository URL with authentication credencials, I double-checked it using getRepositoryInfo, but my apache was returning 401 error and registering in access_log requests without authentication (user name must appear in place of "- -"). I was using basic authentication in dav enabled directory, I tested it a lot with another webdav client and it worked fine. Once I disabled the basic authentication, everthing started to work and I received the zip file.

I see that these are not big issues, I just noted things that is not well documented, or even appear to be WRONG documented, in this feature. I have another question about shared models synchronization in recorded sessions, but I will create another topic for it.

Many thanks!

Renato A. Ferreira

[Former Community Member]

Hi Renato,

Strange - I just set up a WebDAV at Dreamhost, with authentication, and

used the usual http://username:password@ourdomain.dreamhost.com/webdav, and

it worked for me.

You're saying :

A) You need to have an existing file for it to work?

B) It never works with basic authentication?

Very odd - this completely worked for me.

nigel

[Former Community Member]

A) Yes!  look at these logs:

209.34.68.17 - - [29/Jul/2011:17:12:19 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 200 -
209.34.68.17 - - [29/Jul/2011:17:12:19 -0300] "PUT /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 204 -
209.34.68.17 - - [29/Jul/2011:18:13:20 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 200 -
209.34.68.17 - - [29/Jul/2011:18:13:20 -0300] "PUT /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 204 -

LCCS recorder always request HEAD first to check the file before the PUT request... If HEAD request returns 404 the upload fail as logged bellow:

209.34.68.17 - - [28/Jul/2011:12:20:30 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 404 -
209.34.68.17 - - [28/Jul/2011:12:38:41 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5Ftime2011628124236510%2Ezip HTTP/1.1" 404 -

B) The third parameter from log lines above is the remote user and all of them returned "-". I configured my URL to http://voiteldav:****@***.***.***.***/lccs-dav/, but I'm still receving authentication-less requests. Before to disable the authentication I was getting the following log line:

209.34.68.17 - - [28/Jul/2011:15:39:21 -0300] "PUT /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 401 401

With another webdav client, a BitKinex that I've installed only to test this configuration, running on my laptop everything worked fine and all requests were logged as follows:

192.168.0.130 - voiteldav [26/Jul/2011:14:02:38 -0300] "PROPFIND /lccs-dav/ HTTP/1.1" 207 681
192.168.0.130 - voiteldav [26/Jul/2011:14:02:38 -0300] "PUT /lccs-dav/Audiocasting.swf HTTP/1.1" 201 198
192.168.0.130 - voiteldav [26/Jul/2011:14:02:39 -0300] "PROPFIND /lccs-dav/ HTTP/1.1" 207 1175
192.168.0.130 - voiteldav [26/Jul/2011:14:42:56 -0300] "DELETE /lccs-dav/Audiocasting.swf HTTP/1.1" 204 -

And, finaly, I tcpdumped a recording process. First without authentication:

======= head to check the file =======

HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1
User-Agent: curl/7.12.1 (x86_64-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6
Host: ***.***.***.***
Pragma: no-cache
Accept: */*

======= server returns OK for file existence =======

HTTP/1.1 200 OK
Date: Mon, 01 Aug 2011 14:26:16 GMT
Server: Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8o DAV/2
Last-Modified: Mon, 01 Aug 2011 14:25:27 GMT
ETag: "7-4eb7-4a97267126352"
Accept-Ranges: bytes
Content-Length: 20151
Content-Type: application/zip

======= then PUT the file =======

PUT /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1
User-Agent: curl/7.12.1 (x86_64-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6
Host: ***.***.***.***
Pragma: no-cache
Accept: */*
Content-Length: 15644
Expect: 100-continue

======= sucess! =======

HTTP/1.1 204 No Content
Date: Mon, 01 Aug 2011 14:26:16 GMT
Server: Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8o DAV/2
Content-Length: 0
Content-Type: application/zip

======= no more requests  =======

Now with authentication enabled:

======= lccs try to put =======

PUT /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1
User-Agent: curl/7.12.1 (x86_64-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6
Host: ***.***.***.***
Pragma: no-cache
Accept: */*
Content-Length: 20922
Expect: 100-continue

======= my server ask for authorization  =======

HTTP/1.1 401 Authorization Required
Date: Mon, 01 Aug 2011 14:38:23 GMT
Server: Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8o DAV/2
WWW-Authenticate: Basic realm="DAV"
Content-Length: 401
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
are authorized to access the document
requested.  Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>

======= no more requests =======

As you could see, LCCS recorder just don't send the Authorization field on HTTP header.

Using bitkinex I get the following request:

PUT /lccs-dav/Audiocasting.pdf HTTP/1.1
Host: 172.16.80.90
User-Agent: BitKinex/3.2.3
Accept: */*
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 262961
Content-Type: application/octet-stream
Translate: f
Authorization: Basic *************************************

[nzezelj]

Hi Renato,

Thank you on a detailed report.  I am hopeful that it will help us narrow down the issue.

I got a better understanding of LCCS code over the weekend and I have a few clarifications to make:

1/ At first, LCCS does not know what type of authentication to use (plain, basic, digest etc.) with the PUT request.  The only information gathered from the client is the repository URL that may or may not contain credentials.

2/ Due to 1/, LCCS uses --anyauth curl paramater which sends a plain HEAD request (or plain PUT request with more recent curl versions) without credentials in order to let WebDAV tell it which authentication type to use (if one is required).  That is why you see "- -" with HEAD request; this is expected.  At this point, all the WebDAV servers I worked against would return 401 error with authentication details.  Then LCCS turns around and sends a proper PUT request with credentials and the desired authentication type.  What is puzzling about your WebDAV is that it returns 404 error and not 401 which also breaks our workflow for PUT. 

3/  I do know that some WebDAV servers prefer PROPFIND request instead of HEAD but at this point we do not support that in our implementation.  It is definitely something we want to add as soon as possible.

At this point, I would focus on a difference between 404 and 401 response to the HEAD request.  Let's see if we can figure out why that is and if your server can possibly return 401 instead.

Thanks,

Nikola

[Former Community Member]

nzezelj, the 404 problem and the 401 one are two distinct issues.

Even when the authentication is disabled, if the HEAD request returns 404, the LCCS recorder (curl) dos not PUT the file.

Maybe I did a mistake with authentication, because I used the sugested configuration for mod_dav from Apache docs. The doc suggests us to configure as following:

<Location /foo>

  Dav On

  AuthType Basic
  AuthName DAV
  AuthUserFile user.passwd

  <LimitExcept GET OPTIONS>

    require user admin
  </LimitExcept>
</Location>

It configures the server to does not ask for authentication in GET requests on /foo, and the HEAD is one kind of GET. Then, as you explained above, once HEAD did not ask for authentication, curl will not try to authenticate the PUT request. But it's stil weird for me, and it does not explain the 404 issue when the authentication is disable. I will try without the LimitExcept parameter.

[nzezelj]

Sure, I didn't mean to imply that 404 and 401 errors are the same issue.  Only that if we somehow "converted" a 404 into a 401, things had a chance of working more smoothly.

Back to the issue, now that you also enabled the authentication for GET (and HEAD) do things work as "advertised" with authentication enabled?  That is, a plain HEAD is responded to with 401, and a subsequent PUT comes with proper credentials and succeeds?

I need to look at the authentication disabled workflow again.

Thanks,

Nikola

[Former Community Member]

nzezelj, I confirmed that! Curl realy uses HEAD to determine if the authentication is needed. It goes ahead even when the file is not there, in other words, it stops on 404 but not on 401. Look at lines above:

209.34.68.17 - - [01/Aug/2011:15:24:11 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 401 -
209.34.68.17 - voiteldav [01/Aug/2011:15:24:11 -0300] "PUT /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 204 -

The remaining question now is only why curl does not go ahead when my server returns 404 for HEAD request. I retried again just right now without success:

209.34.68.17 - - [01/Aug/2011:15:36:24 -0300] "HEAD /lccs-dav/na2%2Dsdk%2De72fda66%2Df06e%2D4702%2Db0b2%2D7c0c9aa51e35%5Fx002F%5Faudiocast%5Fx002F%5FsomeID%2Ezip HTTP/1.1" 404 -
=== no more requests ===

I think that the reason is because 404, as well as 403, is an irreversible error. While the 401 says that that request is possible, buuut you need to provide the required credentials to go ahead. It makes any sense for GET requests, but not for the PUT ones that you use to create files on server.

Thanks for your attention, guys.

Renato A. Ferreira