W3C home > Mailing lists > Public > www-lib@w3.org > July to September 2003

Re: FTP problem: Get dir listing when expecting 404 Not Found

From: Richard Atterer <richard@list03.atterer.net>
Date: Tue, 12 Aug 2003 14:05:30 +0200
To: Renate Bahnemann <renate@micromuse.com>
Cc: www-lib@w3.org, jose.kahan@w3.org
Message-ID: <20030812120529.GA1710@nenya.lan>

Hi Renate,

On Mon, Aug 11, 2003 at 12:08:43PM +0100, Renate Bahnemann wrote:
> Problem occurs for the following:
> 
> ftp://<host>:<port>/<top-dir>/<invalid-dir>/<filename>
> 	returns an empty directory listing,
> 	I expected "404 Not Found" error.

I also noticed the same problem, but it hadn't yet hurt bad enough for me
to fix it. :)

I've tried your patch and can confirm that it works and appears to be 
correct - many thanks!

FYI, there are a few other problems with HTFTP:
- Doesn't support resuming downloads from a certain offset
  [me, 13 Feb, patch]
- Breaks with certain ftpd, e.g. OpenBSD ftpd [me, 13 Feb, patch]
- Incorrect timeout triggers if server is a bit slow to open the data 
  connection [fix discussed by Timothee Besset, 6 May]

HTTP:
- Various problems with POST requests, especially when 100 Continue'ing
  [discussed nicely /somewhere/ - uh, need to track that message down]
- Some problems with pipelining [me, 19 Feb, patch / me, 3 Mar, fixed in CVS]
- Some issues with HTTP cookies

Other:
- Various mem leaks [Tim Serong, 26 Mar; Torbjörn Carlsson, 13 May]
- Win32 minor fix [Calum, 21 May]
...and lots more. :-/

Jose, I have a very bad conscience: I was given CVS write access months ago 
and still haven't gotten around to committing things. Sorry about that - I 
still use libwww regularly, so I'm bound to become more active again sooner 
or later.

> I am wondering whether the current behaviour of the library conforms to
> paragraph 3.2.2 of RFC 1738.

RFC 1738 describes how to turn an FTP URL into FTP commands. If you have a 
look, this RFC was actually written in *1994* by the same Tim Berners-Lee 
who wrote the libwww FTP code in *1991*. :-) I agree that the code probably 
doesn't comply to the RFC.

OTOH, e.g. doing a direct "RETR /foo/bar/file" can be a lot faster than
CWDing to the foo/bar directory, especially with high-latency
connections...

> Is my 'fix' valid or am I misunderstanding something and making a mistake
> in my application?

I like your patch! :)

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer     |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯

For posterity, without the patch I get the following log:

FTP Tx...... CWD asdf
Write Socket 10 bytes written to 5
download:       Alert 16 for ftp://localhost/asdf/foo obj 0x8231ae8
FTP Get Data now in state NEED_CWD
Read Socket. WOULD BLOCK fd 5
Host Event.. READ passed to `ftp://localhost/asdf/foo'
FTP Event... now in state FTP_NEED_DATA
FTP Get Data now in state NEED_CWD
Read Socket. 38 bytes read from socket 5
Host........ passing 38 bytes as consumed to 0x8239ee0
Host........ 0 bytes remaining 
FTP Rx...... `550 asdf: No such file or directory.'
Read Socket. Target returns 200
FTP Get Data now in state SUB_ERROR
Error....... Add  22    Severity: 1     Parameter: `Unspecified'        
Where: `HTFTPGetData'
FTP Event... now in state FTP_NEED_SERVER
FTP Server.. now in state NEED_SYST
FTP Tx...... SYST
Write Socket 6 bytes written to 5
download:       Alert 16 for ftp://localhost/asdf/foo obj 0x8231ae8
FTP Server.. now in state NEED_SYST
Read Socket. WOULD BLOCK fd 5
Host Event.. READ passed to `ftp://localhost/asdf/foo'
FTP Event... now in state FTP_NEED_SERVER
FTP Server.. now in state NEED_SYST
Read Socket. 19 bytes read from socket 5
Host........ passing 19 bytes as consumed to 0x8239ee0
Host........ 0 bytes remaining 
FTP Rx...... `215 UNIX Type: L8'
Read Socket. Target returns 200
FTP Server.. now in state CHECK_SYST
FTP Server.. now in state SUB_SUCCESS
FTP Server.. Guessed type 4
FTP Event... now in state FTP_NEED_DATA
FTP Get Data now in state NEED_SELECT
FTP Get Data now in state NEED_REST
FTP Get Data now in state NEED_ACTION
FTP Tx...... LIST foo
Write Socket 10 bytes written to 5
download:       Alert 16 for ftp://localhost/asdf/foo obj 0x8231ae8
FTP Get Data now in state NEED_ACTION
Read Socket. WOULD BLOCK fd 5
Host Event.. READ passed to `ftp://localhost/asdf/foo'
FTP Event... now in state FTP_NEED_DATA
FTP Get Data now in state NEED_ACTION
Read Socket. 56 bytes read from socket 5
Host........ passing 56 bytes as consumed to 0x8239ee0
Host........ 0 bytes remaining 
FTP Rx...... `150 Opening BINARY mode data connection for '/bin/ls'.'
Read Socket. Target returns 200
FTP Get Data now in state NEED_ACCEPT
Accepted.... socket 7
Host connect Unlocking Host 0x81b62d0
FTP Get Data Passive data socket 7
FTP Get Data now in state NEED_STREAM
StreamStack. Constructing stream stack for text/html to */*
StreamStack. Source output
Error....... Add   2    Severity: 8     Parameter: `Unspecified'        
Where: `HTDir_new'
HTDir_new... base is `foo/'
[...]

With the patch:

FTP Tx...... CWD asdf
Write Socket 10 bytes written to 5
download:       Alert 16 for ftp://localhost/asdf/foo obj 0x8231790
FTP Get Data now in state NEED_CWD
Read Socket. 38 bytes read from socket 5
Host........ passing 38 bytes as consumed to 0x82380b0
Host........ 0 bytes remaining 
FTP Rx...... `550 asdf: No such file or directory.'
Read Socket. Target returns 200
FTP Get Data now in state SUB_ERROR
Error....... Add  22    Severity: 1     Parameter: `Unspecified'        
Where: `HTFTPGetData'
FTP Event... now in state FTP_ERROR
Channel..... Delete input stream 0x824f4d8 from channel 0x81c1020
Channel..... Delete input stream 0x824f4d8 from channel 0x81c1020
Net Object.. Delete 0x81cdbe8 and call AFTER filters
Host info... Remove 0x81cdbe8 from pipe
Host Object. closing socket 6
Received on Tuesday, 12 August 2003 08:46:32 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 23 April 2007 18:18:43 GMT