Attaching files fails if file contains HEX character 1A (on Apache, CGI only)

[Ian Leader <__ian.leader__@…>]

Windows XP SP2, Python 2.3.4, Apache 2.0.54, cgi-bin, sqlite 3.2.1

On above environment, as of [1739], adding an attachment to the wiki or a ticket fails unless trac 'understands' the file type. For example: .diff, .txt and .py files all work fine. .jpg, .doc do not.

The error message is as follows:

Invalid Attachment

Attachment wiki//SecondAttachmentTestPage does not exist. 

The log is as follows:

10:35:28 Trac[main] ERROR: Attachment wiki//SecondAttachmentTestPage does not exist.
10:35:28 Trac[main] ERROR: Traceback (most recent call last):
  File "c:\python23\Lib\site-packages\trac\web\cgi_frontend.py", line 103, in run
    dispatch_request(os.getenv('PATH_INFO', ''), req, env)
  File "c:\python23\Lib\site-packages\trac\web\main.py", line 421, in dispatch_request
    dispatcher.dispatch(req)
  File "c:\python23\Lib\site-packages\trac\web\main.py", line 283, in dispatch
    resp = chosen_handler.process_request(req)
  File "c:\python23\Lib\site-packages\trac\attachment.py", line 234, in process_request
    attachment = Attachment(self.env, parent_type, parent_id, filename)
  File "c:\python23\Lib\site-packages\trac\attachment.py", line 45, in __init__
    self._fetch(filename, db)
  File "c:\python23\Lib\site-packages\trac\attachment.py", line 67, in _fetch
    'Invalid Attachment')
TracError: Attachment wiki//SecondAttachmentTestPage does not exist.

[vittorio]

i tried to reproduce, but it does work using:
Debian Sarge, Python 2.3.5, Apache 2.0.54, mod_python, sqlite 2.8.16

[Ian Leader <__ian.leader__@…>]

Retested on revision 1744. My initial analysis of the problem was incorrect. The problem appears to be the presence of the hex character 1A in the file being uploaded.

I have attached the single-byte file I used to reproduce this.

[Ian Leader <__ian.leader__@…>]

File containing single hex character 1A to reproduce bug

[Christian Boos]

It also works for me, I can attach your att_test_FAIL file (tracd [1744] on Windows, python 2.3.2)…

[Ian Leader <__ian.leader__@…>]

It appears that this is an Apache CGI problem.

I retested on [1744] with tracd, and as for cboos the bug was not present. It is still there in Apache 2.0.54.

I will try to set up mod_python on Windows and see if it is isolated to CGI. If anyone has a non-Windows CGI setup it would be useful to know if this is working.

[Ian Leader <__ian.leader__@…>]

Tested [1744] on mod_python 3.1.3 with Win32 mod_python util.py patch from ticket #554, Apache 2.0.54, WXP SP2, Python 2.3.4 - everything worked fine.

This therefore appears to only be a problem with CGI.

[vittorio]

I tested on Debian Sarge, r1751, Python 2.3.5, Apache 2.0.54, sqlite 2.8.16 using mod_python and CGI. Both did work.

This therefore appears to only be a problem with windows.

(what is strange and another issue: theres a HTML-preview for this file. binary)

[Rede]

I found out following mail ​http://mail.mems-exchange.org/pipermail/quixote-users/2004-July/003190.html

Which basically states:

I was having problems with the file upload demo on Windows with Apache/CGI. Turns out that to safely upload binary files, you have to get sys.stdin to be in binary mode, else it quits reading once it hits the first null byte.

I'm not sure is this real reason, but it might be…

[Rede]

Well, I found even solution..

Change firstline or trac.cgi from #!C:\python23\python.exe (or whatever you have there) to #!C:\python23\python.exe -u to use stdin as in binary mode.

I also added this information on TracOnWindows#Fixafewthings

[Ian Leader <__ian.leader__@…>]

Found some mailings on similar problems as well e.g.

​http://aspn.activestate.com/ASPN/Mail/Message/python-list/1576002

>  > One last thing you might try if your web server honors the shebang
>  > convention (as Xitami and Apache both do, for example, even on Windows)
is
>  > to use a first line similar to
>  >
>  > #!C:/python22/python -u
>  >
>  > This will cause Python to run unbuffered, and should obviate the need to
>  > explicitly use the mscvrt module.
> 
>  Well it's not the buffering but the text translation that's the problem,
>  but I see that -u controls both of these.  This looks like a much cleaner
>  solution.

they all recomend that you run python with the -u flag in the first line of trac.cgi to avoid this.

I've tested this, and as a workaround it works.

However, I don't think this was a problem in previous versions, so presumably there is a better fix to be done in the code.

[Ian Leader <__ian.leader__@…>]

Have reopened, as I think this may be better fixed in code.

[Rede]

Can also be fixed with following (didn't do patch file for such a small change):

Index: cgi_frontend.py
===================================================================
--- cgi_frontend.py	(revision 1753)
+++ cgi_frontend.py	(working copy)
@@ -96,8 +96,8 @@
 def run():
     locale.setlocale(locale.LC_ALL, '')
 
+    env = open_environment()
     req = CGIRequest()
-    env = open_environment()
 
     try:
         dispatch_request(os.getenv('PATH_INFO', ''), req, env)

[anonymous]

…Better leave this open yet… :)

[Ian Leader <__ian.leader__@…>]

Yes - would be great if the above patch could be applied.

[Christopher Lenz]

Should be fixed in [1754]. Thanks!