Large wiki pages get truncated when submitting

[tracticket@…]

• How To Reproduce:
  • Edit a wiki page and paste in a good deal of text e.g. 70,000 characters.
  • Submit the change.
  • I got no errors, but the page was truncated.
  • More info:
    • When I saved the text that was left after the truncation to a file, the file size was 64K. Maybe 64K is the limit?
    • Looking in the database, examining the right item in the wiki table also shows it truncated.
    • I have not changed the max_size setting in trac.ini so it is still the default: 262144 (much bigger than my truncation issue).
    • I tried with the latest Firefox and Chrome browsers and got the same thing.
    • This is the Apache access logs, showing a POST that was truncted:

      10.245.191.200 - - [30/Jan/2013:11:58:57 +0000] "POST /wiki/tomtest HTTP/1.1" 303 500 "http://trac.hourlie.com/wiki/tomtest?action=edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Ubuntu Chromium/23.0.1271.97 Chrome/23.0.1271.97 Safari/537.11"
      10.245.191.200 - - [30/Jan/2013:11:58:58 +0000] "GET /wiki/tomtest HTTP/1.1" 200 27138 "http://trac.hourlie.com/wiki/tomtest?action=edit" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Ubuntu Chromium/23.0.1271.97 Chrome/23.0.1271.97 Safari/537.11"
      

    • Note the 303 http status. Is that a problem?
  • I set log_level = DEBUG and captured the output. It doesn't indicate any problem:

    Jan 30 12:06:52 jarvis Trac[main] DEBUG: Dispatching <RequestWithSession "POST '/wiki/tomtest'">
    Jan 30 12:06:53 jarvis Trac[web_ui] DEBUG: Updating session c0f2ce7ba962b476cc0425d4dd8a8b76 for user tom
    Jan 30 12:06:53 jarvis Trac[session] DEBUG: Retrieving session for ID u'tom'
    Jan 30 12:06:53 jarvis Trac[web_ui] DEBUG: Updating session c0f2ce7ba962b476cc0425d4dd8a8b76 for user tom
    Jan 30 12:06:53 jarvis Trac[chrome] DEBUG: Prepare chrome data for request
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing ACCTMGR_CONFIG_ADMIN on None
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing ACCTMGR_USER_ADMIN on None
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing TRAC_ADMIN on None
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing PERMISSION_GRANT on None
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing PERMISSION_REVOKE on None
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing TICKET_ADMIN on None
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing BLOG_ADMIN on <Resource 'blog'>
    Jan 30 12:06:53 jarvis Trac[perm] DEBUG: No policy allowed tom performing WIKI_ADMIN on <Resource u'wiki:tomtest@1'>
    

• System Information: list the software components involved with their version
  • The Operating System: Ubuntu 12.04.1 LTS
  • The Python version used: Python 2.7.3
  • The database backend: mysql 5.5.24
  • Webserver: Apache.

[Christian Boos]

​http://dev.mysql.com/doc/refman/5.5/en/column-count-limit.html?

[anonymous]

Replying to cboos:

​http://dev.mysql.com/doc/refman/5.5/en/column-count-limit.html?

I don't think the MySQL ​Row-Size Limit is the issue because I use the InnoDB storage engine and the field type of wiki.text is TEXT.

[Jun Omae]

If your MySQL database is configured with utf8 encoding and the wiki text has supplementary characters (codepoint ≥ U+10000), the text will truncate. If you need these characters, it should be configured with utf8mb4 and utf8mb4_bin.

Please check the database configurations and the wiki text.

cf. wiki:MySqlDb#MySQL and ​http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html.

[anonymous]

Replying to jomae:

Thanks for your suggestion but my problem affects normal characters. In fact I did a test using only numeric characters (I'll attach the test file). This text just has a list of numbers from 1 to 15999, it is 83K large, and for me, trac truncates it at number 10949.

[anonymous]

Ok, trac would let me upload a attachment because it thinks it's spam, but anyway the file was created with:

<?php

for ($i=1;$i<16000;$i++) {
 echo "$i\n";
}

then

php junk_counting.php > wikiTruncateTest.txt

[Christian Boos]

Well, first thing, use a MySQL tool to try to insert directly such content in one such column. If this works (no truncation), we can eliminate a MySQL issue and start to look at the web server side.

[anonymous]

Replying to cboos:

Bingo. I tried to insert directly into mysql and found the truncation still happened.

I found out that mysql TEXT type can only hold - 65535 bytes.

I fixed the issue in two ways.

Firstly I added max_size = 65535 to trac.ini so at least the users get a warning if their page is too long instead of it being trucated:

Warning: The wiki page is too long (must be less than 65535 characters)

But also (for good measure) migrated the problematic field to a MEDIUMTEXT instead:

ALTER TABLE  `wiki` CHANGE  `text`  `text` MEDIUMTEXT CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL

Thanks so much for your help.

I'm happy if you want to close this ticket. I won't close it myself in case you feel you want to start using MEDIUMTEXT in your MySQL schema. I have since found a similar suggestion for the ticket description field: #8396.

[Christian Boos]

Replying to anonymous:

Thanks so much for your help.

You're welcome.

I'm happy if you want to close this ticket. I won't close it myself in case you feel you want to start using MEDIUMTEXT in your MySQL schema. I have since found a similar suggestion for the ticket description field: #8396.

The present issue is indeed a shortcoming we'd like to fix in some future version.

Closing as duplicate of #8396.