Replace StringIO and cStringIO with io.StringIO and io.BytesIO

[Ryan J Ollos]

The ​io module is available since Python 2.6. We can therefore replace StringIO.StringIO and cStringIO.StringIO with io.StringIO and io.BytesIO.

io.StringIO requires a unicode string. io.BytesIO requires a bytes string. StringIO.StringIO allows either unicode or bytes string. cStringIO.StringIO requires a string that is encoded as a bytes string.

[Ryan J Ollos]

Based on what I've read, it would seem that the replacement cStringIO.StringIO → io.BytesIO can be made everywhere. We'll have to be more careful when replacing StringIO.StringIO though.

[Ryan J Ollos]

Proposed changes in log:rjollos.git:t12046_stringio_replacement. I'll do more extensive testing before committing early in milestone:1.3.1. Errors in the proposed changes may point to the need for additional test coverage as well, with the exception of cases that I modified the test case incorrectly. I'm least certain of the changes in the formatter module, and need to test changes to bugzilla2trac.py (unsure whether the attachment content will be utf-8 encoded or unicode). All tests pass on OSX for SQLite, MySQL and PostgreSQL.

[Jun Omae]

Two things:

1. Workflow macro with unicode string raises TypeError: 'unicode' does not have the buffer interface.
2. I think io.BytesIO('...') should be io.BytesIO(b'...')

[Ryan J Ollos]

I've addressed the comment:4 issues in log:rjollos.git:t12046_stringio_replacement.2.

[Jun Omae]

Unit and functional tests pass with the branch on Windows 10 (python 2.7, 32 bit).

However, the following changes in [28fc0d3b/rjollos.git], I think we shouldn't use unicode(wikidom).

@@ -1532,5 +1532,5 @@
         if isinstance(wikidom, basestring):
             wikidom = WikiParser(env).parse(wikidom)
-        self.wikidom = wikidom
+        self.wikidom = unicode(wikidom)
 
     def generate(self, escape_newlines=False):

The wikidom variable would be passed to text argument of Formatter.parse. The text argument accepts a basestring and a iterable instance. See trunk/trac/wiki/formatter.py@14969:1277-1278,1280#L1275​.

Instead, how about the following changes which convert to a unicode if it is a str instance in WikiParser.parse?

trac/wiki/formatter.py

@@ -1531 +1531 @@
         self.context = context
         if isinstance(wikidom, basestring):
             wikidom = WikiParser(env).parse(wikidom)
-        self.wikidom = unicode(wikidom)
+        self.wikidom = wikidom
 
     def generate(self, escape_newlines=False):
         """Generate HTML elements.
@@ -1558 +1558 @@
         self.context = context
         if isinstance(wikidom, basestring):
             wikidom = WikiParser(env).parse(wikidom)
-        self.wikidom = unicode(wikidom)
+        self.wikidom = wikidom
 
     def generate(self, shorten=False):
         """Generate HTML inline elements.

trac/wiki/parser.py

@@ -223 +223 @@
     def parse(self, wikitext):
         """Parse `wikitext` and produce a WikiDOM tree."""
         # obviously still some work to do here ;)
+        if isinstance(wikitext, str):
+            wikitext = wikitext.decode('utf-8')
         return wikitext
 
 

[Ryan J Ollos]

That looks good to me, but I don't know that area of the code very well. Is it possible that wikidom could be an iterable of strings that need to be decoded?

[Jun Omae]

Replying to Ryan J Ollos:

That looks good to me, but I don't know that area of the code very well. Is it possible that wikidom could be an iterable of strings that need to be decoded?

Sounds good.

trac/wiki/formatter.py

@@ -1278 +1278 @@
             text = text.splitlines()
 
         for line in text:
+            if isinstance(line, str):
+                line = line.decode('utf-8')
             # Detect start of code block (new block or embedded block)
             block_start_match = None
             if WikiParser.ENDBLOCK not in line:
@@ -1404 +1406 @@
         processor = None
         buf = io.StringIO()
         for line in text.strip().splitlines():
+            if isinstance(line, str):
+                line = line.decode('utf-8')
             if WikiParser.ENDBLOCK not in line and \
                    WikiParser._startblock_re.match(line):
                 in_code_block += 1
@@ -1531 +1535 @@
         self.context = context
         if isinstance(wikidom, basestring):
             wikidom = WikiParser(env).parse(wikidom)
-        self.wikidom = unicode(wikidom)
+        self.wikidom = wikidom
 
     def generate(self, escape_newlines=False):
         """Generate HTML elements.
@@ -1558 +1562 @@
         self.context = context
         if isinstance(wikidom, basestring):
             wikidom = WikiParser(env).parse(wikidom)
-        self.wikidom = unicode(wikidom)
+        self.wikidom = wikidom
 
     def generate(self, shorten=False):
         """Generate HTML inline elements.

[Ryan J Ollos]

Thanks, committed to trunk in r15010, r15011.