bugzilla2trac does not work with bugzilla 3.0

[dekimsey@…]

The bugzilla2trac.py script does nto seem to work with bugzilla 3.0. Using the default version number nor a version of '3' will function. However, it should be noted that using the default bugzilla version in the script, actually gets farther along in the import before failing.

Using Bugzilla v3 schema.
Bugzilla MySQL('*':'*':'*':'*******'): connecting...
Trac SQLite('/home/trac/projects/signin'): connecting...

Cleaning all tickets...
All tickets cleaned...

0. Filtering products...
Traceback (most recent call last):
  File "bugzilla2trac.py", line 894, in <module>
    main()
  File "bugzilla2trac.py", line 891, in main
    convert(BZ_DB, BZ_HOST, BZ_USER, BZ_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "bugzilla2trac.py", line 505, in convert
    mysql_cur.execute("SELECT product AS name FROM products")
  File "/usr/lib/python2.5/site-packages/MySQLdb/cursors.py", line 164, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.5/site-packages/MySQLdb/connections.py", line 35, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1054, "Unknown column 'product' in 'field list'")

Using Bugzilla v2180 schema.
Bugzilla MySQL('*':'*':'*':'*******'): connecting...
Trac SQLite('/home/trac/projects/signin'): connecting...

Cleaning all tickets...
All tickets cleaned...

0. Filtering products...
  Using products Signin

1. Import severities...
  inserting severity 'blocker' - '1'
  inserting severity 'critical' - '2'
  inserting severity 'major' - '3'
  inserting severity 'normal' - '4'
  inserting severity 'minor' - '5'
  inserting severity 'trivial' - '6'

2. Import components...
  inserting component 'General', owner 'bellaire'
  inserting component 'Scheduling, Preference Forms', owner 'bellaire'
  inserting component 'Sign-in, Timesheets', owner 'bellaire'

3. Import priorities...
  inserting priority 'highest' - '1'
  inserting priority 'high' - '2'
  inserting priority 'normal' - '3'
  inserting priority 'low' - '4'
  inserting priority 'lowest' - '5'

4. Import versions...
  inserting version '.Production'
  inserting version 'Development'
  inserting version 'Release'
  inserting version '2.0'
  inserting version 'unspecified'

5. Import milestones...

6. Retrieving bugs...

7. Import bugs and bug activity...
Traceback (most recent call last):
  File "bugzilla2trac.py", line 894, in <module>
    main()
  File "bugzilla2trac.py", line 891, in main
    convert(BZ_DB, BZ_HOST, BZ_USER, BZ_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "bugzilla2trac.py", line 681, in convert
    field_name = trac.getFieldName(mysql_cur, activity['fieldid']).lower()
  File "bugzilla2trac.py", line 431, in getFieldName
    (fieldid))
  File "/usr/lib/python2.5/site-packages/MySQLdb/cursors.py", line 164, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.5/site-packages/MySQLdb/connections.py", line 35, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1054, "Unknown column 'fieldid' in 'where clause'")

[dekimsey@…]

in bugzilla 3.0 the pkey of the fielddefs changed. this patch should fix that. additionally, removes a newline from an sql statement that was preventing compile.

[sid]

Can someone test this out and give feedback on the patch?

[dave@…]

Patch fails for me

#patch < patch
(Stripping trailing CRs from patch.)
patching file bugzilla2trac.py
Hunk #2 FAILED at 816.
1 out of 2 hunks FAILED -- saving rejects to file bugzilla2trac.py.rej
#cat bugzilla2trac.py.rej
***************
*** 812,819 ****
          ticketid = trac.addTicket(**ticket)

          if BZ_VERSION >= 2180:
-             mysql_cur.execute("SELECT attachments.*, attach_data.thedata FROM attachments, attach_data
-                     WHERE attachments.bug_id = %s AND attachments.attach_id = attach_data.id" % bugid)
          else:
              mysql_cur.execute("SELECT * FROM attachments WHERE bug_id = %s" % bugid)
          attachments = mysql_cur.fetchall()
--- 816,822 ----
          ticketid = trac.addTicket(**ticket)

          if BZ_VERSION >= 2180:
+             mysql_cur.execute("SELECT attachments.*, attach_data.thedata FROM attachments, attach_data WHERE attachments.bug_id = %s AND attachments.attach_id = attach_data.id" % bugid)
          else:
              mysql_cur.execute("SELECT * FROM attachments WHERE bug_id = %s" % bugid)
          attachments = mysql_cur.fetchall()

Running Version: 0.10.3-1etch2

[dave@…]

redownloaded original script and patch from here - can apply patch but get the following erro:

7. Import bugs and bug activity…

   Traceback (most recent call last):
     File "./bugzilla2trac.py", line 898, in ?
       main()
     File "./bugzilla2trac.py", line 895, in main
       convert(BZ_DB, BZ_HOST, BZ_USER, BZ_PASSWORD, TRAC_ENV, TRAC_CLEAN)
     File "./bugzilla2trac.py", line 685, in convert
       field_name = trac.getFieldName(mysql_cur, activity['fieldid']).lower()
     File "./bugzilla2trac.py", line 435, in getFieldName
       (fieldid))
     File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 163, in execute
       self.errorhandler(self, exc, value)
     File "/usr/lib/python2.4/site-packages/MySQLdb/connections.py", line 35, in defaulterrorhandler
       raise errorclass, errorvalue
   _mysql_exceptions.OperationalError: (1054, "Unknown column 'fieldid' in 'where clause'")
   MONITOR_SERVER#                                                                                   }}}
   

[Larry Silverman]

I'm running Bugzilla 3.0, also hitting this issue. I think you might need to remove the tostring() on the filedata = StringIO.StringIO line.

[maxence@…]

I've also run into problems when trying to migrate from Bugzilla 3.0 under Debian 5.0 (Lenny). Note that I'm running standard APT packages for both Trac (0.11.1-2.1) and Bugzilla (3.0.4.1-2+lenny1).

I'm attaching a modified version of bugzilla2trac.py that worked flawlessy for me, hoping that it will save time to other people.

Besides the 'fieldid' column modification, I had to rename a SQL statement that spelled the 'ticket' table 'Ticket', causing a MySQL query to fail.

With only minor modifications, bugzilla2trac.py could work out-of-the-box, which would make the migration experience much smoother.

[maxence@…]

Modified bugzilla2trac.py which works under Debian 5.0 (Lenny) with standard Trac and Bugzilla packages.

[anonymous]

Wow! It works for me with bugzilla 3.0.4, after changing BZ_VERSION to 2233

[NickLeverton]

I can confirm that the bugzilla2trac contained in the Trac 0.11.5 tarball fails on Bugzilla 3.0 (also installed from tarball), with the fieldid error dump given here.

I can also confirm that the script in the bugzilla2trac_debian_lenny.py works for my own non-Debian installation of Trac 0.11.5 and BZ3.0. BZ 3.0 was released some years ago now.

I think, therefore, that this fix needs to go into Trac's contrib scripts to make bugzilla2trac.py usable with modern Bugzillas.

[Michael Akinde]

The debian_lenny script fix unfortunately did not work for me (Bugzilla 3.4.2 + Ubuntu 8.04 standard setup from debian packages, i.e., Trac 0.10.4).

It required significant changes and workarounds (a good deal of them hardcoded, so I will not be posting it here) in the script to get it to work, and I had to forget about moving attachments. Not having a working migrating script is a major pain, so IMO this really ought to be prioritized a bit higher.

[Jeff]

The 'debian_lenny' script fix had 2 problems for my setup (Bugzilla 3.4.4 + Fedora 12 gnome, Trac 0.11.6 using PostgreSQL).

1. Has an error after importing the first ticket into PostgreSQL. My workaround was to settle for using the default SQLite database, and it would work after fixing the other problem below. Error msg:

   ...
   6. Retrieving bugs...
   
   7. Import bugs and bug activity...
     inserting ticket 1 -- the database has no bugs!
   Traceback (most recent call last):
     File "bugzilla2trac.py", line 907, in <module>
       main()
     File "bugzilla2trac.py", line 904, in main
       convert(BZ_DB, BZ_HOST, BZ_USER, BZ_PASSWORD, TRAC_ENV, TRAC_CLEAN)
     File "bugzilla2trac.py", line 820, in convert
       ticketid = trac.addTicket(**ticket)
     File "bugzilla2trac.py", line 349, in addTicket
       return self.db().get_last_id(c, 'ticket')
     File "/usr/lib/python2.6/site-packages/trac/db/postgres_backend.py", line 220, in get_last_id
       cursor.execute("SELECT CURRVAL('%s_%s_seq')" % (table, column))
     File "/usr/lib/python2.6/site-packages/trac/db/util.py", line 65, in execute
       return self.cursor.execute(sql)
     File "/usr/lib/python2.6/site-packages/trac/db/util.py", line 65, in execute
       return self.cursor.execute(sql)
   psycopg2.OperationalError: currval of sequence "ticket_id_seq" is not yet defined in this session
   

2. The script would fail when an attachment was deleted in Bugzilla. I'll attach my minor fix that worked for me. Hopefully it works well enough for others. This was the error I was getting without my minor modification to skip adding attachments:

   6. Retrieving bugs...
   
   7. Import bugs and bug activity...
     inserting ticket 1 -- ...
     ...
     inserting ticket 38 -- Provider Tab: Deleting acct causes probs
       ->inserting attachment '' for ticket 38 -- video showing steps.
   Traceback (most recent call last):
     File "bugzilla2trac.py", line 906, in <module>
       main()
     File "bugzilla2trac.py", line 903, in main
       convert(BZ_DB, BZ_HOST, BZ_USER, BZ_PASSWORD, TRAC_ENV, TRAC_CLEAN)
     File "bugzilla2trac.py", line 829, in convert
       trac.addAttachment(author, a)
     File "bugzilla2trac.py", line 407, in addAttachment
       attachment.insert(filename, filedata, filesize, datetime2epoch(time))
     File "/usr/lib/python2.6/site-packages/trac/attachment.py", line 222, in insert
       filename))
     File "/usr/lib/python2.6/site-packages/trac/util/__init__.py", line 227, in create_unique_file
       return path, os.fdopen(os.open(path, flags, 0666), 'w')
   OSError: [Errno 21] Is a directory: '/var/trac/test1/attachments/ticket/38/'
   

Please make this process less painful… This script is almost there for converting from Bugzilla v3. It's makes getting away from Bugzilla a much harder sell if this script doesn't work easily. Thanks!

[Jeff]

Minor modification to the 'debian_lenny' script to skip adding attachments when they were deleted from Bugzilla.

[Christian Boos]

That change can be integrated, yes, but before that please put your contact information as the last author in the script.

Also, IIUC, your script still won't work with PostgreSQL but only with SQLite as a backend, right? If that's the case, please add a comment at the beginning of the script making this clear (it only talks about MySQL for now).

In the long run, it would be nice to import from ​exported Mozilla data instead of looking directly in their database… (see also #8383).

[Jeff]

A co-worker came up with the fix to my problem 1.) above for my setup (Bugzilla 3.4.4 + Fedora 12 gnome, Trac 0.11.6 using Postgresql-8.4.2-1.fc12.x86_64). He used the current head revision of the bugzilla2trac.py script, but it was still missing a critical fix in comment 12 of #2061 that is necessary for import into PostgreSQL. This fix, skipping over deleted attachments, and some others from other Trac tickets, are in the attached version of the script (bugzilla2trac-postgresql.py). I tested this and it worked for my setup on both PostgreSQL and SQLite. I didn't test this script with MySQL.

[Jeff]

Works with Trac 11.6 using PostGreSQL 8.4 and Bugzilla 3.04.4

[Christian Boos]

Replying to Jeff:

This fix, skipping over deleted attachments, and some others from other Trac tickets, are in the attached version of the script (bugzilla2trac-postgresql.py). I tested this and it worked for my setup on both PostgreSQL and SQLite. I didn't test this script with MySQL.

Great! Thanks for contributing this new version. Committed as [9168].