`WikiPage` doesn't need the `ipnr` attribute

[Martin Scharrer <martin@…>]

Just saw this while changing ​th:WatchlistPlugin from own SQL code to the trac.wiki.model.WikiPage class:

Instances of the WikiPage class do not provide the ipnr as attribute as for all other wiki page values (version, author, etc.). In fact the _fetch method doesn't read this value from the DB.

If this is not by intention this should be fixed for consistency. Some plug-ins probably like to read this information out, e.g. I would have given the users the option to show it in their watchlist.

I would be willing to provide a patch.

[Christian Boos]

Well, why do you need the ipnr? I always thought nobody was actually interested in that and thought that we should even remove it in later releases.

[Remy Blank]

We removed the display of that information some time ago in 0.11-stable IIRC.

[Martin Scharrer <martin@…>]

Replying to cboos:

Well, why do you need the ipnr? I always thought nobody was actually interested in that and thought that we should even remove it in later releases.

As I said, I'm writing a plugin which shows the wiki data to the user as a table and I just wanted to be as comprehensive as possible. If you think about this that way I will just drop it. It's not an important thing for the plugin.

[Christian Boos]

All clear then. Let's remember to remove the ipnr information from the wiki table, at some later point.

[Thijs Triemstra]

Replying to cboos:

All clear then. Let's remember to remove the ipnr information from the wiki table, at some later point.

What version do you have in mind?

[Christian Boos]

0.14 of course. I don't think we should do any more schema change at this point for 0.13, as we should slowly try to finalize this version.

[Ryan J Ollos]

I was thinking we should add some notices to the documentation about the pending removal: log:rjollos.git:t9612-schedule-ipnr-removal.

[Ryan J Ollos]

Changes from comment:8 committed to 1.0-stable in [13512], merged to trunk in [13513].

[Ryan J Ollos]

Should we also remove ipnr from the Attachment class?

[Jun Omae]

Should we also remove ipnr from the Attachment class?

Yeah. We should consistently remove attachment.ipnr if wiki.ipnr will be removed.

[Ryan J Ollos]

Thanks. I'll add notices to attachment module about deprecation of ipnr and pending removal, as in comment:8. Changing milestone since work will be done in milestone:1.3.1.

[Jun Omae]

Related question, should we also remove [trac] show_ip_addresses option in 1.3.1?

[Ryan J Ollos]

Yeah, it looks like that should be removed. The only uses I see aside from adding the value to the Chrome data dictionary are:

• tags/trac-1.0.3/trac/templates/diff_view.html@:65#L65​
• tags/trac-1.0.3/trac/templates/history_view.html@:58#L58​

Side note: I didn't realize it was possible to link to a specific option and highlight it. I guess that works with any wiki word or phrase though. Very cool!

[Ryan J Ollos]

Deprecated ipnr attribute of attachment class in [13729:13730]. Deprecated show_ip_addresses in [13731:13732].

[carstenklein@…]

What about a hacked trac and where people upload unwanted attachments or alter wiki pages in ways we cannot imagine? Wouldn't it be nice to at least have the ip address from where the alteration or malicious upload came from?

[Ryan J Ollos]

For a publicly-accessible site, you probably want to be running SpamFilter. See also #12398.

For the specific case in which a site was hacked and you want to know the IP address, there are probably better ways to obtain that information such as logging from the web server.

[Ryan J Ollos]

Proposed changes in log:rjollos.git:t9612.

[Christian Boos]

LGTM, + 2 remarks:

• we should also remove show_ip_addresses in chrome.py
• db42.py makes me think we could start to introduce a DDL helper functions, like upgrade_table(old_Table, new_Table) and a Table.remove_columns that would create a new Table instance, so that one could write upgrade_table(attachment_table.remove_columns('ipnr'), attachment_table)

[Ryan J Ollos]

Replying to Christian Boos:

• we should also remove show_ip_addresses in chrome.py

show_ip_addresses was removed in r14909. That change should probably have been committed with these changes.

• db42.py makes me think we could start to introduce a DDL helper functions, like upgrade_table(old_Table, new_Table) and a Table.remove_columns that would create a new Table instance, so that one could write upgrade_table(attachment_table.remove_columns('ipnr'), attachment_table)

Similar aim in #11872. I'll look into it.

[Ryan J Ollos]

Proposed changes in log:rjollos.git:t9612.2. Some of the changes will be committed to 1.2-stable to address #11872.

[Jun Omae]

I think the upgrade_tables method should call update_sequence if the table has an auto increment primary key.

[Ryan J Ollos]

Replying to Jun Omae:

I think the upgrade_tables method should call update_sequence if the table has an auto increment primary key.

Is this only needed if the column with auto_increment has changed, or has been removed? If the column with auto_increment has not changed, then that column will have the same data in the new table as in the old.

[Jun Omae]

Replying to Ryan J Ollos:

Replying to Jun Omae:

I think the upgrade_tables method should call update_sequence if the table has an auto increment primary key.

Is this only needed if the column with auto_increment has changed, or has been removed? If the column with auto_increment has not changed, then that column will have the same data in the new table as in the old.

The upgrade_tables re-create tables. Auto-increment column uses sequence object in PostgreSQL. Then, the sequence object would be re-created and started with 1. Inserting old records wouldn't change the sequence.

See also #8575.

[Ryan J Ollos]

[7ea7e2d5/rjollos.git] calls update_sequence. I'll propose additional changes after #11872 is implemented.

[Ryan J Ollos]

Fix for Cannot operate on closed database error committed to 1.0-stable in r15018, merged to 1.2-stable in r15019, merged to trunk in r15020.

DatabaseManager.upgrade_tables committed to 1.2-stable in r15021,r15023, merged to trunk in r15022,r15024.

[Ryan J Ollos]

Committed to trunk in r15027, r15028, r15037. Eventually I'll start remembering to add files when pushing from a Git repos to SVN.

[Ryan J Ollos]

I was dumb and didn't run tests before committing r15037. To fix the broken build, I guess I need to create a temp directory:

trac/upgrades/tests/db42.py

@@ -12 +12 @@
 # history and logs, available at http://trac.edgewall.org/.
 
 import copy
+import tempfile
 import unittest
 
 from trac.db.api import DatabaseManager
@@ -56 +57 @@
 class UpgradeTestCase(unittest.TestCase):
 
     def setUp(self):
-        self.env = EnvironmentStub()
+        self.env = EnvironmentStub(path=tempfile.mkdtemp())
         self.dbm = DatabaseManager(self.env)
         with self.env.db_transaction:
             self.dbm.drop_tables(new_schema)

Or instead just call self.env.reset_db() followed by self.env.shutdown()?

[Ryan J Ollos]

The test failure I introduced in r15037 was fixed in r15038. However, tests are ​still failing with on-disk SQLite. I suspect the pathway through destroy_db in db42.py has exposed an existing bug.

[Jun Omae]

I think we should close connections before calling destroy_db

trac/test.py

@@ -384 +384 @@
                 tables = dbm.reset_tables()
             else:
                 # different version or version unknown, drop the tables
+                if self.dburi.startswith('sqlite:') and \
+                        self.dburi != 'sqlite::memory:':
+                    # closes connections to avoid self.destroy_db() raises
+                    # "WindowsError: [Error 32]" when deleting on-disk SQLite
+                    dbm.shutdown()
                 self.destroy_db()
 
         if not tables:

[Ryan J Ollos]

Yeah, that looks good, I was working on a similar patch. I was hoping to keep the db specific information out of EnvironmentStub, but I don't see a way.

[Ryan J Ollos]

It might be worth considering to put the logic in DatabaseManager.destroy_db()​. I suppose a test would be useful to prove the case, but from looking at the code, wouldn't the logic be needed if there was a reason to destroy an on-disk SQLite database outside of EnvironmentStub?

[Jun Omae]

Okay. Putting it in DatabaseManager.destroy_db() sounds good.

[Ryan J Ollos]

Thanks, I'll go ahead and prepare that change, with tests.

[Ryan J Ollos]

I added db_exists to DatabaseManager and made the changes to destroy_db: log:rjollos.git:t9612_destroydb. I haven't tested with PostgreSQL 8.x. Tests are passing on Travis CI and AppVeyor.

[Jun Omae]

Replying to Ryan J Ollos:

The test failure I introduced in r15037 was fixed in r15038.

Running unit tests with r15037 while investigating the failure, the EnvironmentStub.reset_db_and_disk() accidentally removed trac in my source directory. EnvironmentStub.path is set to os.path.dirname(trac.__file__) if path argument is empty at tags/trac-1.0.12/trac/test.py@:360-365#L332​.

>>> env = EnvironmentStub()
>>> env.path
'/src/tracdev/git/trac'

I think we should check whether self.path is in current directory to avoid the accident before removing self.path directory.

trac/test.py

@@ -475 +475 @@
         """
         self.reset_db()
         self.shutdown() # really closes the db connections
-        shutil.rmtree(self.path)
+        cwd = os.getcwd()
+        # avoid removing source directory
+        if os.path.commonprefix([cwd, self.path]) != cwd:
+            shutil.rmtree(self.path)
         if self._old_registry is not None:
             self.restore_component_registry()
 

[Ryan J Ollos]

I've encountered that problem as well. It might not be an issue, but an environment created in the root of the source directory would not be removed.

The reason I started thinking about that was remembering that the functional tests, by default, create the environment in <path to source>/testenv/trac: trunk/trac/tests/functional/__init__.py@14794:133-138#L122​. The functional tests don't call reset_db_and_disk, so your patch doesn't create a problem in this case.

It's probably bad practice to create an environment directory in the source tree, so I think the patch is okay. I have two thoughts on the functional test cases though:

• Should we destroy the environment at the end of the test run?
• Should we put the environment in a temp directory? If putting in a temp directory, we might want to randomize the directory name.

Also, adding a mkdir function in trac.test, we could ensure use of the same prefix for all temporary directories.

log:rjollos.git:t9612_destroy_functional_testenv

[Jun Omae]

Replying to Ryan J Ollos:

I added db_exists to DatabaseManager and made the changes to destroy_db: log:rjollos.git:t9612_destroydb. I haven't tested with PostgreSQL 8.x. Tests are passing on Travis CI and AppVeyor.

Reconsidering, we could simply call self.shutdown() before connector.destroy_db() because destroy_db directly connects a database without ConnectionPool: [bbe878e89/jomae.git]

[Ryan J Ollos]

Replying to Jun Omae:

Reconsidering, we could simply call self.shutdown() before connector.destroy_db() because destroy_db directly connects a database without ConnectionPool: [bbe878e89/jomae.git]

That is much cleaner. I didn't even consider that it might work, but it's definitely the better solution.

[Jun Omae]

Replying to Ryan J Ollos:

I have two thoughts on the functional test cases though:

• Should we destroy the environment at the end of the test run?
• Should we put the environment in a temp directory? If putting in a temp directory, we might want to randomize the directory name.

When an exception is raised from twill, the html is saved as <test-case-name>.html in log directory of the test environment. I think it would be better to keep the environment if tests are failing, at least.

[Ryan J Ollos]

Good point. I added additional changes to consider:

• Create temporary dir for functional test run in system temp directory, but don't delete it at the end of a test run: [8788908e/rjollos.git].
• Use the system temp directory as the default for EnvironmentStub.path, which would make the comment:39 change unnecessary: [808e806f/rjollos.git].

[Jun Omae]

I consider the following code in [808e806f/rjollos.git] would remove all owned files in the temporary directory if the path of EnvironmentStub.__init__ is None and reset_db_and_disk() is wrongly called in tearDown().

            self.path = tempfile.gettempdir()

$ mkdir /dev/shm/t9612
$ touch /dev/shm/t9612/test.txt
$ ls -al /dev/shm/t9612/
total 0
drwxr-xr-x  2 jun66j5 jun66j5  60 Aug  5 22:22 .
drwxrwxrwt 20 root    root    440 Aug  5 22:20 ..
-rw-r--r--  1 jun66j5 jun66j5   0 Aug  5 22:22 test.txt
$ TMP=/dev/shm/t9612 python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import tempfile
>>> from trac.test import EnvironmentStub
>>> tempfile.gettempdir()
'/dev/shm/t9612'
>>> env = EnvironmentStub()
>>> env.path
'/dev/shm/t9612'
>>> env.reset_db_and_disk()
>>> os.listdir(env.path)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 2] No such file or directory: '/dev/shm/t9612'

[Ryan J Ollos]

I added changes to address comment:45 in [8b14dd6b/rjollos.git].

[Jun Omae]

If unit tests run with on-disk SQLite file, we generally use TRAC_TEST_DB_URI=sqlite:test.db. With the branch, /tmp/test.db would be created (currently, trac/test.db is created in source tree). Then, only unit tests could run in the same time on system.

[Ryan J Ollos]

Replying to Ryan J Ollos:

Replying to Jun Omae:

Reconsidering, we could simply call self.shutdown() before connector.destroy_db() because destroy_db directly connects a database without ConnectionPool: [bbe878e89/jomae.git]

That is much cleaner. I didn't even consider that it might work, but it's definitely the better solution.

Committed to 1.2-stable in r15042, merged to trunk in r15043. I tested with PostgreSQL 8.4.20 on CentOS 6.8.

[Ryan J Ollos]

Replying to Ryan J Ollos:

Also, adding a mkdir function in trac.test, we could ensure use of the same prefix for all temporary directories.

Committed to trunk in r15050 (commit message should say mkdtemp rather than mkdir).

[Jun Omae]

I think DatabaseManager.upgrade_tables should quote table name and column names using db.quote().

trac/db/api.py

@@ -425 +425 @@
         """
         for new_table in new_schema:
             temp_table_name = new_table.name + '_old'
-            old_column_names = set(self.get_column_names(new_table))
-            new_column_names = set(col.name for col in new_table.columns)
-            cols_to_copy = ','.join(old_column_names & new_column_names)
+            column_names = (set(self.get_column_names(new_table)) &
+                            set(col.name for col in new_table.columns))
             with self.env.db_transaction as db:
+                cols_to_copy = ','.join(db.quote(name)
+                                        for name in column_names)
                 cursor = db.cursor()
                 cursor.execute("""
                     CREATE TEMPORARY TABLE %s AS SELECT * FROM %s
-                    """ % (temp_table_name, new_table.name))
+                    """ % (db.quote(temp_table_name),
+                           db.quote(new_table.name)))
                 self.drop_tables((new_table,))
                 self.create_tables((new_table,))
                 cursor.execute("""
                     INSERT INTO %s (%s) SELECT %s FROM %s
-                    """ % (new_table.name, cols_to_copy,
-                           cols_to_copy, temp_table_name))
+                    """ % (db.quote(new_table.name), cols_to_copy,
+                           cols_to_copy, db.quote(temp_table_name)))
                 for col in new_table.columns:
                     if col.auto_increment:
                         db.update_sequence(cursor, new_table.name, col.name)

[Jun Omae]

Minor issue: if there is no common columns between old and new tables, INSERT INTO %s (%s) SELECT %s FROM %s would fail to execute.

[Ryan J Ollos]

Proposed changes look good to me.

[Jun Omae]

Replying to Ryan J Ollos:

Proposed changes look good to me.

Thanks for the reviewing. Committed in [15317] and merged in [15318].

[Jun Omae]

Replying to Jun Omae:

Minor issue: if there is no common columns between old and new tables, INSERT INTO %s (%s) SELECT %s FROM %s would fail to execute.

Proposed changes in t9612-no-common-columns.diff​.

[Jun Omae]

I noticed .upgrade_table creates transaction per each table. It would be better to create one transaction for call of .upgrade_table?

trac/db/api.py

@@ -423 +423 @@
 
         :since: version 1.2
         """
-        for new_table in new_schema:
-            temp_table_name = new_table.name + '_old'
-            old_column_names = set(self.get_column_names(new_table))
-            new_column_names = set(col.name for col in new_table.columns)
-            column_names = old_column_names & new_column_names
-            with self.env.db_transaction as db:
+        with self.env.db_transaction as db:
+            cursor = db.cursor()
+            for new_table in new_schema:
+                temp_table_name = new_table.name + '_old'
+                old_column_names = set(self.get_column_names(new_table))
+                new_column_names = set(col.name for col in new_table.columns)
+                column_names = old_column_names & new_column_names
                 cols_to_copy = ','.join(db.quote(name)
                                         for name in column_names)
-                cursor = db.cursor()
                 cursor.execute("""
                     CREATE TEMPORARY TABLE %s AS SELECT * FROM %s
                     """ % (db.quote(temp_table_name),

[Ryan J Ollos]

Both changes looks good.

[Jun Omae]

Thanks. Committed in [15319] and merged in [15320].

[Ryan J Ollos]

Fixed one oversight in r16784. The test is currently unused.