Trac Lockup - Maybe you wont invalid it this time

[Robert_B_Taylor@…]

Here is what is happening since I have nailed down the source. If I have an open trac-admin to a given trac environment, and someone goes to click on the "Timeline" the svn-repo gets locked.

We have to run svnadmin recover to unlock the DB, because unless we do this no one can view the Timeline or Browse the source even when we exit trac-admin.

Here is a stack trace from the problem:

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 525, in cgi_start
    real_cgi_start()
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 520, in real_cgi_start
    dispatch_request(path_info, args, req, env)
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 414, in dispatch_request
    auth_cookie = authenticator.login(req)
  File "/usr/lib/python2.3/site-packages/trac/auth.py", line 45, in login
    int(time.time()));
  File "/usr/lib/python2.3/site-packages/sqlite/main.py", line 237, in execute
    self.con._begin()
  File "/usr/lib/python2.3/site-packages/sqlite/main.py", line 503, in _begin
    self.db.execute("BEGIN")
OperationalError: database is locked

[Matthew Good <trac matt-good net>]

I'm closing this for the same reasons that #1349 was closed. There are problems with the way the BerkeleyDB backend for Subversion works that Trac cannot avoid. If you're not already running Subversion 1.1, upgrade to it, and convert your repository to use the FSFS backend. If you still have problems when using FSFS you can report them, but the BerkeleyDB problems are really beyond the control of Trac.

[Mark Rowe]

This isn't a Subversion problem.

You say that We have to run svnadmin recover to unlock the DB, because unless we do this no one can view the Timeline or Browse the source even when we exit trac-admin. There is no logical way that running svnadmin recover would fix the error that you describe. Can you please investigate a little further and see if you can track down what you are doing that resolves the problem?

[Matthew Good <trac matt-good net>]

Yes, based on the traceback provided, the issue that you're seeing is different from what you described. The traceback indicates that the Trac SQLite database is locked. This is entirely separate from the Subversion repository, so running "svnadmin recover" shouldn't affect the Trac database.

However, since you mentioned in #1349 that you were running SVN with BerkeleyDB, I would still recommend switching to FSFS if you have not already. There's a good description of the benefits of FSFS here: ​http://svn.collab.net/repos/svn/trunk/notes/fsfs

There are very few reasons for sticking to BerkeleyDB and I'm hoping that the SVN developers make FSFS the default soon.

[Robert_B_Taylor@…]

I described to you exactly how I locked up the database. Maybe this is still an issue of Berkeley DB, but running svnadmin recover is the only way to resolve what happens.

I cannot change it from Berkeley db since other people use the db on the machine. The subversion is 1.1.3.

Also, to the comment that there is no logical way that running svnadmin recover would fix the error that you describe…. Well, that is what fixes the problem. We cannot view the timeline or browse the source until we run svnadmin recover on the repo. We have tried killing off all other processes, rebooting, etc. I have also tried every permission combination listed in the msg forums, and this occurs with all which makes me believe this is not a permissions issue.

[Mark Rowe]

What command(s) do you type in trac-admin prior to the viewing of the Timeline page that causes the lockup? When Trac locks up does svn ls http://server/path/to/repository give you an error about the repository being corrupt and that you should run svnadmin recover?

My current line of thinking is that the error handling in Trac doesn't handle a "corrupt" Subversion repository properly. For example, it may somehow manage to lock it's SQLite database prior to attempting to open the repository. When the open fails, it attempts to do something else with the SQLite database which fails due to the lack of cleanup during the previous step. This is just a guess, but hopefully with a little more information we can track the problem down further.

[Matthew Good]

I've tried the steps that you described both with an FSFS repository and with BDB and have been unable to reproduce the problem, so I don't have much information to go on to help you out.

I'm not sure I understand your comment about being unable to convert the repository format due to other people using the database. The Subversion repository is stored internally, and will only ever be accessed through an svn client or server, either which will be able to handle either storage backend just fine. The whole process should only take a few minutes. Here are instructions on ​converting bdb to fsfs if you need them.

What you may want to do is to leave your existing repository intact, but make a copy of it that uses FSFS and see if the problems still occur. This way you won't interrupt the use of your repository, but will be able to experiment and see if FSFS alleviates the issues. Simply follow the first 2 steps to create the new FSFS repository and import the existing data into it, then create a Trac project that uses this new copy of the repository.

[Robert_B_Taylor@…]

Ive been working on the issue for the past week and still have run across more problems. For some reason my supervisor says we cannot implement the FSFS as a final solution because people use the BDB and they will be using it for a while.

Ok, onto something that actually concerns you guys. What I have been doing is examining the processes running when the repository locks up. So I have the trac-admin tool running connected to a given environment and someone clicks on the timeline or browse source. Well before clicking on the timeline/browse source tab there is only one apache process running that is someone using the svn client to co something from the repository. But after the person clicks on the timeline/browse source there are about 10+ apache processes running, which seems a bit high. The other problem is that when the user navigates away from the page those processes arent released. Which seems to eventually lead to this lockup problem were having.

So right now Im talking with the guy who setup the apache/svn configuration in the apache2.conf file because I think something may be incorrect there. If it is that problem I apologize in advance for wasting your time with this ticket. If not, oh well more searching.

[Christian Boos]

you said … there are about 10+ apache processes running, which seems a bit high.

This makes me wonder if it is somehow similar to the multithread issues I have with tracd (#1401)?

[Robert_B_Taylor@…]

I have a recent traceback from the lockup, looks like something with BDB again.

Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 525, in cgi_start
    real_cgi_start()
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 520, in real_cgi_start
    dispatch_request(path_info, args, req, env)
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 433, in dispatch_request
    module = module_factory(args, env, database, req)
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 175, in module_factory
    pool, rep, fs_ptr = open_svn_repos(repos_dir)
  File "/usr/lib/python2.3/site-packages/trac/core.py", line 452, in open_svn_repos
    rep = repos.svn_repos_open(repos_dir, pool)
SubversionException: ('Berkeley DB error while opening environment for filesystem /proj/svnrepos/dev/db:\nDB_RUNRECOVERY: Fatal error, run database recovery', 160029)

Uggg… :)

[Matthew Good <trac matt-good net>]

I still don't understand how people are using the BDB. BDB is not a SQL database. It is simply the storage format that Subversion uses to record the file history. The BDB files are local to a single Subversion repository; it is not stored in a database server. Converting the storage format of your repository will not affect any other databases in use on your system, and the change should be completely transparent to anyone accessing the repository since the storage format is all hidden behind the Subversion client interfaces.

[Mark Rowe]

I'm relatively sure that the BDB vs FSFS is irrelevant in this case. Robert, how large is the repository that Trac is working with? Approximately how many revisions? It may be that Trac is trying to sync the latest revisions before displaying the timeline but is taking too long and is being killed by Apache. In that case a trac-admin /path/to/repos resync should fix it. Aside from that, the only reason for the problem you describe would be incorrect permissions on your Subversion repository.

[Robert_B_Taylor@…]

To answer the BDB question, my boss tells me that people are using BDB with subversion right now, and I cannot change it. I have been pushing for us to go to FSFS with very little support.

To answer the repo question:

Size - 4GB Revisions - ~30,000

The lockup occurs not only on timeline, but also browse source. Again a svnadmin recover repo remedies this. I will try the resync though today.

[Robert_B_Taylor@…]

We finally switched to the FSFS, and have had not a single problem. Thanks alot fellas for your time.