GitPlugin: Fix cache

[Peter Suter]

Created as part of the move of GitPlugin. Tickets originally reported for ​th:GitPlugin: ​th:#8016, ​th:#4505

Quoting ​xarkam:

If files in git repositories as renamed, the git plugin does not read the change.

Change on trac.ini in [git] section cached_repository = true to false resolve this bug.

Quoting ​anonymous:

Once a commit is reset, GitPlugin keeps complaining about the missing changeset even when newer changesets modify the same files. Browsing the git repo is not possible anymore after this. Disabling the cache in trac.ini worked around the errors, but browsing changes is now very slow.

[Christian Boos]

We may still need a cache even after #10606, but what exactly remains to be determined (quite likely some kind of tree → commit mapping).

[Jun Omae]

Replying to psuter:

Quoting ​anonymous:

Once a commit is reset, GitPlugin keeps complaining about the missing changeset even when newer changesets modify the same files. Browsing the git repo is not possible anymore after this. Disabling the cache in trac.ini worked around the errors, but browsing changes is now very slow.

Proposed change are in log:jomae.git@t10602.

• Added GitCachedRepository.sync instead of CachedRepository.sync
• The GitCachedRepository.sync creates revision and node_change records from older revision to newer for each branch.
  • CachedRepository.sync creates from older to newer in date order.
• If HEAD is reset, the sync updates metadata[CACHE_YOUNGEST_REV] to avoid a NoSuchChangeset

Thoughts?

[Jun Omae]

Proposed change are in log:jomae.git@t10602.

Rebased branch on 1.0-stable r12533: log:jomae.git@t10602.1.

[anonymous]

Any news on this? It's an important issue and having git cache disabled slows down everything.

[Ryan J Ollos]

Yes, there is a proposed change in comment:3. You could test it out and report back your findings.

[anonymous]

Unfortunately, the patch conflicts. Also are there any plans to finally cut a new release?

[anonymous]

Hm, there seems like there is fixes for git cache in the upstream 1.0-stable already, how does that integrate with this patch?

Like this for example: [12479/branches/1.0-stable/tracopt/versioncontrol/git/git_fs.py]

Could you rebase it so it would be easier to test it out?

[Jun Omae]

Sorry for the delay. I rebased the changes, jomae.git@t10602.2. Could you please try the new branch?

[anonymous]

Thanks alot, will try it out.

[anonymous]

Hmm.. having some troubles, but im doing a resync currently.. noticed in the log that for each commit it does:

                SELECT DISTINCT s.sid, n.value, e.value
                FROM session AS s
                 LEFT JOIN session_attribute AS n ON (n.sid=s.sid
                  and n.authenticated=1 AND n.name = 'name')
                 LEFT JOIN session_attribute AS e ON (e.sid=s.sid
                  AND e.authenticated=1 AND e.name = 'email')
                WHERE s.authenticated=1 ORDER BY s.sid

We have the trac_user_rlookup = true, which is probably why it does it, but does it really need to do it for EVERY commit?

And if it does need to do that… why don't search for the email(?) directly instead of fetching all users?

Also, what is the recommended settings now for cached_repository and persistent_cache ?

[Jun Omae]

Replying to anonymous:

Hmm.. having some troubles, but im doing a resync currently..

Please post your testing procedure and details of the troubles.

noticed in the log that for each commit it does: […] We have the trac_user_rlookup = true, which is probably why it does it, but does it really need to do it for EVERY commit?

And if it does need to do that… why don't search for the email(?) directly instead of fetching all users?

Currently, users list is not cached.

[Ryan J Ollos]

Replying to jomae:

Currently, users list is not cached.

Is the query from Environment.get_known_users: trunk/trac/env.py@12843:687-695#L679​?

If I understand correctly, caching of the user list is proposed to be added in #7339 (though the patch needs more work).

[anonymous]

Replying to rjollos:

Replying to jomae:

Currently, users list is not cached.

Is the query from Environment.get_known_users: trunk/trac/env.py@12843:687-695#L679​?

If I understand correctly, caching of the user list is proposed to be added in #7339 (though the patch needs more work).

Yes thats the function it uses. Adding a get_user_by_email would be one solution, if there is no global cache.

[anonymous]

Replying to jomae:

Replying to anonymous:

Hmm.. having some troubles, but im doing a resync currently..

Please post your testing procedure and details of the troubles.

Having random problems with some commits not wanting to trigger the ticket update thing.. sometimes they work, sometimes not, don't think its related to this branch tho.

[Jun Omae]

Ah, at least, repository sync should be run via post-receive hook when branches are removed or renamed. Otherwise, Trac wouldn't know git repository is changed.

trac-admin $ENV repository sync $reponame

[anonymous]

I used changeset added "(default)" *SHA1* and sometimes it works and sometimes not, nothing in the log when it dosent work, it never calls the hook.

What is more strange tho is doing a resync:

With 1.0.1: 9254 revisions cached

With jomae.git@t10602.2: 8932 revisions cached

Did a comparison and it seems like its mostly Merge-commits missing

[Jun Omae]

git post-receive hook

[Jun Omae]

Replying to anonymous:

I used changeset added "(default)" *SHA1* and sometimes it works and sometimes not, nothing in the log when it dosent work, it never calls the hook.

Hmm, it works well except merge-commits.

Did a comparison and it seems like its mostly Merge-commits missing

Oh, sorry. I'm wrong. Revised log:jomae.git@t10602.2. I tested with uwsgi (4 processes) and post-receive​ hook for adding, updating, force updating and removing branch.

[anonymous]

Hi

Just tried your commit and now it finds the same number of commits as 1.0.1!

We're using the ​https://github.com/aaugustin/trac-github to take care of the hooks, and it works fine 99% of the times (think the cache is the blame when its not working)

Looking forward to see this branch merged into 1.0-stable

[Jun Omae]

Thanks for the feedback. Committed in [12972] and merged to trunk in [12973].