load macro is slow

[toru.wata@…]

If there are many macros in wiki-macros dir, then performance to render wiki pages will be getting slower.

I got perf data, then I found imp.load_source would be a bottle neck.

imp.load_source is called in UserMacroProvider defined in trac/wiki/macros.py.

   def _load_macro(self, name):
        for path in (self.env_macros, self.site_macros):
            macro_file = os.path.join(path, name + '.py')
            if os.path.isfile(macro_file):
                return imp.load_source(name, macro_file)
        raise TracError, 'Macro %s not found' % name

And more worse, above codes are called every "wiki_to_html". So if some macro uses wiki_to_html, same macro will be loaded so many times, and performance getting worse.

To solve this problem, it is necessary to check whether macro is loaded

or not.

For example if you add following codes at begining of _load_macro, then performance to render wikipage getting better even if there are so many macros.

        from sys import modules
        if modules.has_key(name):
            return modules[name]

But in above codes, there are trade-off. You cannot update macros without restarting on tracd, mod_python environment.

But I'd like you consider performance enhancement for loading macros.

[Alec Thomas]

A better idea might be to store the mtime of the macro and reload it if changed.

[anonymous]

checking mtime requires IO for each execution. And IO is slow, generally.

I think it is better to check mtime only when "timed out". i.e. if it is loaded within recent "60 sec" then it should use loaded module. otherwise, it checks mtime and mtime is newer, then it should reload. (60 sec is a configuration parameter)

[Christian Boos]

Well, UserMacroProvider is now dead, so there's no point in this ticket anymore.