Adding Pygments syntax highlighting for new file type does not work as documented

[kriegaex@…]

I failed for hours in adding syntax highlighting to Trac for a new file type I had added to my local Pygments installation before and which worked well with pygmentize on the console. The file type does not have shebang, but must be detected by a file name pattern (I need globbing, i.e. pattern matching for it because there is no fixed file extension).

The documentation talks about setting mime_map and pygments_modes in the mimeviewer section of trac.ini, but this either did nothing or led to errors in Trac, stating that Pygmentize did not know the specified MIME type (I forgot the exact error message).

What I ended up doing were two things not mentioned in the documentation at all:

• I added a new (self-invented) MIME type text/x-something to /etc/mime.types.
• I explicitly set the MIME type via svn propset svn:mime-type text/x-something my_file for each relevant file in my SVN repository.

Maybe there is another solution, but if so, the documentation does not mention it. What it does describe is incomplete at best and outright wrong in my opinion. So I request the following:

• Please correct and extend the documentation, adding a step-by-step tutorial for adding a new file type for syntax highlighting to Trac via Pygments.
• Please enable Trac to let Pygments determine the file type directly by its own file name patterns, because this works nicely in pygmentize. This would let most users get rid of having to modify mime.types and enable to correctly highlight files without a MIME type explicitly set in SVN. The advantages are clear: no MIME type fiddling, no trac.ini editing.

[Alexander Kriegisch <kriegaex@…>]

I found the error message I had before: "Exception: No Pygments lexer found for mime-type 'text/x-something'."

[olistudent]

Issued in source:trunk/trac/mimeview/pygments.py#L120​

[Remy Blank]

That's to invasive for 0.12.3, and probably even for 0.12-stable.

Note that setting svn:mime-type is only half a solution, as it won't work with Mercurial or Git repositories.

[Alexander Kriegisch <kriegaex@…>]

I know that setting svn:mime-type is a workaround rather than a solution. This is why I wanted it fixed in Trac in the first place. ;-)

And why is it too invasive? You just need to

• call an external porgam (a.k.a. black box) in a slightly different way and
• fix the (semi-)documented behaviour in the product for users who prefer to solve this in Trac rathen than Pygments plus document it better,

don't you?

For my personal needs, though, I would be glad to receive a quick fix as a patch and test it for you, then wait for the next minor or major update, whichever seems more appropriate to you. All I want is get rid of those SVN MIME types and let Pygments do its own work.

[Alexander Kriegisch <kriegaex@…>]

Add file name regex matching to Trac for MIME type identification

[Alexander Kriegisch <kriegaex@…>]

As you can see, with my (unfortunately very limited) Python skills I extended mimeview/api.py​ in a fairly non-invasive way by adding a new config option to trac.ini, section [mimeviewer]. The option is named mime_map_patterns and contains a list of key-value mappings from MIME types to file name regex patterns. This is much more powerful than just literal file extension matches. The patch contains two examples, see there for more info.

BTW, the original location of the patch is ​http://freetz.org/browser/trunk/tools/developer/mime_map_patterns.trac.patch.

As for my claim about the patch being non-invasive: Look where it is applied. It only kicks into action *after* the other MIME type matching mechanisms have failed. Thus, the patch is backward-compatible with what Trac does today and should be safe to be applied to the 0.12.x branch - no need to wait for the next major version IMHO.

[Christian Boos]

See also #5533.

[Alexander@…]

The patch in this ticket has been rotting since October 2011, so I forked the ​Trac Git repository, committed the patch both in trunk and in branch 0.12-stable and created two corresponding requests:

• ​pull request for branch trunk
• ​pull request for branch 0.12-stable

I hope this makes it easier for Trac's maintainers to integrate the new setting upstream.

[Christian Boos]

Thanks for the reminder! Yes, providing a pull request (or just a link to the proper changeset or branch in a git repository) indeed helps, though we don't use github directly for merging the requests.

As for the proposed change itself, I made a few modifications:

• TracDev/CodingStyle cleanups (mostly line length)
• improve docstring of mime_map_patterns, make clear that ':' is simply used to separate the MIME type from the regexp
• ignore the mapping and issue a warning when there's an error in the regexp
• as specifying associations with regexps can be very powerful and much more precise than the suffix matching done by mime_map, it feels a waste to do it as a last resort; doing it first allows one to override a mapping (e.g. override '.txt' → text/plain with text/x-cmake:CMakeLists\.txt)

Finally, 0.12.x is really in feature freeze now, so the feature can only make it for 1.0 (or later).

Integrating one extra docstring for 1.0 is a minor breach to the string freeze, but as most config options are still untranslated, adding one more won't hurt…

See log:cboos.git:ticket10437-mime_map_patterns-override.

[Christian Boos]

Applied in r11229. Thanks Alexander!