syntax highlighting for javascript code broken in 0.12b1

[anonymous]

TracSyntaxColoring for #!js broken with Trac 0.12b1, see

function buildFixesList(wenjians) {
  var rootUL = document.createElement("ul");
  var topLI = document.createElement("li");
  return 0;
}

[Remy Blank]

This doesn't seem to be a regression in Trac, as the same happens on the Bitten Trac. Maybe due to a Pygments upgrade? Strangely, the JavaScript snippet is rendered with the PlainTextRenderer.

I'll look into it.

[Carsten Klein <carsten.klein@…>]

When getting a lexer manually from pygments, e.g. get_lexer_by_name('js') it will return the correct lexer:

Output by Javascript Lexer is

>>> for token in lexer.get_tokens(content):
...     print token
...
(Token.Keyword.Declaration, u'function')
(Token.Text, u' ')
(Token.Name.Other, u'test')
(Token.Punctuation, u'()')
[...]

for an input of

function test()
{
}

What I also notices is that you no longer can use the long name for that language, e.g. 'javascript', it will then fail will an unknown processor error:

{{{#!javascript
function test()
{}
}}}

will yield

function test()
{}

So maybe it actually does not try to query pygments available lexers in the first place, so this should actually be a regression in trac, don't you think?

[Carsten Klein <carsten.klein@…>]

Just noticed that, also PHP and other processors are broken as well.

Perhaps the class names defined in STANDARD_TYPES have changed, or the trac CSS no longer includes these style classes?

[Carsten Klein <carsten.klein@…>]

In mimeview.api there was a slight change to MODE_RE:

From

# Simple builtin autodetection from the content using a regexp
MODE_RE = re.compile(
    r"#!.+?env (\w+)|"                       # look for shebang with env
    r"#!(?:[/\w.-_]+/)?(\w+)|"               # look for regular shebang
    r"-\*-\s*(?:mode:\s*)?([\w+-]+)\s*-\*-|" # look for Emacs' -*- mode -*-
    r"vim:.*?(?:syntax|filetype|ft)=(\w+)"   # look for VIM's syntax=<n>
    )

to

MODE_RE = re.compile(r"""
      \#!.+?env\s+(\w+)                     # 1. look for shebang with env
    | \#!(?:[/\w.-_]+/)?(\w+)               # 2. look for regular shebang
    | -\*-\s*(?:mode:\s*)?([\w+-]+)\s*-\*-  # 3. look for Emacs' -*- mode -*-
    | vim:.*?(?:syntax|filetype|ft)=(\w+)   # 4. look for VIM's syntax=<n>
    """, re.VERBOSE)

Perhaps this is the problem w/ different versions of python?

[Carsten Klein <carsten.klein@…>]

Replying to Carsten Klein <carsten.klein@…>:

Just noticed that, also PHP and other processors are broken as well.

Perhaps the class names defined in STANDARD_TYPES have changed, or the trac CSS no longer includes these style classes?

However, PHP will be lexed by the pygments lexer alright, except that it will not be displayed correctly:

function test()
{
  var $a = 2;
}

[Remy Blank]

It seems that [8617], while using the more standard application/javascript MIME type instead of text/x-javascript, also broke JavaScript rendering with Pygments, as it doesn't recognize the former.

I have opened [pygments:ticket:504 #504] with Pygments to request adding the application/javascript MIME type.

In the meantime, we could work around that with the following patch:

trac/mimeview/pygments.py

@@ -186 +186 @@
             name = aliases and aliases[0] or lexname
             for mimetype in mimetypes:
                 self._types[mimetype] = (name, self.QUALITY_RATIO)
+
+        # Pygments currently doesn't know application/javascript
+        if 'application/javascript' not in self._types:
+            js_entry = self._types.get('text/javascript')
+            if js_entry:
+                self._types['application/javascript'] = js_entry
+
         self._types.update(
             Mimeview(self.env).configured_modes_mapping('pygments')
         )

Thoughts?

[Christian Boos]

'text/javascript' or 'text/x-javascript'?

[Remy Blank]

Replying to cboos:

'text/javascript' or 'text/x-javascript'?

It doesn't matter, both are recognized by Pygments. See the ​lexer documentation for Pygments, more specifically JavascriptLexer.

[Carsten Klein <carsten.klein@…>]

Actually it is always text/javascript. At least that is what modern browsers expect it to be when declaring script sections in the html.

[anonymous]

Replying to Carsten Klein <carsten.klein@…>:

Actually it is always text/javascript. At least that is what modern browsers expect it to be when declaring script sections in the html.

from what ​I've read, the only "modern" webbrowser which requires type="text/javascript" is IE… ;)

and it doesn't help that even the HTML5 spec mentions text/javascript :(

[Remy Blank]

We do use text/javascript as a MIME type for <script> tags. This ticket is about syntax highlighting of .js files, which was actually broken.

Anyway, patch applied in [9639].

[Ryan J Ollos]

Replying to rblank:

It seems that [8617], while using the more standard application/javascript MIME type instead of text/x-javascript, also broke JavaScript rendering with Pygments, as it doesn't recognize the former.

I have opened [pygments:ticket:504 #504] with Pygments to request adding the application/javascript MIME type.

The issue was fixed in Pygments 1.4. I modified the source code comment in [13183:13184] so that it will be clear that the code from [9639] can be removed if we ever drop support for Pygments < 1.4.