Increase token granularity for improved syntax highlighting

[atomiks]

It would be cool to add more tokens so different parts of the code can be highlighted differently. Prism seems more granular than CodeMirror, but it's still missing some nice tokens compared to Atom and VS Code.

For JavaScript, it would be nice to have these:

• token parameter

function add(a, b) { // <-- `a` and `b` receive `parameter`
  return a + b;
}

• token property definition and token property access

const obj = {
  example: true // <-- `example` receives `property` and `definition`
};
obj.example; // <-- `example` receives `property` and `access`

• token function method

this.func(); // `func` receives `method` if being called on an object

• token keyword module

All keywords here should receive the module token.

import { thing as thingy } from 'thing';
export default thingy;

• token keyword special

"special"(?) keywords like this

• token variable dom

window
document
navigator
performance
localStorage

• token console

console.log('hello'); // <-- `console` receives this token

• token operator spread

const copy = [...arr]; // <- `...` receives this token

[RunDevelopment]

If you want more fine-grained control about the way keywords are highlighted, I recommend the Highlight keywords plugin.

Also, token parameter is kind of covered by #1446.

[atomiks]

Thanks, I didn't see that plugin.

And the PR looks good. I'll create my own language extension and copy some of that code while the PR waits to be merged.

Do you have ideas for the regex necessary for the other tokens listed in the OP? I'm really not great with regex and can't figure it out on my own. 😭

[RunDevelopment]

Insert this code into prism-javascript.js in line 37 and you will have most of the features.

The code

Prism.languages.insertBefore('javascript', 'punctuation', {
	'definition': {
		pattern: /([{,]\s*)[a-z]\w*(?=\s*:)/i,
		lookbehind: true,
		alias: 'property'
	},
	'access': {
		pattern: /(\.\s*)[a-z]\w*/i,
		lookbehind: true,
		alias: 'property'
	},
	'variable ': {
		pattern: /\b(?:window|document|navigator|performance|localStorage)\b/,
		alias: 'dom'
	},
	'console': /\bconsole\b/,
	'spread': {
		pattern: /\.{3}/,
		alias: 'punctuation'
	}
});

But let me warn you:

1. I did this quick and dirty, so it's not perfect and no guarantees.
2. It really doesn't look nice IMO, you will probably have to adjust the default theme you're using.

Keep in mind that if you want to use the minified file, you will have to rebuild Prism (refer to README.md).

[atomiks]

Wow thank you so much. For the code snippets I'm using it's working pretty well. I appreciate it!

[atomiks]

Spread wasn't working, adding it before operator works though. Also need _$ in the property regex.

I was able to add method as well inserting it before function.

However, the property definition doesn't work if there are comments within the object.

{
  // Here's a comment
  property: true
}

Any fix for that? Everything's working nicely apart from that 😄. Thanks tons for the help @RunDevelopment

[RunDevelopment]

Spread wasn't working, adding it before operator works though.

My bad.

Also need _$ in the property regex.

Just insert them into the character sets. Change [a-z]\w* to [a-z_$][\w$]*.

However, the property definition doesn't work if there are comments within the object.

That's a really tough one.
You see, the pattern tries to eliminate false positive from uses of the conditional operator foo ? bar: 0 by saying: Every property definition has to preceded by either a , or { which works fine as long as you ignore comments.

We could adjust the lookbehind ([{,]\s*) to be something like this ([{,]\s*(?:\/\/.*\r?\n\s*)?). But then the property definition has to be matched before comments which causes all kinds of problems.
So a perfect solution is probably impossible.

You can however say: My code is well-formatted as such: property: value (no space before the colon) and cond ? ifValue : elseValue (space(s) before the colon).
Then you can remove the lookbehind and change the lookahead from (?=\s*:) to (?=:).

[atomiks]

Just insert them into the character sets. Change [a-z]\w* to [a-z_$][\w$]*.

Yeah sorry I did add them to the character set and it works fine.

I had a feeling the comment thing would make it really hard! Your final sentence works fine for my code though. Thanks!

[atomiks]

For anyone wondering, here are all the tokens I've hacked on to get pretty close to my VS code theme 😜. I'm also using Prism directly from GitHub to get the parameter highlighting.

@RunDevelopment I kind of don't know what I'm doing with alias here and I had to add some "fake" tokens like func to prevent overriding things. Improving this would be welcome because it feels pretty awful to me!

The func definition is meant to fix set in obj.set() receiving keyword as a token instead of function.

Prism.languages.insertBefore('javascript', 'keyword', {
  module: {
    pattern: /\b(?:import|as|export|from|default)\b/,
    alias: 'keyword',
  },
  op: {
    pattern: /\b(?:typeof|new|of|delete)\b/,
    alias: 'keyword',
  },
  nil: {
    pattern: /\b(?:null|undefined)\b/,
    alias: 'keyword',
  },
  flow: {
    pattern: /\b(?:return|await)\b/,
    alias: 'keyword',
  },
  func: {
    pattern: /(\.\s*)[a-z_$][\w$]*(?=(\())/i,
    lookbehind: true,
    alias: 'method',
  },
})

Prism.languages.insertBefore('javascript', 'punctuation', {
  definition: {
    pattern: /[a-z]\w*(?=:)/i,
    lookbehind: true,
    alias: 'property',
  },
  access: {
    pattern: /(\.\s*)[a-z_$][\w$]*/i,
    lookbehind: true,
    alias: 'property',
  },
  dom: {
    pattern: /\b(?:window|document|navigator|performance|localStorage)\b/,
    alias: 'variable',
  },
  console: /\bconsole\b/,
  class: {
    pattern: /\b[A-Z][A-Za-z0-9_]+\b/,
    alias: 'class-name',
  },
})

Prism.languages.insertBefore('javascript', 'operator', {
  spread: {
    pattern: /\.{3}/,
    alias: 'punctuation',
  },
  arrow: {
    pattern: /=>/,
    alias: 'operator',
  },
})

Prism.languages.insertBefore('javascript', 'function', {
  method: {
    pattern: /(\.\s*)[a-z_$][\w$]*(?=(\())/i,
    lookbehind: true,
    alias: 'function',
  },
})

Also one last thing I'd like to have: string highlighting in CSS attribute selectors. Example:

.example[attr="value"] {
  color: red;
}

How would I make "value" receive the string token?

---

Here you can see tons of highlighting granularity as an example that you don't get by default with Prism: https://atomiks.github.io/tippyjs/tippy-instance/

[RunDevelopment]

How would I make "value" receive the string token?

This will come soon to CSS Extras with #1638 and #1671.
The only difference is that I highlight your "value" not as string but as value (that's the token name).

[RunDevelopment]

I gotta say: The code on your website looks really good!

[RunDevelopment]

@atomiks Can I use this for a new language JS Extras?

[atomiks]

Yup sure! You'd need to test it with a bunch of code samples of unexpected code (as you mentioned before with the ? : thing, but for most code using Prettier, it should work well. I haven't noticed anything unexpected on my website.

[RunDevelopment]

Thank you very much!

[RunDevelopment]

I'm going to close this now because #1743 and #1671 are merged.

[atomiks]

@RunDevelopment

The following to highlight object properties no longer seems to work (latest Prism does not tokenize property):

Prism.languages.insertBefore('javascript', 'punctuation', {
  definition: {
    pattern: /([{,]\s*)[a-z]\w*(?=\s*:)/i,
    lookbehind: true,
    alias: 'property',
  },
});

const code = `
const object = {
  property: true
};
`.trim();

Prism.highlight(code, Prism.languages.javascript, 'javascript');

Do you have an updated example?

The last working version is 1.17.1, it stopped working in 1.18.0.

[RunDevelopment]

Prism new supports JS object properties out of the box (#3099).

If you want to keep using your definition token, change your code to:

-Prism.languages.insertBefore('javascript', 'punctuation', {
+Prism.languages.insertBefore('javascript', 'property', {

[atomiks]

Ah okay, I see that the change hasn't been released to npm yet. I'm also using prism-react-renderer which copies the output to better support React projects.

I've added:

Prism.languages.insertBefore('javascript', 'operator', {
	'literal-property': {
		pattern: /((?:^|[,{])[ \t]*)(?!\s)[_$a-zA-Z\xA0-\uFFFF](?:(?!\s)[$\w\xA0-\uFFFF])*(?=\s*:)/m,
		lookbehind: true,
		alias: 'property'
	},
});

and it works. Thanks!