Use MDN data to power the CSS language server

[auchenberg]

I just had a chat with the MDN team, and they have published a NPM module that enables tools to take advantage of the detailed MDN data.

We should explore using https://github.com/mdn/browser-compat-data to power this language server

[praveenpuglia]

This is something we had a discussion on but no progress was made.
#40

[octref]

Actually the repo we should be using is https://github.com/mdn/data

[octref]

Some more insights:

• mdn/data has data for syntax of css properties, but not descriptions (discussion on this happening at Use MDN data to power auto completion in editors through Language Server Protocol mdn/data#199)

• Using MDN data could make us always up-to-date on newly added properties. An example: for this one [css] Add support for 'font-display' CSS property vscode#46423, the mdn data is:

      "font-display": {
        "syntax": "[ auto | block | swap | fallback | optional ]",
        "media": "visual",
        "percentages": "no",
        "initial": "auto",
        "computed": "asSpecified",
        "order": "uniqueOrder",
        "status": "experimental"
      }

[octref]

An analysis on CSS properties:

• Our side: Generated browsers.ts
• MDN: at-rules and properties

When we compare our browser data with MDN data, there are 239 rules in our side and not in MDN side, here is a complete list:

Properties in VSC not in MDN

alt
behavior
clip-rule
color-interpolation-filters
enable-background
fill
fill-opacity
fill-rule
flood-color
flood-opacity
glyph-orientation-horizontal
glyph-orientation-vertical
kerning
lighting-color
marker
marker-end
marker-mid
marker-start
motion
motion-offset
motion-path
motion-rotation
-moz-animation
-moz-animation-delay
-moz-animation-direction
-moz-animation-duration
-moz-animation-iteration-count
-moz-animation-name
-moz-animation-play-state
-moz-animation-timing-function
-moz-backface-visibility
-moz-background-clip
-moz-background-inline-policy
-moz-background-origin
-moz-border-image
-moz-box-align
-moz-box-direction
-moz-box-flex
-moz-box-flexgroup
-moz-box-ordinal-group
-moz-box-orient
-moz-box-pack
-moz-box-sizing
-moz-column-count
-moz-column-gap
-moz-column-rule
-moz-column-rule-color
-moz-column-rule-style
-moz-column-rule-width
-moz-columns
-moz-column-width
-moz-font-feature-settings
-moz-hyphens
-moz-perspective
-moz-perspective-origin
-moz-text-align-last
-moz-text-decoration-color
-moz-text-decoration-line
-moz-text-decoration-style
-moz-text-size-adjust
-moz-transform
-moz-transform-origin
-moz-transition
-moz-transition-delay
-moz-transition-duration
-moz-transition-property
-moz-transition-timing-function
-moz-user-select
-ms-behavior
-ms-flex
-ms-flex-align
-ms-flex-direction
-ms-flex-flow
-ms-flex-item-align
-ms-flex-line-pack
-ms-flex-order
-ms-flex-pack
-ms-flex-wrap
-ms-grid-column
-ms-grid-column-align
-ms-grid-columns
-ms-grid-column-span
-ms-grid-layer
-ms-grid-row
-ms-grid-row-align
-ms-grid-rows
-ms-grid-row-span
-ms-hyphens
-ms-ime-mode
-ms-interpolation-mode
-ms-layout-grid
-ms-layout-grid-char
-ms-layout-grid-line
-ms-layout-grid-mode
-ms-layout-grid-type
-ms-line-break
-ms-perspective
-ms-perspective-origin
-ms-perspective-origin-x
-ms-perspective-origin-y
-ms-progress-appearance
-ms-text-align-last
-ms-text-combine-horizontal
-ms-text-justify
-ms-text-kashida-space
-ms-text-overflow
-ms-text-size-adjust
-ms-text-underline-position
-ms-touch-action
-ms-transform
-ms-transform-origin
-ms-transform-origin-x
-ms-transform-origin-y
-ms-transform-origin-z
-ms-word-break
-ms-word-wrap
-ms-writing-mode
-ms-zoom
-ms-zoom-animation
nav-down
nav-index
nav-left
nav-right
nav-up
-o-animation
-o-animation-delay
-o-animation-direction
-o-animation-duration
-o-animation-fill-mode
-o-animation-iteration-count
-o-animation-name
-o-animation-play-state
-o-animation-timing-function
-o-border-image
-o-object-fit
-o-object-position
-o-table-baseline
-o-tab-size
-o-text-overflow
-o-transform
-o-transform-origin
-o-transition
-o-transition-delay
-o-transition-duration
-o-transition-property
-o-transition-timing-function
ruby-overhang
ruby-span
scrollbar-3dlight-color
scrollbar-arrow-color
scrollbar-base-color
scrollbar-darkshadow-color
scrollbar-face-color
scrollbar-highlight-color
scrollbar-shadow-color
scrollbar-track-color
shape-rendering
size
stop-color
stop-opacity
stroke
stroke-dasharray
stroke-dashoffset
stroke-linecap
stroke-linejoin
stroke-miterlimit
stroke-opacity
stroke-width
text-anchor
-webkit-animation
-webkit-animation-delay
-webkit-animation-direction
-webkit-animation-duration
-webkit-animation-fill-mode
-webkit-animation-iteration-count
-webkit-animation-name
-webkit-animation-play-state
-webkit-animation-timing-function
-webkit-appearance
-webkit-backdrop-filter
-webkit-backface-visibility
-webkit-background-clip
-webkit-background-composite
-webkit-background-origin
-webkit-border-image
-webkit-box-align
-webkit-box-direction
-webkit-box-flex
-webkit-box-flex-group
-webkit-box-ordinal-group
-webkit-box-orient
-webkit-box-pack
-webkit-box-sizing
-webkit-break-after
-webkit-break-before
-webkit-break-inside
-webkit-column-break-after
-webkit-column-break-before
-webkit-column-break-inside
-webkit-column-count
-webkit-column-gap
-webkit-column-rule
-webkit-column-rule-color
-webkit-column-rule-style
-webkit-column-rule-width
-webkit-columns
-webkit-column-span
-webkit-column-width
-webkit-filter
-webkit-flow-from
-webkit-flow-into
-webkit-font-feature-settings
-webkit-hyphens
-webkit-line-break
-webkit-margin-bottom-collapse
-webkit-margin-collapse
-webkit-margin-start
-webkit-margin-top-collapse
-webkit-mask-size
-webkit-nbsp-mode
-webkit-padding-start
-webkit-perspective
-webkit-perspective-origin
-webkit-region-fragment
-webkit-text-size-adjust
-webkit-transform
-webkit-transform-origin
-webkit-transform-origin-x
-webkit-transform-origin-y
-webkit-transform-origin-z
-webkit-transform-style
-webkit-transition
-webkit-transition-delay
-webkit-transition-duration
-webkit-transition-property
-webkit-transition-timing-function
-webkit-user-drag
-webkit-user-modify
-webkit-user-select

Observations:

• 189 of them are prefixed properties. Many are already implemented natively in browser and prefixes are no-longer required, but they might still exist in people's code-base
• Some of these properties are non-standard or deprecated

When we compare MDN data against ours, there are 87 properties not in our css (31 prefixed), and 9 at-rules not in our css:

Properties in MDN data but not in VSC data

-ms-ime-align
-moz-binding
-moz-context-properties
-moz-float-edge
-moz-force-broken-image-icon
-moz-image-region
-moz-orient
-moz-outline-radius
-moz-outline-radius-bottomleft
-moz-outline-radius-bottomright
-moz-outline-radius-topleft
-moz-outline-radius-topright
-moz-stack-sizing
-moz-text-blink
-moz-user-input
-moz-user-modify
-moz-window-dragging
-moz-window-shadow
-webkit-border-before
-webkit-border-before-color
-webkit-border-before-style
-webkit-border-before-width
-webkit-mask
-webkit-mask-attachment
-webkit-mask-composite
-webkit-mask-position
-webkit-mask-position-x
-webkit-mask-position-y
-webkit-mask-repeat-x
-webkit-mask-repeat-y
appearance
azimuth
backdrop-filter
box-align
box-direction
box-flex
box-flex-group
box-lines
box-ordinal-group
box-orient
box-pack
color-adjust
font-variation-settings
gap
hanging-punctuation
image-resolution
initial-letter
initial-letter-align
line-height-step
mask
mask-border
mask-border-mode
mask-border-outset
mask-border-repeat
mask-border-slice
mask-border-source
mask-border-width
mask-clip
mask-composite
offset
offset-anchor
offset-distance
offset-path
offset-position
offset-rotate
overflow-clip-box
overscroll-behavior
overscroll-behavior-x
overscroll-behavior-y
place-content
rotate
row-gap
ruby-merge
scale
scroll-snap-type-x
scroll-snap-type-y
text-combine-upright
text-decoration-skip
text-decoration-skip-ink
text-emphasis
text-emphasis-color
text-emphasis-position
text-emphasis-style
text-size-adjust
transform-box
translate
speak-as
font-display
font-variation-settings
bleed
marks
max-zoom
min-zoom
orientation
user-zoom

Observations:

• MDN data differentiate between at-rule properties and properties. Using this data we could check if a property is properly used, which is currently impossible with our data
• Most of these are newly added css properties
• MDN data has a status on each properties. For the 87 properties, we have:
  • 18 standard, 22 experimental, 43 nonstandard

Overall consideration:

• As MDN don't have short description for each css property/value, it's not possible to replace our browsers.ts entirely as of now
• MDN data is much more actively maintained than ours
• MDN data is richer and allows us to give more detailed, up-to-date suggestions, for example:
  • status: We can inform user when he's using a non-standard property
  • syntax: We might use this to provide correct property value suggestions, or hint invalid ones. This requires a parser that understand CSS Value definition syntax
  • Combining mdn/data and mdn/browser-compat-data, we can provide always up-to-date support matrix for each properties. Solves issues like Maintaining browser support data #40

[aeschli]

@octref Great analysis, thanks!
For the deprecated properties that are in css-schema.xml but not in MDN, let's add a 'deprecated' flag to our data. Deprecated properties would not proposed in code completions but still used for validation (we can even emit a deprecation warning)

[aeschli]

@octref I assume you wrote a script that extract the data from MDN. I'd suggest to put that script to this repo in the build folder.

[octref]

@aeschli Yes I did, I also tried combining its data with ours to provide better IntelliSense.

Here is showing appearence, which is a property recorded in MDN data but not ours.
Also in properties in both database, I added status and syntax to the end. (Just to show one thing we could do).

There are many things we could do with the data. What are the concrete things you suggest that we do for this month?

Also, I can write a crawler that gets the short description of each page too, so we can actually ship the properties in MDN data but not in our css-schema.xml.

[octref]

Overall:

• There are promising features we could do with MDN data, BUT
• We need more data from MDN before we can switch over
• For now we'll use MDN data together with our data to enhance the existing features.
• During the process we will build the pipeline for digesting MDN data
• Once MDN provides all data we need, we can do the switch over

Current plan:

• Have a data importer that imports data from MDN (might need some hand curation now)
• Find overlapping css properties between VSC and MDN, add MDN's status and syntax to the entry, and show them at the end of CompletionItem's documentation
• ✋ Suggest properties in MDN but not in VSC (mostly new properties). There are no structured data for their documentation but we can write a crawler that gets this info.
• ✋ Use mdn/browser-compat-data to show the up-to-date compatibility for properties, in hover and completion.
• ✋ Understand at-rules and differentiate common properties and at-rule properties in auto completion. We don't have any such logic now but MDN data allows us to do this.

Issues to track:

• Structured data for property / property value short description Use MDN data to power auto completion in editors through Language Server Protocol mdn/data#199
• SVG data: Add SVG data mdn/data#59
• Property Value Validation: Validate syntaxes mdn/data#172 (This could be really helpful to users)

[lahmatiy]

@octref

Property Value Validation: mdn/data#172 (This could be really helpful to users)

This task is about syntax validation rather than a value validation.

You might be interesting to look at a value validation that CSSTree does:

• Syntax explorer shows a structure of syntax and a match to a value
• VS Code plugin (repo) to validate CSS (property values only at the moment) according a syntax

(there are more links in CSSTree repo)

[praveenpuglia]

@octref I had previously worked a little bit with the language server. Really interested in working with MDN data and helping. Let me know how :)

[octref]

@lahmatiy I did take a look of CSSTree, but didn't know about https://csstree.github.io/docs/syntax.html. This is really cool!

This task is about syntax validation rather than a value validation.

Yea, the proper term should be "Syntax validation for CSS property-values".

@praveenpuglia Now a lot of things is still in the air so I don't think it's a good time to jump in. After the major changes go in there should be more specific bugs/features with smaller scopes where you could contribute. Thanks!

[Tyriar]

Related: microsoft/vscode#7008

[xfq]

FWIW, the CSS Indexes produced by W3C contains some properties that are neither in MDN nor in VSC (e.g., text-decoration-width, text-underline-offset, text-emphasis-skip, and corner-shape). However, it contains a few experimental properties and is not available as a JSON file.

[Tyriar]

@xfq if MDN accept contributions on these properties then we will get them.

[lahmatiy]

You might be interested in my work on CSS features dictionaries (properties, functions, pseudo-classes, pseudo-elements etc). I used various sources (mdn/data and other packages), my own research and real sites usage stats. Currently it's just a lists of features (w/o description), but it can be used for simple tasks (e.g. suggestions or validation) and help to estimate the amount of work for fullness dictionaries you do.

• CSS features dictionaries
• Alexa Top 250 sites CSS usage stat

[connorshea]

For "Use mdn/browser-compat-data to show the up-to-date compatibility for properties, in hover and completion.", is there a good way to map information between mdn/data and mdn/browser-compat-data? They use different identifiers and don't link between each other, as far as I can tell.

[octref]

@connorshea I haven't looked into adopting browser-compat-data yet. I have created #102 to track it. Not having same identifiers would probably be one of the issues I'll run into.

If you do have concrete examples of non-matching identifiers, would you mind writing them down and open an issue on mdn/data?

[connorshea]

I did some research and commented in #102 with everything I could find about mapping between mdn/data and mdn/browser-compat-data.

[aeschli]

@octref Can we close this?

[octref]

Let's close this. I'll track further improvements in new issues.