semanticTokensProvider is returned as Null when semantic tokens are unavailable #6

Environment

• Wolfram Mathematica 14.3
• OS: Arch Linux (Kernel 7.0.12-zen1-1-zen)
• Neovim 0.12.3

Issue

When semantic tokens are disabled, or when the client doesn't advertise semantic token support, the initialize response contains:

{
    "capabilities": {
      "semanticTokensProvider": null
    }
}

The value comes from LSPServer/Kernel/LSPServer.wl in lines 1090-1108:

If[$SemanticTokens,
    If[KeyExistsQ[textDocument, "semanticTokens"],
      semanticTokensProviderValue = <|
        "legend" -> <|
          "tokenTypes" -> Keys[$SemanticTokenTypes],
          "tokenModifiers" -> Keys[$SemanticTokenModifiers]
        |>,
        "range" -> False,
        "full" -> <| "delta" -> False |>
      |>
      ,
      (*
      if client does not advertise semantic token support, then do not respond with any support
      *)
      semanticTokensProviderValue = Null
    ];
    ,
    semanticTokensProviderValue = Null
  ];

and later

"semanticTokensProvider" -> semanticTokensProviderValue,

The LSP specification indicates that a missing property should be interpreted as an absence of the capability. The comment in the source also seems to suggest that the intent was not to advertise semantic token support when the client does not request it:

(*
if client does not advertise semantic token support, then do not respond with any support
*)

Effect in Neovim

With Neovim 0.12.3, the language server attaches successfully, but repeated errors are produced from vim.lsp.semantic_tokens:

vim.schedule callback:
/usr/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:437:
/usr/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua:85:
attempt to index field 'semanticTokensProvider' (a userdata value)

LSP works despite the repeated errors.

Local workaround

Replacing the line

"semanticTokensProvider" -> semanticTokensProviderValue

with

Sequence @@ If[
  semanticTokensProviderValue === Null,
  {},
  {"semanticTokensProvider" -> semanticTokensProviderValue}
],

omits the capability when semantic tokens are unavailable. After that change, the Neovim errors disappeared.

I'm not sure whether returning Null here is intentional, but I thought I'd report the behavior in case omitting the capability is more appropriate.