shiki-net

GitHub

shiki's JavaScript, TypeScript, JSX and TSX tokenizers as handwired neural networks, on your GPU through WebGPU. The page runs the network and shiki on the same code and compares their codeToTokens output. The full version is identical to shiki. The lite version is faster on long lines, and its tokens can differ from shiki's in rare cases.

sample.js

Loading…

Network

Handwired neural network, on the GPU

Total time codeToTokens, without render —

Parts of the total

Stage A: the regex network
—
Stage B: the driver network
—
Host: stitch the lines
—
Decode to tokens
—

Not in the total: one-time setup of weights and shaders—

shiki

Oniguruma regex engine (WASM), on the CPU

Total time codeToTokens, without render —

Parts of the total

One step: shiki's codeToTokens, without its time limit of 500 ms per line, in the main thread of this page. It runs after the network, so the two do not share the machine. The code above shows the network’s tokens.