Skip to content

Latest commit

 

History

History
 
 

syntax-highlighter

README.md

Syntax Highlighter (And Associated Crates)

Crates:

  • The main syntect_server executable
  • crates/scip-treesitter-languages/: All the grammars and parsers live here to make shipping parsers with the same tree-sitter version (and associated build tooling) very easy. All new grammars and parsers should be added here.
  • crates/scip-treesitter/: Associated utilities for tree-sitter and scip. Not required to be used for other projects
  • crates/scip-syntax/: local navigation calculation (and some other utilities) live here.
  • crates/sg-syntax/: Sourcegraph code to glue together whatever from the above crates to be used for our purposes.

scip-ctags

See queries

Syntect Server

This is an HTTP server that exposes the Rust Syntect syntax highlighting library for use by other services. Send it some code, and it'll send you syntax-highlighted code in response. This service is horizontally scalable, but please give #21942 and #32359 a read before scaling it up.

Cargo Usage

cargo run --bin syntect_server

You can set the SRC_SYNTECT_SERVER environment var to whatever port this connects to and test against local Sourcegraph instance.

Docker Usage (can be used with sg start)

docker run --detach --name=syntax-highlighter -p 9238:9238 sourcegraph/syntax-highlighter

You can then e.g. GET http://localhost:9238/health or http://host.docker.internal:9238/health to confirm it is working.

API

See API

Configuration

By default on startup, syntect_server will list all features (themes + file types) it supports. This can be disabled by setting QUIET=true in the environment.

Development

  1. Use cargo test --workspace to run all the tests. To update snapshots, run cargo insta review.
  2. Use cargo run --bin syntect_server to run the server locally.
  3. You can change the SRC_SYNTECT_SERVER option in your sg.config.yaml to point to whatever port you're running on (usually 8000) and test against that without building the docker image.

Building docker image

./build.sh will build your current repository checkout into a final Docker image. You DO NOT need to do this when you push to get it publish. But, you should do this to make sure that it is possible to build the image 😄.

AGAIN NOTE: The docker image will be published automatically via CI.

Updating Sourcegraph

Once published, the image version will need to be updated in the following locations to make Sourcegraph use it:

Additionally, it's worth doing a search for other uses in case this list is stale.

Adding themes

TODO: Maybe we can just remove themes entirely. I think they are u

  • Copy a .tmTheme file anywhere under ./syntect/testdata (make a new dir if needed) in our fork.
  • cd syntect && make assets
  • In this repo, cargo update -p syntect.
  • Build a new binary.

Adding languages (tree-sitter):

See scip-treesitter-languages

Adding languages (syntect -- outdated):

1) Find an open-source .tmLanguage or .sublime-syntax file and send a PR to our package registry

https://github.com/sourcegraph/Packages is the package registry we use which holds all of the syntax definitions we use in syntect_server and Sourcegraph. Send a PR there by following these steps

2) Update our temporary fork of syntect

We use a temporary fork of syntect as a hack to get our Packages registry into the binary. Update it by creating a PR with two commits like:

3) Update syntect_server to use the new version of syntect

Run the following in this directory.

$ cargo update -p syntect

Supported languages:

Run: cargo run --bin syntect_server to see supported languages and themes.