Technical notes › Browse the code of locally-cached crates
Rust’s package manager and build system, Cargo, downloads the entirety of each crate used as a project’s dependency and stores it locally.
When I need to read the crate’s documentation, I’m usually both lazy and online, so I just use the superb docs.rs service, which builds and indexes each published crate’s documentation automatically.
If I happen to not have internet access, I can still build a copy of the docs locally with
cargo doc --open, which very handily opens my Web browser to read it.
Sometimes, though, I’m not interested in the public documentation of each module, I need to view the code directly.
And that code is stored locally, after
cargo fetch has been run — it just needs to be extracted from its archive.
The browse-crate function
To make it easier to browse a crate’s code, I wrote this Fish function:
function browse-crate -d "Browse a locally-cached Rust crate" -a crate_name if test -z "$crate_name" echo "browse-crate: No crate name given" >&2 return 3 end mkdir -p ~/tmp/crates cd ~/tmp/crates; or exit $status set -l base_crate_name (string replace -r 'crate$' '' $crate_name) set -l crates_dir ~/.cargo/registry/cache/github.com-*/ if test -e "$base_crate_name" echo "browse-crate: ‘$base_crate_name’ is already extracted" >&2 else if test -f "$crates_dir/$crate_name" echo "Extracting ‘$base_crate_name’..." tar -xf $crates_dir/$crate_name else echo "browse-crate: crate at ‘$crates_dir/$crate_name’ does not exist" >&2 return 1 end cd $base_crate_name pwd end
This function takes one argument — the name of the crate archive file to extract, such as
anyhow-1.0.59.crate — and creates a directory within
~/tmp/crates (or wherever you deem best to put temporary files such as these), and then extracts the crate into it.
cd-s into that directory, leaving you free to search the code, or open it in your editor, or similar.
The actual directory we are searching contains both
github.com, the name of the default Cargo registry, and a random hash, so we have to glob the file name in order to find it.
There should be only one such directory present.
If you’re following the advice in “Unclutter $HOME with wrapper scripts” and are setting the
CARGO_HOME environment variable, you’ll have to re-write the glob from searching
That’s the price of a clean home directory: constant vigilance.
So far so good. There’s just one problem: we need to know not just the name of the crate to extract, but its version, which will vary from project to project. Almost all the time, I’m using the most recent version of a dependency, but this is no means a guarantee, and I’d still like to be able to specify the version two or three updates behind if necessary.
The best way I’ve found to handle all possibilities is to write a custom completions function for the script that sorts in reverse version order: one that will offer the completion for the latest version of the crate first, the second-latest version second, and so on. This means that I get the behaviour I usually want first — all I have to do is type the first few letters of the crate’s name and hit Tab — while still letting me choose from older versions if I happen to be lagging behind.
There are two challenges with this approach. The first challenge is that even though we want the versions of each crate to be sorted in reverse order, we want the names of the crate to be sorted in non-reverse alphabetical order. For example, the first few crates that I happen to have cached locally are these:
addr2line-0.14.1.crate addr2line-0.16.0.crate addr2line-0.17.0.crate aho-corasick-0.7.15.crate aho-corasick-0.7.18.crate
a is on the command-line buffer, then the completions script should, ideally, offer the third line as the first completion, as it’s the latest version of the crate that comes first alphabetically.
Next would be the second line, and then the first line — and then the fifth, and lastly the fourth.
The second challenge is that we need to extract the version number from each line: the crate name and version are separated by a hyphen, but the crate name could contain hyphens itself, so it’s everything after the final hyphen.
Is such a thing even possible without resorting to a proper programming language?
Believe it or not, it is, though you have to write a crazy train of shell pipelines, and use a feature of
sort only present in GNU coreutils, in order to do it correctly.
This is what I came up with, placed in my Fish completions directory:
function __complete_crates command ls -1 ~/.cargo/registry/cache/github.com-*/ \ | rev | sed 's/-/#/' | rev | gsort -t'#' -k'1,1' -k'2,2rV' | sed 's/#/-/' end complete --command browse-crate --no-files complete --command browse-crate --arguments "(__complete_crates)" --keep-order
Let’s go through this, step-by-step.
Firstly, we print all the file names in the cargo cache directory, one file name per line, without printing the directory the files are in. It’s usually frowned upon to use ls to print the names of arbitrary files, since the files’ names could contain newlines themselves; in this case, the crate archives follow a known format, so this is safe.
Next, we need to transform each file name so that the version part of the name can be more easily plucked out, replacing the final hyphen with a different character that can be scanned for specifically; here,
#. This is the
rev | sed 's/-/#/' | revconstruct: as there is no straightforward way of replacing the final hyphen on the line, we reverse each string, replace the first hyphen, and then reverse it back again. This transformation turns the list of files into the following:
addr2line#0.14.1.crate addr2line#0.16.0.crate addr2line#0.17.0.crate aho-corasick#0.7.15.crate aho-corasick#0.7.18.crate
With this in place, we can sort each line using the
sortbinary that comes with GNU coreutils. This may be already installed as the
sortexecutable on your system; on macOS, though, we default to BSD
sort, so I’ve had to install the GNU coreutils through Homebrew, which calls it
gsortto distinguish it from the system default. We need GNU sort because, astoundingly, it comes with a version sort feature!
gsort -t'#' -k'1,1' -k'2,2rV', uses two rarely-used features: the field separator,
-t, and sorting by keys,
-k. Remember how we need to sort the version numbers in reverse, but sort the names non-reverse? We do this first by passing
#as the field separator, then telling
sortto first sort by field #1 (
-k'1,1') and secondarily sort by field #2 using reverse version sort (
rmeaning “reverse”, the
Finally, our strings still have
#signs in, so we need to turn them back to hyphens.
This is shell scripting in a nutshell.
It’s ordinary programming turned on its head: what’s usually hard is easy, but what’s usually easy is hard.
Sorting strings as version numbers in an ordinary programming langauge would either require writing a complex comparison function or using an external dependency, but for GNU
sort, all it took was adding one extra character.
On the other hand, trying to split each filename on the last hyphen rather than the first hyphen would usually be a routine operation — and in shell script land, we not only have to pipe through
rev twice, but
sed twice, too!∎