Please check your docs: `hoedown` -> `pulldown-cmark`

announcements
1

Hey all,

We've intended to transition rustdoc away from hoedown, which is written in C, to pulldown-cmark, which is written in Rust, for a long time. Recently, the change finally landed. :confetti_ball: This knocks out another C dependency of the Rust toolchain.

Another thing it does, though, is introduce some changes in markdown parsing: if you've followed Markdown at all, you've seen that there are tons of different implementations that do things slightly differently. Out of this mess, CommonMark was formed, and is attempting to standardize Markdown. I've wanted Rustdoc to use CommonMark for a long time, and now, with pulldown-cmark, we will be! In related news, GitHub is also transitioning their own GitHub Flavored Markdown to be CommonMark based, so this would also let us be 1-1 with GitHub's rendering.

So, there's no good way to run tests of a change like this across the ecosystem. As such, we're asking you to please try out the latest nightly, which contains this difference, and see if everything looks okay. There are a few known issues that are currently being worked on, and we're tracking them here:

https://github.com/rust-lang/rust/issues/40912

There's also a PR for two out of the three biggest ones:

https://github.com/rust-lang/rust/pull/40919

Please file any bugs you find so that we can fix them up; this change will land in Rust 1.18, June 8. You may also need to adjust your markdown depending; commonmark.js demo is a great tool for checking out exactly how CommonMark interprets your Markdown.

Thank you!

EDIT: @nikomatsakis and I were chatting and realized that @brson's cargobomb may be able t detect some things crater couldn't; I'll be watching those closely.

21 Likes
2

Do we know when these changes will appear on docs.rs? Specifically, will they re-render already existing docs or will the docs be updated as crates are updated? This will change my timeline from "Oh, I'll check my docs before my next push" to "shoot, I need to check my docs tonight"

3

@onur would be the person who'd know that; I believe that docs.rs uses stable rustdoc, and generates docs whenever you push new crate versions. I don't believe older docs get re-rendered.

4

Like @steveklabnik said, docs.rs is building crates only once when they are released. This change won't affect the old version of crates.

I will update rustc on docs.rs when this transition becomes more mature.

6 Likes
5

Thanks!

6

The only thing I'm missing are the ---- rulers, so it looks pretty good.

1 Like
7

Is there a tracking issue for non-Rust dependencies of the Rust toolchain? I'm curious about what other work needs to be done in that area.

8

There isn't. However, the big ones are:

  • LLVM (of course; not likely to be replaced any time soon)
  • jemalloc
  • libbacktrace
  • miniz
1 Like
9

There's also compiler-rt and of course libc used by rustc. If you expand the Rust toolchain to include cargo, then this adds libgit2, openssl, and winapi, varying between platforms.

3 Likes
10

It should be fixed in the fix PR as well.

11

An update: https://github.com/rust-lang/rust/pull/40919 was merged, which knocked out some of the bigger issues.

We've only seen a few situations where this has caused problems, mostly around markdown that looks like this:

# title
```
   invalid rust code
```

hoedown interpreted this as not-code, and so would not run it as a test. pulldown-cmark interprets this as rust code, and so tries to make this a test, and fails. As usual, adding ignore to a code block makes it easy to ignore things like this.

On the "visual" side of this, stuff like this:

2^5

would render as

25

previously, but now will just be 2^5. Use <sup></sup> tags if you want the prettier rendering.

To help test with this, rustup update nightly-2017-03-28 is the last nightly before the change, and it looks like the most recent nightly has not quite gotten the new fix; tomorrow's nightly should have it.

1 Like
12

A small update here: even though there was little chatter here, cargobomb showed a number of crates whose tests were failing due to this change. As such,

https://github.com/rust-lang/rust/pull/41290

was landed, which brings hoedown back in tree, for the purposes of checking your tests. Instead of producing an error, discrepancies will be reported as a warning instead. Someday in the future once crates have migrated, we'll remove hoedown again.

Thanks!

1 Like
13

See also this thread on internals, which is discussing how best to phase in this change.