Introducing Colored: the most simple way to add colors in your term!

announcements
21

(I'm the author of term-painter)

@mackwic: have you seen this thread? It suggest using CLICOLOR and CLICOLOR_FORCED variables; apparently CMake use those variables, too. I would use them instead of Rust specific ones. What do you think?

@starblue:

Since you shouldn't put essential information in color (there are color-blind people who can't see it) that shouldn't be a big loss, just a loss in emphasis on some text.

I have to disagree. As a human, parsing text can be done much quicker when relevant parts are emphasized. Doing this without colors is sadly somewhat limited. This difference in readability can be quite notable and shouldn't be ignored. Yes, there are colorblind people, but most of them (like myself) "just" have problems with a few colors. The inability to see any color at all is very rare. And I have no problem with adjusting my color scheme to avoid using some colors that my eyes don't like.

Ability to disable colors easily: Absolutely. Off by default/colors should be avoided: No.

Regarding the Win-support discussion:

Supporting different operating systems is hard, but I (!!) think that only supporting one (ansi_term) or having different APIs (like @mackwic suggested) is not the right way to go. With my library I chose the "expose a maybe-surprising API and tell the users about it"-philosophy. But I am certainly not perfectly happy with that!

I tried to summarize the discussion:

  1. Using println! can lead to strange behaviour, because the time of Displaying is not defined. Everything that can be used with println! can be used with format! and the Display impl can't tell the difference. Every solution that works with println! and alike will not be able to fully work on Windows (again, because of format!). Furthermore such a solution would not be able to warn it's user about the usage of format! -- the user would get confusing results.
  2. Controlling the output to stdout within the painting library would resolve the issue above. But even if the library would define a macro (say coloredln!) to make printing easy, using println! is still the natural choice for Rust programmers. Furthermore there are many macros and functionality that is build around println!. With solution (1.) it would be possible to use those macros as well, which is a big advantage IMO.
  3. Proxying stdout and do stuff right before printing, like @DanielKeep mentioned. While this sounds crazy at first, I think it's a fairly good idea. It solves both problems above, at least.

I am really interested in developing a solution better than everything we have so far!

EDIT: I also want to add, that solution (1.) (and hence my crate term-painter) also depends on the exact implementation of println! -- in particular the order of operations. This is not good.

2 Likes
22

This is great. I like it a lot. Relevant link specifying the behavior: Standard for ANSI Colors in Terminals
I will update my issue accordingly.

My ruby programmer heart is telling me to give the most simple api which will work for 80% of use cases, and let the power-programmers which do awesome libs using the power-api. It's only a strong feeling, but it's still a strong feeling. :wink:

Regarding your points:

  1. agree, but I also think we don't have to go too far to fix the user environment in worst cases. There is good terminal emulators for windows like Cmder. Not everyone has the control of the installed softwares, right, but I would ask what use do they have of reading the terminal output, then.
  2. I don't know. If the macro writers want to paint, they will use our libs, if they don't, they don't care anyway, right ? That's why I think the special-purpose macro was a good middle ground. The one that lost it don't want it anyway.
  3. Acting at the stdout buffering level seems indeed the most elegant solution, the most general, the most portable, and the least intrusive. Bonus point if it can be deactivated by the user respecting the env variables CLICOLOR. Thing is, I see a dozen of reason for the rust team to dislike this change: it's too late for the rust stdlib at this point to add this behavior, or it's unwanted, or too much of a burden to support it for all platforms and termina combinations, it's a leaky abstraction, etc. @brson can we have an advice here ?

Whatever the outcome of this discussion, I am very happy to have it. Thanks for coming with new remarks and ideas.

23

Well, it would not be as intrusive as a change.

It's a matter of changing this line: https://github.com/rust-lang/rust/blob/master/src/libstd/io/stdio.rs#L392

392   Arc::new(ReentrantMutex::new(RefCell::new(LineWriter::new(stdout))))

By something in the like of:

392   Arc::new(ReentrantMutex::new(RefCell::new(LinePainter::new(LineWriter::new(stdout)))))
                                                ^^^^^^^^^^^^^^^^

LinePainter impl-ing the standard Write trait and applying the platform-local, terminal-local, program-local (tty + env) painting configuration from a predefined output for painting (like... ansi escapes). This should not be a breaking change. I think.

EDIT: maybe making stdlib dependent of libterm is. There is a libterm in rust/src/ but I don't know how it is used.

24

I would be quite happy with that, especially if it develops into a standard. BTW, a quick search also found that BSD ls is using CLICOLOR.

25

FYI I just published Colored 1.1.0 which respect the CLICOLOR behavior.

26

I'm the author of ansi_term and I'd definitely support adding an environment variable that automatically turns the colours on and off. Are we settled on CLICOLOR?

It does annoy me that ansi_term (and therefore exa) don't support Windows terminals, but as noted in this thread, it's not a simple fix. ansi_term was designed to make coloured strings Display-able, and as you can't pass any extra arguments when Display-ing something, the terminal checking has to be done elsewhere. It looks like we have three options:

  1. Separate the idea of "building a string with styles and colours" from "displaying that string in a terminal" by introducing a new type or trait that displays them. This would use ANSI codes or Windows system calls depending on what's available.
  2. Store the terminal type within the coloured string itself, and use that information to print it.
  3. Just go with ANSI codes and print and parse them at runtime (as @DanielKeep suggests).

Option 1 seems like the 'best' solution from a technical perspective, but it ruins the user-friendliness of the library; Options 2 and 3 keep it, but at a cost.

All of our libraries allow you to do something like:

println!("{}", "This message is blue".blue());

which is nice and user-friendly but assumes that you can change the text colour by printing. Having to construct a terminal value (option 1) would change this to:

let term = Term::get_term_somehow();
term.write("This message is blue".blue());

and now you have another value that needs to be passed around the place. Storing the terminal info in the string (option 2) would simplify it back to

"This message is still blue".blue().display_somehow();

where the display_somehow call would check the terminal type, cache it, and use the correct codes each time (either printing to stdout along with the message, or using the Windows system calls before and after the printing). In both cases, we lose the ability to use println or write or format or similar.

Anyway, this is just me thinking out loud. I agree that it needs some work, because printing in colour seems like something that simple to do correctly, even when it isn't!

27

Wonderful ! Everyone is in the discussion. :slight_smile: Consider CLICOLOR to be definitive. CMake implement it, soon clang will, it's the way to do.

Nice summary of the situation, @ogham . I personally thank that printing to stdout should not be optimized too much as it will be a costly call, whatever you try to do.

So I think the best way to do these things would be option 3:

  • all libs emits ansi escapes. This is kind of the color templating @brson wanted. It's also easy to emit and easy to parse, and standardized, and zero-cost to print on all ansi terms. So why not ? It's ideal.
  • stdout is not really stdout, but a Writer which parse ansi escapes and intrument terminal if isatty()

The more I think of it, the more I like this idea. I want to try it with a prototype.

I don't know where you are on you parser @DanielKeep , but I started mine as an experiment, and it seems it's almost finished: GitHub - mackwic/ansi-escapes: (Rust) ansi escape parser
As soon as the parser is finished, I will make a crate TextPainter which will define println!-like macros, and will strip the escapes if needed (CLICOLOR=0 or !isatty() or...), or instrument the term as needed using the term crate.

It's only all prototypes for now. But would like the stdlib stdout to have this behavior, so my end goal would be a rust RFC. What do you think of it ?

28

Mine's at https://github.com/DanielKeep/rust-ansi-interpreter; the ANSI parser itself is in src/ansi.rs. The major difference I see off the bat is that mine's designed to be inserted into a stream; i.e. it handles the case where an escape sequence has only partially been written through the stream (it fell on a buffer boundary, or is simply being constructed piecemeal), and it uses a trait callback for handling the sequences. Also, I think mine might be slightly more complete than yours.

Aside: the reason I went with callbacks over returning an enum is that the first thing any non-trivial implementation is going to do on getting an enum is... match it to decide what to do. I decided to just cut out the middle man and invoke the appropriate method directly.

Also, I don't know that this is appropriate for std. It involves a level of messing with the IO system that I'm not really comfortable happening automatically behind people's backs. I mean, what if you want to output binary data to stdout?

29

Those are some pretty 'big names' to get behind an environment variable! If that doesn't swing it, nothing will. One thing, though, @mackwic:

Are you thinking about the cost on the computer, or on the programmer? Writing to a stream is going to be expensive, but with wrappers like println it's still easy to do. Having to pass round a terminal value would be one more thing to think about, even if it's low-overhead compared to just writing the ANSI codes manually.

30

Thanks Daniel. I will look at it carefully. There is a lot of good ideas and good code there, especially the reentrant parser, the windows stdout hijacking. You didn't provide any licence, though. I can't steal anything from it (oh the sweeeet utf-8 test strings), nor use it as a reference, is it on purpose ?

Seems sensible. Maybe I will do that.

I think, the mantra here should be "sane defaults for everyone, but let the power programmer tweak the system". You rarely want to output binary data on stdout, but on a pipe, which should disable the ansi parsing anyway.

Here is how I envision my hypothetical TextPainter:

// normal use
tprintln!("hello world".red()); // colored "hello world"

TextPainter::set_mode(PaintMode::RemoveEscapes);
tprintln!("hello world".red()); // plain text "hello world"

TextPainter::set_mode(PaintMode::PlainEscapes);
tprintln!("hello world".red()); // ugly "^Esc[42mhello world^Esc[0m"

TextPainter::set_mode(PaintMode::Default)
tprintln!("hello world".red()); // colored "hello world"

There is a nice decoupling between the painting and the writing throw the ANSI escapes codes. We can also have TextPainter respect some runtime configuration via env variables. Ideally, it would use printf and be bundled into stdout.
Providing a standard stdout, which act nicely cross-platform would be a great feature for programmers I think.

31

I'd definitely support adding an environment variable that automatically turns the colours on and off. Are we settled on CLICOLOR?

Apparently :wink:
Will add it to my crate soon. But I'll make it a cargo-feature (which is enabled by default) in case some people want to disable it.

So I think the best way to do these things would be option 3:

  • all libs emits ansi escapes.

I don't think we need to settle for the one and only solution here. There is a reason there are three libraries and I think that's OK, isn't it? I definitely want to talk about everything, but I don't think we have to agree on something in the end :stuck_out_tongue:

I think that the newest crate colored may be a good candidate for using the solution we will come up with. ansi_term somehow already implies it's for ansi-color-terminals with it's name. And I myself have some kind of working solution already (see below). But I'm open for everything here!

Storing the terminal info in the string (option 2) would simplify it back to

"This message is still blue".blue().display_somehow()

where the display_somehow call would check the terminal type, cache it, and use the correct codes each time (either printing to stdout along with the message, or using the Windows system calls before and after the printing). In both cases, we lose the ability to use println or write or format or similar.

Why would we lose that ability? My crate works a bit like that. The string "This message is still blue" is wrapped into another type, that in turn implements Display. In that impl I call appropriate functions of the term crate to do the right thing depending on the platform (it does work with Windows). The call of Display::fmt corresponds to your display_somehow(). It just doesn't work for format! and still has some small issues... but it does work in general!

About the plug-before-stdout-hack: if it's used (which I think is a great idea), it should be definitely possible to disable it (globally for just temporarily for outputting binary data for example).

32

I thought the point of the ansi-parser would be to be able to use the standard println! and friends macros. This would be achieved as @DanielKeep explained it:

I have a WIP ANSI interpreter for Windows that works by replacing the default console handles before std acquires them.

33

Computer cost. In the end, it's a context switch to kernel-land, acquering a mutex,
copying buffers, coming back to user land. Depending on the OSes, the codepath could go through IO drivers (would it be SSD, network mounted drive, or simply the graphic card). All this is very costly. We should not be affraid of doing a little string manipulation before printing, here. well, that's my stance anyway.

Agreed that it should be disabled by user or programmer will.

34

I thought the point of the ansi-parser would be to be able to use the standard println! and friends macros. This would be achieved as @DanielKeep explained it:

I have a WIP ANSI interpreter for Windows that works by replacing the default console handles before std acquires them.

As long the default handle can be replaced, yes. I was not aware of this possibility and will study Daniel's code and test a bit. This approach is promising.

35

Fact of the day:

At one point, when Rust was still built on top of libuv, it would automatically translate ansi escape codes into the appropriate Windows API calls.

36

Why did it change ?

37

Because we got rid of libuv and libgreen and did all IO ourselves natively.

38

this one was pretty obvious. I was asking why the regression ?

39

It's not a regression. It'd be pretty strange behavior for a systems language to mess with bytes written to stdout.

2 Likes
40

I strongly (but respectfully) disagree with your choice of words. I agree with you that a system language should enable the programmer to control and tickle every bit of the system as he wish. It doesn't imply that the language shouldn't have sane defaults which abstract the cross platform specifics. Especially when we speak about common boilerplate which add no value whatsoever.

I think that stdout should be able to react to ansi codes accordingly, but "should be disabled by user or programmer will".

1 Like