Dlang adds a borrowchecker, called the "OB system" (for ownership-borrowing)

https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1021.md

6 Likes

Interesting! It looks like a prototype with limited additional safety for now, but I like the direction they're going with it. And honestly, having ownership tracking is useful for self-documenting code, even without the no-UB-in-safe-code kind of assurance that Rust has.

I'd be interested to know what anyone who writes or has written D thinks of this! I've personally read a lot about D in articles, but I haven't ever written any and don't know it that well.

4 Likes

I currently use D for some desktop applications, using GTK+ via GtkD. I tend to assume the garbage collector keeps the heap tidy and that the stack takes care of itself. I haven't had any problems with memory as yet.

I am also using Rust + gtk-rs for GTK+ desktop applications as well. I think D is a better programming language than Rust for working with GTK+, but there are some features is gtk-rs that are really very nice that are in GtkD but in a not so nice way. Also the Rust plugin for CLion is excellent whilst the D plugin for CLion is very much a "work in progress" – usable but not of the same functionality as the Rust plugin. Emacs (and I am told VIM and VS Code) support for both languages is roughly the same.

If D had the same traction that Rust has, it would be no contest, I would be using D for all my UI programs. As it is, I am using Rust quite a lot.

3 Likes

I know very little about D. I get the impression it wants to be a fixed and grown up C but without all the accumulated insanity of C++. Which is an attractive proposition.

Anyway, it's good to see that the memory safety ideas that Rust has demonstrated are finding their way into other languages. I have even heard rumor that this is being worked on for the Ada language. And even more vague rumor about such talk in the C++ world.

All of which sounds like a good thing.

I do wonder how well it can work to retrofit borrow checking to an existing language though. I can imagine Ada pulling it off. C++ would of course just bolt it on some how and make the language even more impenetrable.

10 Likes

Not very well. It isn't a first class citizen like in Rust, where it manages it for you. Not to mention D already has a GC, and it was developed with that in mind originally. It doesn't do memory collection for you, in the way Rust does. Along with it's optimizations and thread safety guarantee. It pretty much boils down to just warning that you didn't call free() on a pointer in the scope of a function. Unlike in Rust, the borrow checker isn't tied to safety, so you get these "guarantees" in unsafe code, so there are many holes that can be used to circumvent the borrow checker. Not to mention, a pointer can be either manually allocated or GC. There's nothing stopping you from passing a GC'd pointers into one of these borrow checked functions that will give you a compile error if you don't free() it (which would be an error with a GC'd pointer).

It is inspired by Rust's borrow checker, but it fails in comparison to it. It provides very little guarantees and does not prioritize safety. It is being adding (IMO) simply as a PR stunt as the current implementation stands.

2 Likes

"PR stunt" might be a bit of an overly negative view of things. I prefer to think that D is an on ongoing experiment in exploring language design in a genuine attempt to provide a programmer friendly, easier to use, safer, C++ like language.

How well it is succeeding I have no idea but it sounds like a noble enough goal.

Philosophically one could make an argument that "Perfection is the enemy of the good".

Imagine that 80% of the worlds C/C++ programmers adopted D because it is so much nicer to program in and much quicker and easier to develop with. But it only provides 80% of the memory safety guarantees.

That would make the software landscape much more robust overall than if only 20% of the worlds programmers adopted Rust, the other 80% put off by it's complexity and slow compile times.

That is to say, overall, safety features that are used are better than those which are not.

Bah, well, that is all idle hand waving. I want Rust for practical reasons as well as it's safety, like performance and the ability to run bare metal with no run time.

9 Likes

Moderation note: Please keep conversation respectful and constructive. In particular, we expect everyone here to not only be mindful of each other, but also of the work done by others. Please keep bad faith accusations ("this work is just a PR stunt" for example) out of this forum.

A good way to move forward would be a detailed comparison between D's new borrow checker and what Rust has.

A bad way to move forward would be to turn this thread into a general list of complaints about D. That ground has been well covered elsewhere on the Internet.

11 Likes

That's a bit cliche. No one said it had to be perfect. Rust was specifically designed around the borrow checker, most of it's features are catered to it's limitations. What D is doing is an entangled mess of varying philosophies and ideologies. A garbage collector does what a borrow checker does. D has a means of managing memory that is safe. You go into anecdotal statements, what if 80% this and 20% that. D hasn't been adopted because people don't care about perfection, they don't even care about 80% of perfection. They are perfectly fine and happy with sticking with C++. That statement is exactly part of the reason why C++ survives, people are fine with what they have; they don't need perfection.

That's the biggest problem, is that D's borrow checker isn't practical. It actually introduced more limitations into the language, and because pointers are used by both manual memory management and the GC, it actually is detrimental to those individuals that use the GC where they don't care about or have to worry about memory management. This was the case where array concatenation was going to be deprecated, because an array could be a slice of manually allocated memory. But it would still be treated as if it were GC, so it wouldn't be free'd. The solution that was made up with was to simply remove concatenation altogether. There was a huge backlash and it didn't happen though. But that's the kind of impracticality that is being introduced.

Current you can do something like this:

@live int* test1()
{

    auto p = cast(int*)malloc(4);
    size_t d = cast(size_t)p;

    free(p);

    p = cast(int*)d;
    printf("%d", *p);

    return p;

}

Obviously that is a use after free bug. Part of the problem is that @live isn't used with @safe. And for the most part, you can't use @safe which would disable most of those dangerous feature (like Rust) because it would then also disable malloc() and free().

Basically the @live says I provide some guarantees (very little, almost none). But it can call functions that don't use @live. So those functions don't need to provide any guarantees that @live has it. Effectively that would be like removing the requirement to use "unsafe" in Rust, and you can just call any function you want without worrying about whether it says it provides the same guarantees or not.

So that free() function can actually just be:

void consume(int*) { /* do nothing */ }

Which with this example:

@live int* test1()
{
    auto p = cast(int*)malloc(4);

    consume(p);

    return p; // error main2.test1.p has undefined state and cannot be read
              //       main2.test1.p is returned but is Undefined

}

Now even though consume() is a perfectly valid function and it isn't labelled as @live, it still treats it as though it is following @live's principles. So it assumes that it consumes the pointer, when in reality it doesn't do anything. So you get error messages about the pointer being invalid, as it thinks it is being used after free'd. I don't know how to compare that to Rust, as it is is just so blatantly obviously wrong. It is not what a borrow checker should be or do.

2 Likes

I've used D in a professional context for more than 5 years now (with several years' more experience of using it for personal and research projects), and have also now written a small but significant amount of production code in Rust. I enjoy working with both languages a lot: both have significantly innovated, both have vibrant and fun communities full of interesting people, and both make it easy to write highly performant code.

The short version is that I really welcome this new feature. It's obviously a first step on a longer road, but it's natural that D should start to evolve in the direction of offering much stronger memory-safety guarantees without relying on the GC.

If it continues to evolve in the direction I'm hoping for, it will develop into something quite powerful. One of the most important ways in which D is distinct from Rust is that it tends to be permissive by default in terms of what the programmer can do, but offers extensive opt-in features to lock things down (whether it's enforced memory safety, constness/immutability, functional purity, or whatever). Having the ability to opt in to Rust-like borrow-checks and lifetime guarantees -- and to implement containers and other data structures that take advantage of that -- would be a big advance in terms of the guarantees one could offer in embedded development or other environments where GC may be problematic.

On the other hand the fact that it is opt-in offers a lot of benefits too. One of the things D does really well, which Rust doesn't, is that very fast compile times and permissive-by-default mean that it's very easy to throw together simple things that Just Work without putting a lot of time or effort in. This means (for example) that one can use D as a scripting language just as readily as for building more advanced codebases, but also that a simple codebase can be made more strict and careful at the time when it starts to matter, and not before. So one rarely has to pay a heavier up-front development cost than is strictly needed by the constraints of one's problem.

Another advantage here is that one can readily port an existing C or C++ codebase, in a form close or even identical to the original, and steadily evolve it from the inside out with the stronger constraints that D allows. I'm sure there are parallels for this in Rust, but I doubt they are as easy to achieve.

The flipside of course is that because Rust is restrictive-by-default, and because of the strong emphasis its creators have placed on provable properties, there's much less chance of the different constraints it enforces clashing with one another in unexpected ways. But the tradeoff that involves -- of having to write for a worst-case scenario, when one's actual use case may be much simpler -- is not always a nice one.

So overall, I think this is a promising first step on a road that could in time allow D to offer similarly strong compile-time memory safety guarantees as Rust, but with the interesting alternative of only having to pay their price when you really need or want them.

6 Likes

As Bryan Cantrill says it's all about values. What does one value?

All engineering is about compromise and trade offs. Those design trade offs get made on the basis of what one values, or how one would rank features according to ones values.

Apparently D strongly values a simple to use language and fast build times. Whilst Rust values correctness more highly.

So then the choice comes down to what values prospective users have.

Perhaps, possibly, maybe, with enough development, Rust might converge on D's speed of development turn around and D might converge on Rust's level of correctness checking. It seems like a tall order either way around. We shall see.

Personally I think it's good that language developers/community take a stand on their values and stick to them. Whatever they are. Then there is some chance the language evolves in a coherent manner.

3 Likes

As someone who spent a lot of time pointing out these exact problems when the feature was first described (and has mostly been ignored), it's both validating and frustrating to see someone else point them out.

Overall, I feel the feature so far is a bad compromise, that gets the worst of both worlds. The author sees it as a first step towards getting a Rust-like memory model that is actually airtight, but I'm skeptical the @live feature actually makes implementing such a model much easier.

2 Likes

It's nice to see Rust's concepts getting borrowed by other languages. For now, it looks like it's in quite the early stages. I expect retrofitting ownership into a language not designed with it in mind to be quite an endeavor.

Let's hope the diverted effort isn't going to cause D to skip leg day :stuck_out_tongue:

3 Likes

That's funny, if you know what it means, but I imagine it's very confusing without the context.

9 Likes

It's interesting to look at Andrei's "10X advantages":

D:
10x faster to compile
10x faster than scripting languages for comparable convenience.
10x easier to interface with C and C++
10x better than any other system language at generic and generative programming

Go:
10x better strategy.
10x engineering.
10x branding.

Rust:
10x better theorists.
10x better safety than other systems programming languages.
10x better PR

Interestingly almost none of the 10X features are anything to do with the actual language or technical merits. I marked those in bold above.

Go, scores none on the merits of the language itself. D has generics. Rust has safety.

One can apply ones values to this. In my case safety is a high value item. Generics not so much, in many decades I have produced almost no generic code.

How do your values fit the list?

That is not to say Andrei is wrong in devising that "bullet" list. Obviously the success or otherwise of a language is dependent on all manner of non-technical circumstances. Likely non-technical features have more influence than the technical. Look how well Ada fared for example.

1 Like

[Mod note: Let's keep this topic focused on D’s OB system.]

2 Likes

I don't think that's really a fair or accurate reflection of what I said or the values of the D community and development team. D has plentiful correctness checks, including some (like functional purity) that Rust doesn't have. The major philosophical difference here is that Rust imposes its checks by default whereas D leaves it to the developer to decide which checks are important for their use case.

Arguably D has a pretty good record of integrating innovations from a variety of different languages and paradigms, and making them work in a useful and coherent manner. This new move towards tracking ownership and borrowing doesn't seem particularly different from many other historical evolutions of the language.

To bring things back to the feature itself, it seems unwise to critique too heavily an early, highly provisional feature that is obviously just one component of an ongoing set of memory-safety developments.

For example the use-after-free situation described above seems a bit contrived given that it arises out of doing two distinctly unsafe actions in the same function -- both malloc and casts from pointer types to value types and back. But it's trivial to get a @safe check working here:

@live int* test1() @safe
{
    import core.stdc.stdlib : free, malloc;
    import std.stdio : writefln;

    // minor note here: we have to create a lambda for this
    // because @trusted can only be applied to D functions,
    // not code blocks
    auto p = () @trusted { return cast(int*) malloc(4); }();
    size_t d = cast(size_t) p;

    () @ trusted { free(p); }();

    p = cast(int*) d;
    writefln!"%d"(*p);

    return p;
}

... which immediately catches the cast(size_t) as unsafe before we even get to the free-after-use. That's probably much closer to what a real long-term use-case would be, where there would be some kind of safe allocator, or where the actual allocation would be hidden under the hood of a container (because while now the @live feature just applies to pointers, it's clearly intended that it be extended to more general reference-containing data structures over time).

I can't see how Rust would catch such a scenario either if explicit mallocs, casts, and frees were being done all together inside an unsafe function. Rust borrow- and lifetime-checking succeeds because the unsafe bits (the actual mallocs and frees) are wrapped away.

2 Likes

BTW, it's worth noting (when talking about how @live is less effective without @safe) that D is going to move to @safe being the default (i.e. memory safety checks being performed by default on all code unless specifically tagged with @trusted or @system).

1 Like

Exactly what I said then "Whilst Rust values correctness more highly." It shows that value judgement by taking away the decision from the developer.

As "Uncle Bob Martin" recently pointed out in an opening presentation somewhere high level languages remove choices from the programmer so as to try and ensure they write more robust code. For example the move from assembler to the likes of C/Pascal/PL/M/etc enforced "structured programming" on the developer. No more random control flow. Type systems remove choices. Later goto stopped appearing in languages, removing that possibility. Purely functional languages remove the possibility of mutation.

Anyway, we are on the same page. I think it's great that D adopts Rust style lifetime checking. I'm sure it is early days yet.

1 Like

We're going to go off-topic with this again :wink: The problem I have with this description is that "correctness" is a broader topic than Rust's compiler checks cover. If a language designer's definition of "correct" forces a developer to wrestle with solving problems that actually don't apply to their use-case, it may well distract from correctness of the code, because it focuses their attention on solving problems they don't have, as opposed to the problems that they do.

I honestly think that Rust's strictness is less a value judgement than a combination of 2 things:

  • Rust is designed first and foremost to support a particular use-case (low-level multithreaded systems programming) where these correctness concerns do apply

  • having the language be restrictive by default makes it easier for the language design theorists to work

Neither is actually about values, but rather the problems that people are trying to solve.

Where there is a value, I think, is the strong wish for theoretically coherent language features. But even that really derives from the core use-case.

Well, yes and no. For example Python removes the choice to interact with low-level control mechanisms over data structures, memory usage, etc., but also adds choices that lower level languages don't offer (e.g. dynamic typing, a much more dynamic interplay between code and data, and so on). It's more robust for a certain definition of robust, but not necessarily by others.

Agreed. And thanks for the fun discussion along the way.

Moderation note: Folks, we've asked y'all twice now to keep this thread on topic. Please resist the urge to continue a general language comparison and keep this about D's borrow checker.

2 Likes