Lady Deirdre 2: Framework for Language Compilers and LSP Servers

announcements
1

Greetings!

I would like to share with you my project, Lady Deirdre.

Developing a standalone language compiler and interpreter in Rust is not as difficult as it may seem. There are many excellent tools in the Rust ecosystem for this purpose (e.g., nom or chumsky), and creating such a program manually is also an option. However, building a comprehensive code editor extension with features like code completion, jump-to-symbol definitions/usages, inlay type hints, and other functionalities that require a deep understanding of incomplete and dynamically evolving source code is a challenging problem. Programming language authors usually leave this task for a later stage, making it nearly impossible to build such a code analyzer on top of an existing compiler architecture. Lady Deirdre addresses this issue by helping you create a hybrid application with a unified architecture capable of acting as both a language compiler/interpreter and a code editor extension (LSP server) from day one.

Lady Deirdre is an infrastructure framework designed for authors of new (primarily scripting) programming languages. It provides a toolset to create a lexical scanner, syntax parser, and semantic analyzer for source code files. The framework creates in-memory representations of files, including their text, lexis, syntax, and semantics, and incrementally updates this metadata when the end-user changes the codebase in real-time. This process is error-resistant and specifically designed to infer as much semantic metadata as possible from an incomplete source code state at each point in the code's evolution.

The framework includes several other useful components that you may need to create a full-featured compiler infrastructure and a language server, such as components to develop source code formatters and to organize multi-threaded concurrent programs capable of managing simultaneous semantic analysis tasks.

In principle, a similar infrastructure can be built using existing tools from the Rust ecosystem. For example, Rowan, Tree-Sitter, Ropey, and Salsa together can provide a comparable toolset. However, Lady Deirdre aims to offer a unified framework API that guides you through the steps of programming language development.

I benchmarked individual features of Lady Deirdre against some well-known tools from the Rust ecosystem. My framework demonstrated better computational performance characteristics than Tree-Sitter in both non-incremental and incremental modes on files of average size (~2000 lines of code). In some other tests, more specialized libraries outperformed Lady Deirdre. But overall, Lady Deirdre provides fine computational performance, at least based on my benchmark tests.

I'll be glad to answer any questions.

Ilya

2 Likes
2

People should probably be aware of the license going in. In particular, it seems that any language tooling created with this would be considered a derivative work by my reading of the license text, which effectively grants the ownership completely, to the point that you might not even be allowed to distribute the code you write without permission.

I don't want to be too harsh to people trying to provide a product, but this is a scary as heck license, certainly not something I would use.

5 Likes
3

Thank you for your reply, Simon.

I'm not a lawyer, so I can't provide a comprehensive assessment of the legal aspects of project distribution. However, I would greatly appreciate any suggestions and recommendations you have for improving the project. My goal is to create open and helpful tools for everyone and to share knowledge.

Thank you again for your insights!

4

A proprietary licence is in direct conflict with this; a proprietary licence says that you want money in preference to sharing knowledge.

You might want to use https://choosealicense.com/ - in particular, Licenses | Choose a License - to see if there's a pre-existing lawyer-tested licence that suits your goals.

Otherwise, software licensing is a minefield, and you need a lawyer involved.

1 Like
5

While I understand the concerns about non-FOSS licenses, this is the best option I can offer under the current circumstances.

Allow me to explain the situation. From the beginning, Lady Deirdre has been part of a larger initiative that is technically incompatible with open-source. To benefit the community, I decided to make a portion of my works available to the public under the least restrictive terms I could manage. The alternative would be to keep the codebase fully enclosed and not available for public use.

The source code and related materials are now fully disclosed on GitHub and available to everyone. Even though there may be concerns regarding commercial use, I believe that the current form of publication can at least benefit those who want to use Lady Deirdre for educational and research purposes. In this sense, it is a share of knowledge, and it is free.

I have a history of contributing to open-source projects, and I would love to make Lady Deirdre part of the Rust open-source ecosystem. But due to the current circumstances, this cannot be fully implemented at this time.

Thank you for your understanding.

4 Likes
6

It's not just that it's non-FOSS, although that doesn't help.

It's that there's a lot of language in there that matches EULAs I've run past lawyers before and had them say "don't go near that code - the licence text is sufficiently confused that I cannot give you a clear answer on whether you're breaking the EULA or not".

Given that you can't just switch to a FOSS licence, I'd really recommend getting a lawyer to go over what you want the EULA to do, and ask them to help you make the licence clear. I see issues with the "Derivative Works" section, the "Grant of the Commercial-Limited License" section (notably, what's the exchange rate if I cannot legally convert my local currency to USD? You do not want me arguing that it's USD 0.00 per local currency unit), and the "Unrestricted Commercial License Rights and Obligations" section (where I think you simply say far too much).

On top of that, you don't have a clear indication of jurisdiction, and as written, you have several clauses where the law in some jurisdictions says that the entire licence is replaced with "it's public domain" if you attempt to make the claims you're making (even in the presence of a separability clause); a jurisdiction clause would avoid this because it would make clear that the licence is intended to be interpreted according to the rules of a jurisdiction without such laws.

That's just my non-lawyer read over, but I'm confident that if you took your desires to a lawyer, they'd be able to find other issues with the text where what you've got won't cause the effects you want them to cause, and might even backfire on you.

2 Likes
7

Note that this is functionally useless without a license that lets me use it in other works. Indeed, it's arguably worse than useless because if I ever look at it then unwittingly and unintentionally create a derivative work, I might still be in violation of copyright law.

See chapter 11 of oslbook for more.

5 Likes
8

The work is currently published as is. As I mentioned, I'm not a lawyer, so I don't have the right to provide public clarifications and assessments. If you have specific questions about the License Agreement, please feel free to direct message me.

I appreciate your feedback and will consider ways to make the work more user-friendly in terms of licenses. However, Rust Users might not be the best place to discuss copyright laws, so I suggest we pause our conversation about the legal aspects for now.

We can continue discussing the technical aspects of the project, and I'll be happy to answer any questions you have.

9

(Your last comment arrived while I was writing this post, but to be clear, I don't necessarily expect a response at all, and I'd like to talk more about how the legalities are impacting the social aspects of your project.)

Can I use your work in my free software project?

Yes, absolutely. [...]

Now, I appreciate the distinction isn't pervasive, and it is easy to miss it speaking English as secondary language, but "free software" is often used as a term of art. It does not necessarily mean the intuitive reading of "freeware", but has a meaning that is stronger than merely "open-source" software - "free software" is that which is specifically required by license to be open to sharing and modification conditional only on maintaining that freedom for downstream users.

Conversely, my (IANAL also) reading of this section assigns all rights to derivative works to you. This means the best possible interpretion of your statement is implicitly continuing that "Yes" with, "...with ongoing, express permission from me," because you'd own the distribution rights. As a result, "my" derived work is not only no longer free software by definition, it can't be open source without your permission. This is an enormously bad deal, actively detrimental to any sort of research or knowledge sharing, but more on that in a moment.

That section of the license also directly impacts the quality of the derivative work itself, because it is incompatible with all OSS licenses, "free" or not, because the DW's author can't assign the rights to anyone else's code. It's even quite possible that they can't assign the rights to their own code if it's previously been published under something like the GPL or APL that grant rights irrevocably, but you'd definitely want to consult a lawyer in that situation.

So, as well as the software itself being negatively impacted and complying with that license section being enormously impractical making your readme arguably straight-up incorrect by definition, this is as mentioned, a bad deal that essentially nobody would want to adhere to, because the license effectively amounts to, "Give me (and only me) the copyrights to all that work you did, for free."

I think @scottmcm is right that making the source available in these circumstances is worse than useless, but I'd go even further: IMO, surrounding such an expansive, one-sided license with language like the FAQ itself that suggests your license allows derivative works similarly to other OSS licenses could plausibly be argued to be outright deceptive - that you are trying to "catch out" authors who take your readme statements at face value only to inadvertantly sign over rights to their work in a way that they would not have done if they'd understood the conditions upfront. Or put another way, its plausible to argue that the license is so one-sided, and so at odds with the surrounding colloquial language, that it is unconscionable and a court should assume that anyone who did apparently accept it did not do so knowingly. It would be clearer and more straightforward to outright forbid derivative works, and use a "source available" license if you still want to share the code.

To be very clear, I don't think you are trying to trick anybody but I think it's important to point out just how bad your look is for someone reading less charitably. I think what's more likely to have happened is that you have an idea in your head of what you want people to be able to do with your software, and you have got a license text from somewhere you thought was appropriate (perhaps L.D.'s parent project) and not realized how badly these clash. This is a big problem if you are wanting other people to contribute, and you should address it before you do anything about sharing your project publicly any further, because the license text is what governs in a dispute, and authors know this. Nobody will want to touch your software in a development capacity if they're agreeing to you saying, "I can do whatever I want with your work, including take it off you" by doing so, regardless of your informal comments otherwise. Not educators, not researchers, not even most hobbyists.

I'm also not trying to be discouraging - there is nothing inherently unacceptable with the model you seem to be going for, where your code is available for derivative works with a fee payable to you if those works make enough money. But you definitely need legal help for how to formulate a license that will make that happen, along with adhering to whatever other conditions your parent project imposes on this code. As it is, in this layman's opinion, your current offering is the legal equivalent of code so laden with spaghetti that it's unclear what you're trying to achieve, let alone that you're doing so successfully.

5 Likes
10

To make it clear to everyone reading this topic, Lady Deirdre is not a crowd-development project; it is developed solely by me.

I do accept small contributions if someone needs to fix something quickly, but in general, I'm not calling for contributors.

Actually, for this reason, the license agreement sets the terms to assign these contributions back to me. It is fine if you need to change a code comment in the GitHub sources or suggest a fix for a small bug that you accidentally found. I will accept these changes, but with the requirement that the resulting codebase will not be owned by both of us.

That's basically just how it is. I, of course, don't have any intentions to exploit someone else's labor for free. If someone mistakenly recognizes my project as a classical crowd-developed project and suggests their regular contribution efforts, I will have to politely refuse.

In fact, the term about derivative works was introduced to benefit you, not me. It leaves the possibility for you to change the code if you need a quick fix. And of course, this term does not have any intention to take over other people's work.

Looking at Lady Deirdre as an open-source project with a "strange" license would be a mistake. It's not open-source software in the popular sense. You can see my work as just any creative endeavor in the common sense, such as books, art, or music. From this point of view, perhaps, the distribution policy would be more clear and appropriate.

I hope my explanation provides the necessary clarifications regarding the licensing.

11

The problem is more about the fact that "derivative work" is already legally understood (as I understand it) to include anything that uses a work, even in part, even in binary output form, even via dynamic loading.

Literally any use of this library is owned by you, in other words.

2 Likes
12

That's disputed. Notoriously, the Free Software Foundation holds this position (and it's a key element of interpretation of the GNU GPL, because that license only has the intended effect if linking creates derivative works), but there are various opinions on the matter, and it has not yet been seriously tested in court, as far as I have heard.

(I am not a lawyer. This is not legal advice.)

5 Likes
13

Interesting, but hardly reassuring!

I suppose a license ideally enumerates what rights you have without respect to what a court would consider "derivative" in a copyright sense.

Copyleft resolves the issue by simply holding everyone to the same standard, commercial libraries, in my experience, by careful enumeration of exactly what you are permitted (and not permitted!) to do. The former has the benefit of well analyzed and understood licenses to pick from, and the latter generally has a team of lawyers.

The OP is seemingly in the unhappy middle.

3 Likes
14

Following our conversation, I made some changes to the FAQ in the readme to prevent potential misunderstandings regarding the contribution policy and the use of the term "free software".

Thank you for your feedback!

2 Likes
15

When humanity gets tto a higher level any license for anything should be: 'do whatever you want with it'. Until that time life will be complicated :slight_smile:

16

My amateur judgement is that programmers would be well-advised to only ever interact with this project's code via the actions available in the GitHub interface, since you're leveraging the license the author granted via the GitHub terms of use instead of the EULA.

1 Like
17

That's correct. My license agreement does not attempt to supersede GitHub's terms of use, which is mentioned in the agreement and follows from the fact that I uploaded the sources to GitHub, thereby automatically accepting their terms of use.

I believe this is applicable to almost any GitHub repository legally uploaded to GitHub, regardless of the project-specific license terms (or even without any explicit license grant). As far as I know, GitHub provides you the right to read repository files and to make fork-copies as well. However, without accepting the license terms set by the copyright owner(s), the set of actions you can take with the project is limited.

This is not legal advice, just my thoughts as a non-lawyer.

18

Just a little bit off-topic, but if I were a lawyer, I would suggest taking a look at the definition given by the law in the first place, rather than the opinions of random people from the internet.

In particular, The US Copyright Act paragraph 101 defines "derivative work" as follows:

A “derivative work” is a work based upon one or more preexisting works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which a work may be recast, transformed, or adapted. A work consisting of editorial revisions, annotations, elaborations, or other modifications which, as a whole, represent an original work of authorship, is a “derivative work”.

I think the key phrase here is "represent an original work of authorship", meaning that third-party libraries that are merely separate from the original work of authorship do not fall under this definition.

Perhaps including just this definition from The Copyright Act would be enough, because that's how it is understood by many jurisdictions around the world, not only by the US jurisdiction. However, to make the license clearer, I would add additional clarifications that separable works should not be considered derivatives.

Since I'm not a lawyer, I will leave it to your discretion whether this approach is sufficient to protect the users. But it's worth mentioning that the authors of the Apache License found this approach satisfactory. From the APL2 text:

"Derivative Works " shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.

I'm not sure how well the Apache License, as well as many other OSS licenses, have been tested in court, but I suppose they were at least reviewed by lawyers and seem to be trusted by many applicants.

Furthermore, I would enumerate concrete examples in the license agreement text mentioning what should and should not be considered "derivative works". Perhaps, it's not feasible for generic licenses such as APL or any other OSS license, but I think it's a good idea for a project-specific license to include such examples to increase end-user protection even further.

I believe that anyone who reads my license agreement text, lawyer or not, should be aware that the text contains the inclusions that I mentioned above.

19

I believe the last paragraph about severability makes a tautological argument that would make the whole thing null and void in many jurisdiction.

Because that paragraph is null and void, then all the others have to be null and void. Or if is valid then all the rest can't be.

20

In my opinion, this paragraph states exactly the opposite, but I will leave the assessments to professional lawyers.