I'm looking for a scripting language for an embedded project because users are wanting features from the competition. Unfortunately, the competition has twice the memory of our common platforms, and can install Lua (their choice) with no issues. But it - and the interpreters I could find that were otherwise acceptable - needs a stripped-down version to fit on our hardware at all, much less if we're using it.
One of the people asking for these features enumerated them with the comment "A VM and garbage collector seems like overkill for these." Well, I only know of three alternatives to GC: let the programmer do it (not acceptable), the DDI algorithm (looking into it elsewhere), and the Rust technique of doing it in the compiler.
Which brings up some questions: Anyone ever looked into either of these things? How would the memory footprint of an interpreter that included a borrower and lifetime tracking compare to the ones that include a GC? Is this idea even remotely sane?
I'll bet you could implement a language like TRAC in very little memory in Rust. Its speed wouldn't compete with a JIT, but it'd work.
EDIT: Wow, TRAC is really dead. I've posted the MINT tutorial from the Freemacs distribution; MINT is a variant of TRAC used by Freemacs, an extensible text editor meant to resemble GNU Emacs.
I'm not sure if I can recommend building your own language with good conscience, since that is the kind of project das can spiral out of control very quickly. But it should be easy enough to create a little text-based interpreted command language like tcl or shell with very small space requirements.
In respect to the GC question: What about reference counting? The space overhead is small and it would be much easier to share values between the hosting and the scripting language without having a mismatch between memory management strategies.
I'll assume that using miri is out of the question? You'd need to get large parts of the compiler into the flash of your device. Memory usage can probably be tweaked by removing some memory safety checks. You'd get Rust as an interpreted language.
Since your target is so restricted it has trouble fitting Lua, one of the smallest scripting interpreters, I suppose it does not fit the editor either, does it? Therefore there is a tool on user's desktop that pushes the scripts to the target. And this tool could compile them.
For that, Rust would make sense, because if you restrict (in the tool) use of unsafe and restrict the interfaces you make available, the user should not be able to break the system while still doing what they need and without any interpreter on target.
Rust is much more complex than Lua, and so I think it is not possible to have a Rust compiler or interpreter smaller than a Lua interpreter.
There are small interpreters for other languages, though. I remember in the '80s there were small interpreters for BASIC, Forth, and Lisp. Lisp uses garbage collection, but old BASICs do not, using only arrays. I don't know about Forth.
More info on requirements: Target device is an RC Transmitter: your UI is 6 buttons and a small LCD. So there is no need for either an editor or a REPL, though I'd love to have one even if only for debug builds that include a serial interface.
Our users are NOT programmers/engineers/etc. We've already had complaints about our program being to difficult for non-technical people. So the program that puts things on the transmitter is "cp" or "finder" or whatever copies programs. We've played with desktop tools, but generally have bad luck with them, so no compiler. Maybe a visual DSL, but then I'm back to designing my own language.
Similarly, all the languages with non-intuitive syntax like FORTH and LISPS are out. A suitable BASIC might do given how many languages have adopted that name, but that really just changes the search criteria, as if I wrote my own I'd call it something else.
I was about to recommend Forth before I saw that you had discounted it. Rather than implementing your own full language though, it might be worth investigating still using Forth but implementing a bunch of domain-specific "words" (Forth's idea of a function) to make Forth look more like a Domain Specific Language. Yes, anyone who wanted to do anything complex would still need to learn Forth, but for basic, macro-like behavior the Forth-y-ness would be pretty opaque.
Just a comment on continuing to search for an alternative, since at one point I did a similar search (in that case I was able to get Lua working). If you are as resource constrained as it sounds, you are pretty much looking at Forth, Lua, or a LISP-like to do anything interpreted. At some point your platform becomes so restrictive that you end up having to rely on the programmer for something, and the available user-friendly languages goes way down. I'd love to be proven wrong though!
It isn't clear whether you've looked at the "normal" Lua or at it's embedded-adapted version ELua. There's also MicroPython.
Do you need to run script parser on device, or can that be offloaded to the host computer?
It'd be easier to offer suggestions if you gave us your actual flash & RAM constraints...
I doubt the people who find it difficult to just control will be writing any scripts. Writing scripts—any kind of scripts, for anything, even quite simple ones—requires certain way of technical thinking they don't have. So the non-technical users will mainly scourge forums for useful enhancements they could use. That is the case you have to keep simple.
However, the people capable of writing the scripts should also be able to run some kind of simple tool to compile or preprocess the scripts. It would be making it slightly harder for them to share their creations, but if that's what it takes to fit the scripts into the device, it should work fine. So I think you shouldn't completely rule compilation or preprocessing out.
Note that since the device is too small to fit in even a rather small interpreter like Lua, the scripts it will be able to handle also won't be very big. I expect usually one, at most a couple of files. That does not need dependency management or complex build system and such. Just a tool that you run and it will find all sources in a given folder and produce corresponding binaries to be copied over to the device.
Um, no, this doesn't work. While the people writing the scripts can certainly deal with running some kind of compiler/preprocessor, as you yourself observe, they aren't the majority of your users. Those people have issues dealing with the text files we currently distributer - they don't know where to put them, break them trying to use Notepad to edit them, have issues with names, etc.
Based on the experience with those text files, sharing pre-compiled files is a non-starter. We support a dozen different hardware platforms, any combination of four different RF modules, dozens of radio protocols and hundreds of models. It's almost unheard of for a model to work on more than one or two of those combinations unmodified. Simple scripts might work on most or even all of them, but anything fancy is liable to need tweaking for the target hardware if not the users tastes.
This is also why I flash constraints are kinda hard to specify. The most popular transmitter has 128K of flash, but that may change as a vendor has just released the first Tx designed to run this software, which is very similar to that, but with an upgraded µ-processor with 256K. That's also the most common flash size, though there are some models with 512K. We use somewhere around 200K on those processors, depending on what peripherals are available.
That's about what I found. If you do the math, 256k minus the 200k or more used by the application leaves at most 56k, which means micropython at 80k, or 60k without the REPL, doesn't fit.
I don't see why it being Rust has any more implications than Python. The requirements for our target audience come from supporting them over the last half-dozen years:
No desktop processing. This is because the single most common issue we see is with installing extra software on the desktop. Desktop tools designed to make their lives easier have been written, but none have caught on.
Nothing that's going to put off non-programmers. So arithmetic should match what they got taught in school. Things that are surprising to programmers - which is where Rust's gotchas are - are fine.
56K is not that far from 60K and these are rough numbers anyways. If I were you, I'd at least try to compile uPython (with opt for size) and see what happens.
I doubt Rust interpreter would be more compact even if one existed.
Much as we all like Rust around here, I don't think you'll find anyone who will claim with a straight face that Rust is easier to pick up than Python. The target audience you are describing is not consistent with someone who'd be programming in Rust.
How do they communicate with your boards then? Don't they need some sort of a toolset?
How do you think I arrived at the 60K figure for size? 56K is the absolute best case for the most popular chip choice. That's using the small, B&W LCD, not the larger color touch one (>12K more flash used), that doesn't include the new voice-based audio output (a very popular feature), the current rewrite of the UI for new transmitters with a different set of buttons. or any other ongoing work.
There is a guy who's working on porting micropython to the 128K STM32F103 that we use in our low end devices, and he claims to have it down to 40K, which is small enough to be interesting. But he also says you have to use compiled-into-flash Python code, not scripts from the file system, because he took out the REPL. Taking out the REPL shouldn't require compiled-into-flash, and I'm fine without a REPL, but can't live with compiled-into-flash. I've asked him for clarification, but haven't gotten an answer yet.
But Rust being easier isn't the relevant question - it's whether or not it's harder for non-programmers than Python. The borrower and associated stuff are a PITA - for programmers. People with no programming experience won't be having to unlearn things, so I don't think it'll be much harder than dealing with a garbage collector, and certainly easier than what C/C++ provides. A lot of people are learning embedded programming using C/C++ to control Arduino hardware (which frankly surprised me), so I'd expect them to be fine with Rust.
And a lot of users won't be doing programming per se; they'll be editing scripts to replace the name of a switch that was on the original authors transmitter with one that's on theirs, or inverting the sense of a switch because that's their preference, or changing a value because the used a 4x voltage divider instead of the 5.5x divider the author used, or something equally trivial.
Well, they need specific tools to install the software - and that's the major source of user problems. Once it's installed, it presents it's sd flash as a USB drive, and that's how it communicate with desktop systems. There's an LCD screen - possibly with touch - and a collection of buttons, switches & analog input devices for tweaking settings and the like, but those aren't really available for editing scripts. Imagine programming with an XBox controller.
So the primary tool for installing shared model files is the desktop, and the primary tool for editing them on the desktop Notepad++. And that causes sufficient pain that we've had people write desktop tools for managing/installing or editing those files - and almost no one bothers installing those tools.
Imagine if you could have the simplicity and freedom of Arduino with the power and features of a Raspberry Pi. With the ease of a microcontroller and the flexibility of a real computer, Ultibo gives you a platform for creating anything without the limitations of a traditional operating system.