Lightflus: a cloud-native, typescript-api Dataflow Engine

announcements
1

What kind of Big-Data Computation Framework do we need?

Hi all, I'm very happy to announce that Lightflus, a new distributed dataflow engine, has been in preview phase. Unlike Flink or Spark, only very professional teams can work on them, Lightflus is designed for most developer teams, even no one is familiar with Dataflow processing. You can write and deploy a distributed, high-performance Dataflow task like write and run stream code on your computer.

I think a big-data computation framework for the future should satisfy the following criteria:

  • Cloud-Native, which means highly scalable and available;
  • Cloud regardless, which means it can be integrated with any of cloud infra;
  • Pay-As-You-Go, which means much money will be saved for you;

Unfortunately, both Flink and Spark fail to match any of these features. They are not born for cloud, none of them are supported very well by mainstream cloud vendors, and you must invest a lot of money to build a basically usable data system to meet your team's demands.

So I decide to launch a new project, Lightflus, to resolve all these problems.

The tech stack of Lightflus

TypeScript API + Rust Runtime

Lightflus is powered by Deno and implemented in Rust which can ensure memory safe and real-time performance. We embed v8 engine into Lightflus engine with minimum dependencies makes it light and fast; With the help of Deno, you can run Typescript user-defined functions or WebAssembly encoded bytes code (for better performance) in Lightflus with stream-like API;

Architecture

We apply the mainstream distribute computation architecture, Coordinator-Worker, to satisfy our demands of fault tolerance, scalability and availability in the future.

Docker and k8s Support

Before release version, only Docker is planed to be supported.

And in release version, we will support k8s (means you can try Lightflus on production environment).

Github

You can get the project on Github . Your star/fork/watch is very important to us and we're very appreciate if you can give us any feedback or pull request.

When to release demo version?

I decide to release demo version Feb next year. I will announce this news on Rust community at the first time.

BTW, thanks a lot for your attention on this post!

2

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.