Apitap — open-source ingestion engine: move big tables fast on tiny machines (Rust + Python, MIT). Contributors welcome!

announcements
1

Hi everyone :waving_hand:

I'd like to introduce apitap, an open-source data-transfer engine I've been building — Rust core, Python bindings, MIT-licensed.

The idea behind it is simple: moving a lot of data shouldn't require a lot of machine. Most ingestion pipelines cost money twice — once in wall-clock hours, once in the oversized workers those hours run on. apitap streams raw wire formats through parallel pipes with bounded memory, so the same job that needs gigabytes elsewhere finishes on the smallest container you can rent.

Where it stands today (10M rows, 15 column types, every number checksum-validated against the source; full methodology in the repo):

route apitap ingestr 1.0.75 dlt 1.x
Postgres → Postgres 20.2 s 500 s 2,604 s
Postgres → ClickHouse 9.9 s 111 s 1,893 s
MySQL → ClickHouse 10.4 s 97 s 2,231 s
MySQL → Postgres 22.5 s 481 s 2,899 s
Postgres → BigQuery 28.4 s 860 s 2,160 s

On a 0.5 vCPU / 256 MB container apitap still completes every route; in our runs the other tools either crawled or were OOM-killed there. (BigQuery bonus: the whole ingestion path is free — load + copy jobs only, no DML — so it works on sandbox projects without billing.)

The goal we're quietly working toward: 10M rows per minute on that 256 MB box. We're about 3× away, and honestly we may never fully get there — but every step is measured, one lever at a time, and written down with its caveats and our own mistakes: benchmarks/README.md

I'd love help. If any of this sounds fun, contributions of every size are welcome:

  • :electric_plug: New connectors — MySQL → BigQuery is next on the list, then Snowflake; the Source/Sink traits are small and documented
  • :bug: Break our benchmarks — run them on your workload; if a number doesn't hold up or a type gets mangled, an issue with a repro is the most valuable gift
  • :test_tube: Real-world testing — weird schemas, huge rows, exotic types, flaky networks
  • :open_book: Docs — if something confused you, that's a bug in the docs
  • :red_apple: Platform wheels — aarch64 / macOS builds via CI (maturin-action)

Repo: GitHub - apitap/apitap-lib · GitHub
Usage guide: apitap-lib/docs/usage.md at main · apitap/apitap-lib · GitHub
Origin story: I moved 10 million rows in 9.9 seconds with pip install apitap

Full disclosure: it's built with Claude Code as pair programmer, and this project has been corrected by its own failed runs more than once. If you see something wrong, please say so loudly — that's exactly how it gets better. Thank you! :folded_hands:

2

Awesome!

3

Thanks :smiley:

4

I think this is pretty significant stuff.

A short anecdote... I worked for a company/startup in the '06-'07 time frame that bought acquired another company. Moving the data to us in Florida was not going to be quick! What wound up happening was a couple of people getting on planes to the north east from Florida, picking up the drives from a co-lo, then driving back down the eastern seaboard with them!

A different crazy maneuver I pulled.... A db dump piped to a running server in a co-lo facility. Mysql makes that easy...

mysqldump -u user_name -ppassword target_db | mysql -h some_host_some_place -u user_name -ppassword receiving_db

Damn that was slow! LOL!!!

I can see this being useful for a lot of dev-ops peeps going forward! I'm not sure how I can help, but I'll at least cheer lead for the short term and see if there is some way I can test.

:+1:

5

yes i've similar experiecen like that when i was devops few years ago. aggree with you might apitap can help devops do that more faster. so if you want try you can do:

pip install apitap

python -c "import apitap; r = apitap.transfer(

    'mysql://user:pass@old-host/db',

    'mysql://user:pass@new-host/db',

    schema='db')   # every table, one call

print(r.rows)"

needs local_infile=1 on the destination — same switch mysqlimport needs)

The feedback that would help me most, in order:

1. a table where it's slower than you expected (with rough row count + column shapes)

2. any type that doesn't round-trip exactly — I checksum-test releases but real

schemas always find something

3. how it compares against your mysqldump pipe on the same table, if you still

have one handy

Issues welcome however small: Issues · apitap/apitap-lib · GitHub

Cheerleading gratefully accepted too. Thanks for the kind words!