Strange "random" cargo install error on Linux could not compile `serde_derive`

help
1

I have been getting strange random build errors on Linux, in this case I disconnected, re-ran the command and it worked, when it has just failed several times previously. Below is copy/paste of my session in case nobody believes it! Error message is

"error: could not compile serde_derive"

I have no idea what is going on here.

ubuntu@ip-172-26-13-111:~$ cargo install rustdb-axum-example
    Updating crates.io index
  Installing rustdb-axum-example v0.1.28
   Compiling proc-macro2 v1.0.43
   Compiling quote v1.0.21
   Compiling unicode-ident v1.0.3
   Compiling syn v1.0.99
   Compiling autocfg v1.1.0
   Compiling libc v0.2.127
   Compiling cfg-if v1.0.0
   Compiling version_check v0.9.4
   Compiling log v0.4.17
   Compiling pin-project-lite v0.2.9
   Compiling futures-core v0.3.21
   Compiling memchr v2.5.0
   Compiling cc v1.0.73
   Compiling futures-task v0.3.21
   Compiling smallvec v1.9.0
   Compiling once_cell v1.13.0
   Compiling futures-util v0.3.21
   Compiling bytes v1.2.1
   Compiling scopeguard v1.1.0
   Compiling itoa v1.0.3
   Compiling futures-io v0.3.21
   Compiling parking_lot_core v0.9.3
   Compiling pin-utils v0.1.0
   Compiling fnv v1.0.7
   Compiling pkg-config v0.3.25
   Compiling proc-macro-hack v0.5.19
   Compiling serde_derive v1.0.142
   Compiling serde v1.0.142
   Compiling percent-encoding v2.1.0
   Compiling tower-service v0.3.2
   Compiling matches v0.1.9
   Compiling httparse v1.7.1
   Compiling bitflags v1.3.2
   Compiling mime v0.3.16
   Compiling const_fn v0.4.9
   Compiling async-trait v0.1.57
   Compiling foreign-types-shared v0.1.1
   Compiling futures-sink v0.3.21
   Compiling encoding_rs v0.8.31
   Compiling parking_lot_core v0.8.5
   Compiling tinyvec_macros v0.1.0
   Compiling tower-layer v0.3.1
   Compiling arrayvec v0.4.12
   Compiling futures-channel v0.3.21
   Compiling openssl v0.10.41
   Compiling nodrop v0.1.14
   Compiling httpdate v1.0.2
   Compiling native-tls v0.2.10
   Compiling try-lock v0.2.3
   Compiling ryu v1.0.11
   Compiling unicode-bidi v0.3.8
   Compiling serde_json v1.0.83
   Compiling minimal-lexical v0.2.1
   Compiling match_cfg v0.1.0
   Compiling base64 v0.13.0
   Compiling spin v0.9.4
   Compiling http-range-header v0.3.0
   Compiling openssl-probe v0.1.5
   Compiling constant_time_eq v0.1.5
   Compiling scoped_threadpool v0.1.9
   Compiling rustc-hash v1.1.0
   Compiling email_address v0.2.1
   Compiling matchit v0.4.6
   Compiling sync_wrapper v0.1.1
   Compiling fastrand v1.8.0
   Compiling quoted_printable v0.4.5
   Compiling lock_api v0.4.7
   Compiling slab v0.4.7
   Compiling tokio v1.20.1
   Compiling instant v0.1.12
   Compiling standback v0.2.17
   Compiling time v0.2.27
   Compiling multer v2.0.3
   Compiling cookie v0.15.1
   Compiling libmimalloc-sys v0.1.25
   Compiling tracing-core v0.1.29
   Compiling http v0.2.8
   Compiling openssl-sys v0.9.75
   Compiling form_urlencoded v1.0.1
   Compiling foreign-types v0.3.2
   Compiling tinyvec v1.6.0
   Compiling http-body v0.4.5
   Compiling unicode-normalization v0.1.21
   Compiling socket2 v0.4.4
   Compiling num_cpus v1.13.1
   Compiling hostname v0.3.1
   Compiling mio v0.8.4
   Compiling tracing v0.1.36
   Compiling want v0.3.0
   Compiling nom v7.1.1
   Compiling email-encoding v0.1.3
   Compiling idna v0.2.3
   Compiling blake2-rfc v0.2.18
   Compiling parking_lot v0.12.1
   Compiling parking_lot v0.11.2
   Compiling mimalloc v0.1.29
   Compiling argon2rs v0.2.5
   Compiling futures-macro v0.3.21
   Compiling tokio-macros v1.8.0
error: could not compile `serde_derive`

Caused by:
  process didn't exit successfully: `rustc --crate-name serde_derive /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.142/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="default"' -C metadata=d11c20883e470905 -C extra-filename=-d11c20883e470905 --out-dir /tmp/cargo-installw30RPX/release/deps -L dependency=/tmp/cargo-installw30RPX/release/deps --extern proc_macro2=/tmp/cargo-installw30RPX/release/deps/libproc_macro2-1a522b1cd25c81a4.rlib --extern quote=/tmp/cargo-installw30RPX/release/deps/libquote-1f5041d6e22e1c50.rlib --extern syn=/tmp/cargo-installw30RPX/release/deps/libsyn-d9650de73047a8ef.rlib --extern proc_macro --cap-lints allow` (signal: 9, SIGKILL: kill)
error: failed to compile `rustdb-axum-example v0.1.28`, intermediate artifacts can be found at `/tmp/cargo-installw30RPX`
ubuntu@ip-172-26-13-111:~$ client_loop: send disconnect: Connection reset

C:\Users\pc\Downloads>ssh -i PomesoftUbuntu.pem ubuntu@13.40.58.204
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-1018-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Aug  4 13:24:00 UTC 2022

  System load:           0.0
  Usage of /:            34.7% of 19.32GB
  Memory usage:          46%
  Swap usage:            0%
  Processes:             107
  Users logged in:       0
  IPv4 address for eth0: 172.26.13.111
  IPv6 address for eth0: 2a05:d01c:568:7d00:d314:ba4d:65bf:a0a

 * Ubuntu Pro delivers the most comprehensive open source security and
   compliance features.

   https://ubuntu.com/aws/pro

47 updates can be applied immediately.
9 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable


3 updates could not be installed automatically. For more details,
see /var/log/unattended-upgrades/unattended-upgrades.log

Last login: Thu Aug  4 08:26:32 2022 from 146.200.201.79
ubuntu@ip-172-26-13-111:~$ cargo install rustdb-axum-example
    Updating crates.io index
  Installing rustdb-axum-example v0.1.28
   Compiling proc-macro2 v1.0.43
   Compiling quote v1.0.21
   Compiling unicode-ident v1.0.3
   Compiling syn v1.0.99
   Compiling autocfg v1.1.0
   Compiling libc v0.2.127
   Compiling cfg-if v1.0.0
   Compiling version_check v0.9.4
   Compiling log v0.4.17
   Compiling pin-project-lite v0.2.9
   Compiling memchr v2.5.0
   Compiling futures-core v0.3.21
   Compiling cc v1.0.73
   Compiling futures-task v0.3.21
   Compiling once_cell v1.13.0
   Compiling smallvec v1.9.0
   Compiling scopeguard v1.1.0
   Compiling bytes v1.2.1
   Compiling itoa v1.0.3
   Compiling futures-util v0.3.21
   Compiling futures-io v0.3.21
   Compiling parking_lot_core v0.9.3
   Compiling pin-utils v0.1.0
   Compiling pkg-config v0.3.25
   Compiling fnv v1.0.7
   Compiling serde_derive v1.0.142
   Compiling proc-macro-hack v0.5.19
   Compiling serde v1.0.142
   Compiling matches v0.1.9
   Compiling percent-encoding v2.1.0
   Compiling httparse v1.7.1
   Compiling mime v0.3.16
   Compiling async-trait v0.1.57
   Compiling bitflags v1.3.2
   Compiling tower-service v0.3.2
   Compiling const_fn v0.4.9
   Compiling arrayvec v0.4.12
   Compiling tinyvec_macros v0.1.0
   Compiling openssl v0.10.41
   Compiling encoding_rs v0.8.31
   Compiling futures-channel v0.3.21
   Compiling foreign-types-shared v0.1.1
   Compiling futures-sink v0.3.21
   Compiling parking_lot_core v0.8.5
   Compiling tower-layer v0.3.1
   Compiling native-tls v0.2.10
   Compiling ryu v1.0.11
   Compiling try-lock v0.2.3
   Compiling httpdate v1.0.2
   Compiling nodrop v0.1.14
   Compiling spin v0.9.4
   Compiling openssl-probe v0.1.5
   Compiling serde_json v1.0.83
   Compiling constant_time_eq v0.1.5
   Compiling match_cfg v0.1.0
   Compiling unicode-bidi v0.3.8
   Compiling minimal-lexical v0.2.1
   Compiling http-range-header v0.3.0
   Compiling base64 v0.13.0
   Compiling scoped_threadpool v0.1.9
   Compiling quoted_printable v0.4.5
   Compiling rustc-hash v1.1.0
   Compiling matchit v0.4.6
   Compiling fastrand v1.8.0
   Compiling sync_wrapper v0.1.1
   Compiling email_address v0.2.1
   Compiling lock_api v0.4.7
   Compiling slab v0.4.7
   Compiling tokio v1.20.1
   Compiling instant v0.1.12
   Compiling standback v0.2.17
   Compiling time v0.2.27
   Compiling cookie v0.15.1
   Compiling multer v2.0.3
   Compiling libmimalloc-sys v0.1.25
   Compiling tracing-core v0.1.29
   Compiling openssl-sys v0.9.75
   Compiling http v0.2.8
   Compiling form_urlencoded v1.0.1
   Compiling tinyvec v1.6.0
   Compiling foreign-types v0.3.2
   Compiling http-body v0.4.5
   Compiling unicode-normalization v0.1.21
   Compiling socket2 v0.4.4
   Compiling num_cpus v1.13.1
   Compiling hostname v0.3.1
   Compiling mio v0.8.4
   Compiling tracing v0.1.36
   Compiling want v0.3.0
   Compiling email-encoding v0.1.3
   Compiling nom v7.1.1
   Compiling blake2-rfc v0.2.18
   Compiling idna v0.2.3
   Compiling parking_lot v0.12.1
   Compiling parking_lot v0.11.2
   Compiling mimalloc v0.1.29
   Compiling argon2rs v0.2.5
   Compiling futures-macro v0.3.21
   Compiling tokio-macros v1.8.0
   Compiling pin-project-internal v1.0.11
   Compiling time-macros-impl v0.1.2
   Compiling openssl-macros v0.1.0
   Compiling pin-project v1.0.11
   Compiling time-macros v0.1.1
   Compiling axum-core v0.1.2
   Compiling tokio-util v0.7.3
   Compiling hyper v0.14.20
   Compiling serde_urlencoded v0.7.1
   Compiling rustdb v1.2.3
   Compiling tower v0.4.13
   Compiling lettre v0.10.1
   Compiling tower-http v0.2.5
   Compiling tower-cookies v0.4.1
   Compiling axum v0.4.8
   Compiling rustdb-axum-example v0.1.28
    Finished release [optimized] target(s) in 4m 33s
   Replacing /home/ubuntu/.cargo/bin/rustdb-axum-example
    Replaced package `rustdb-axum-example v0.1.27` with `rustdb-axum-example v0.1.28` (executable `rustdb-axum-example`)
ubuntu@ip-172-26-13-111:~$ sudo systemctl restart rustweb
u
2

SIGKILL could be an indication of the kernel killing the process, maybe the machine was out of memory. Did you check dmesg?

2 Likes
3

No, never heard of it, I am very new to Linux. I just looked now and at the bottom of the report it says:

[785931.582523] Out of memory: Killed process 108772 (rustc) total-vm:354444kB, anon-rss:147964kB, file-rss:780kB, shmem-rss:0kB, UID:1000 pgtables:648kB oom_score_adj:0
[785931.604912] oom_reaper: reaped process 108772 (rustc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

A bit further up it says:

[785786.245041] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-331.scope,task=rustc,pid=105859,uid=1000
[785786.245053] Out of memory: Killed process 105859 (rustc) total-vm:348300kB, anon-rss:148704kB, file-rss:676kB, shmem-rss:0kB, UID:1000 pgtables:636kB oom_score_adj:0
[785786.271825] oom_reaper: reaped process 105859 (rustc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[785931.582361] cargo invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[785931.582364] CPU: 0 PID: 105889 Comm: cargo Not tainted 5.4.0-1018-aws #18-Ubuntu
[785931.582365] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
[785931.582366] Call Trace:
[785931.582372]  dump_stack+0x6d/0x9a

Which seems to be saying "cargo invoked oom-killer" ??

4

The phrasing it indeed a little confusing. It means you ran cargo install, which called rustc, which allocated too much memory, which invoked the OOM Killer.

It killed the rustc process because it used 348MB of memory. Please check how much memory your machine has and how much of it is used. The command for that is free -htw.

1 Like
5 
ubuntu@ip-172-26-13-111:~$ free -htw
              total        used        free      shared     buffers       cache   available
Mem:          475Mi       166Mi        84Mi       0.0Ki        18Mi       205Mi       287Mi
Swap:            0B          0B          0B
Total:        475Mi       166Mi        84Mi
ubuntu@ip-172-26-13-111:~$
6

You machine only has 457 MiB (roughly 500MB) of RAM. If you're using a cloud provider, I suggest getting the next bigger option, especially if you are still testing things out.

7

Thanks, at least I now know what is going on. Maybe I should look at cross-compiling? I think I can build a binary on my on machine and upload it to the cloud machine?

[ I am also a bit surprised... I assumed Linux would use virtual memory, there is lots of disk space? It seems not, which means it is quite easy to run out of memory?? ]

8

Yes linux can do that, it's called "swap". But it's going to be very slow.

1 Like
9

It seems to work better after re-booting the server. I also tried a few other things ( I have some suspicion maybe inconsistent serde feature selection could be something to do with it... ), and it now builds "fairly reliably". Maybe!

10

Adding some swap space will help (at the expense of performance).

3 Likes
11

What I am doing now is doing a conventional cargo build rather than cargo install.

It did fail compiling serde_derive as before, but then when I repeated the cargo build, due to incremental compilation, it carried on and completed the build just fine.

My conclusion: avoid cargo install on machines that have limited memory.

Edit: I spoke too soon, now it is failing at the link stage.

12

In general I've learned to avoid compiling C++ and Rust on small machines. And I choose my PCs partly based on that.

13

Thanks! That worked. I used this video: Adding Swap to an AWS Lightsail Ubuntu Instance (running Wordpress on NGINX) - YouTube

2 Likes
14

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.