Native vs containerized; different runtime behavior - perhaps explained by lazy_static!?

Hello Everyone,

I'm having a blast working with axum and "friends" - all the types axum "gets for free". I'm also really impressed with how little we can get a rust bin "stood-up" in a docker/kubernetes setting.

It's often said: there's no such thing as a free lunch. I'm trying to work out a user-friendly way to toggle between a development and production version of the app. I'm using the config 0.13 package to accomplish the task.

impl Settings {
    pub fn new() -> Result<Self, ConfigError> {
        // defaults to development when missing
        let mode = env::var("RUST_ENV").unwrap_or_else(|_| "Development".into());

        tracing::info!("\n🦀 mode: {}\n", &mode);

        let settings = Config::builder()
            .add_source(File::with_name(CONFIG_FILE_PATH))
            // use env Development, Production or Testing to set override
            .add_source(File::with_name(&format!("{}/{}", CONFIG_FILE_ROOT, mode)).required(false))
            .add_source(Environment::default().separator("_"))
            .build()?;

        settings.try_deserialize()
    }
}

lazy_static! {
    pub static ref CONFIG: Settings = Settings::new().unwrap();
}

When I run the bin on my machine, the binary loads the dev version of the configuration.

🦀 mode: Development

2022-04-22T14:55:16.428632Z  INFO oauth: RUST_ENV:Development
2022-04-22T14:55:16.428693Z  INFO oauth: RUST_LOG:oauth=trace,tower_http=trace
2022-04-22T14:55:16.428743Z  INFO oauth: listening on 0.0.0.0:3099
2022-04-22T14:55:16.428778Z  INFO oauth: auth redirect url http://localhost:3099/auth/authorized/<auth-agent>

However, when I run the binary in a docker container I get the following:

🦀 mode: Development

2022-04-22T14:58:45.146430Z  INFO oauth: RUST_ENV:Development
2022-04-22T14:58:45.146519Z  INFO oauth: RUST_LOG:oauth=trace,tower_http=trace
2022-04-22T14:58:45.146531Z  INFO oauth: listening on 0.0.0.0:xxxx
2022-04-22T14:58:45.146540Z  INFO oauth: auth redirect url https://xxx.xx/auth/auth/authorized/<auth-agent>
Here is the Dockerfile.local used to build the image
####################################################################################################
## Use this for development
## the runtime configuration location: /app/config/development.toml
##
## Builder stage
## 🔖 Every call to RUN COPY generates a new layer
## ref: https://wlkn.io/blog/rust-ci-speedup
####################################################################################################
FROM ekidd/rust-musl-builder:1.57.0 as builder
WORKDIR /app
USER root
####################################################################################################

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
    tree \
 && rm -rf /var/lib/apt/lists/*

## Create a dependency shell
COPY ./Cargo.* ./
COPY ./file-server ./file-server
COPY ./async-redis-session ./async-redis-session
## tricky picking
# COPY ./oauth ./oauth
# RUN rm -rf /app/oauth/src/*
RUN mkdir -p oauth/src
COPY ./oauth/Cargo.toml ./oauth/Cargo.toml
COPY ./dummy-main.rs /app/oauth/src/main.rs
RUN cargo fetch
RUN cargo build \
        --release \
        --target x86_64-unknown-linux-musl \
        --bin oauth \
        --target-dir target
RUN find target -type f -name "oauth[-]*"

####################################################################################################
## Compute any additional artifacts to be copied
####################################################################################################
# Create appuser
ARG USER=appuser
RUN adduser \
        --disabled-login \
        --disabled-password \
        --gecos "" \
        --no-create-home \
        "${USER}"

####################################################################################################
## build the application
COPY ./oauth/src ./oauth/src
RUN cargo build \
        --release \
        --target x86_64-unknown-linux-musl \
        --bin oauth \
        --locked \
        --target-dir target

# separate layer for config
COPY ./config ./config

# assemble binary, config and chown to appuser
RUN echo "📁root: $(pwd)" && \
    echo "👉app/oauth/src" && ls -l /app/oauth/src && \
    echo "📁/app/target" && tree -L 3 -a /app/target && \
    mkdir /app/bin && cp /app/target/x86_64-unknown-linux-musl/release/oauth /app/bin/ && \
    chown -R "${USER}" /app && \
    echo "✅/app" && ls -la /app && tree -L 2 -a /app

####################################################################################################
## Final image
####################################################################################################
FROM frolvlad/alpine-bash:latest as runtime
WORKDIR /app
ARG USER=appuser

# Import from builder
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /app/bin    /app/bin
COPY --from=builder /app/config /app/config

ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_DIR=/etc/ssl/certs
ENV RUST_ENV=Development
ENV RUST_LOG="oauth=trace,tower_http=trace"

USER ${USER}
CMD ["/app/bin/oauth"]

I've confirmed the presence of the config/* files in the docker container.

Here is `development.toml` contents pulled from the **container**:
> docker exec -it auth-dev /bin/bash
bash-5.1$ ls config
default.toml  development.toml  scope.data.json  testing.toml

bash-5.1$ cat config/development.toml
[options]
# be sure to match for cookie-passing
tnc_domain = "localhost"

# auth redirect url
tnc_auth_host = "http://localhost:3099"

# align with ingress prefix for the service
tnc_auth_prefix = ""

# service to register authorized users
tnc_session_service = "localhost:3000"

Question

What might prevent the docker version from reading-in the development version of the configuration?

While I understand the concept of "lazy" evaluation, I'm not precisely sure how that plays out in the lazy macro. Could it be that the static value is being set prior to "reading-in" the environment settings?

1 Like

I'm not sure what you mean by "prevent from reading" – it seems that the process in the Docker container runs successfully as well?

Hi @H2CO3 -- Unfortunately it doesn't work because the redirection host must match that from where the endpoint was "hit" (required for cookie sharing, in turn required for session retrieval).

So, instead of https://xxx.xx/auth/authorized/.., it should be http://localhost:3099/auth/authorized/....

The default and development versions of the configuration each use 0.0.0.0 as the host, so I can "hit" the hosted endpoints. However, the docker version fails midway through the process because xxx.xx cannot retrieve the cookies set by the localhost.

"prevent from reading" the configuration -- I used that terminology because the config builder first reads-in the default.toml. It must subsequently read-in the development.toml. This last step may be "prevented".

Can you modify the code in a way that confirms with certainty which file is actually read in, and if it fails, it shows the relevant error (likely an io::Error)?

1 Like

I’ll play with it some more when I get back to my desktop. In the meantime,

  1. I know that both contexts read-in the default config. The app would not work at all without the information provided in the default.toml file.

  2. I know the native version also reads-in the development.toml file because it works as expected (default + development).

Both seem to have the same env variable value that controls whether to read-in development.toml (RUST_ENV=Development per the mode local variable set using env::..).

Is there something else I should confirm?

There are no io errors in either context.

If you believe this file is not being found when it's present, you might be able to provoke an error (in order to confirm or reject this theory) by changing required to true.

Beyond that, it may be worth reading the config source code to see if there are any debugging options you can use to watch how it's locating your config, and whether it's successfully loading the file you want or not. Right now you're using it as a black box, which is great, but if the black box isn't doing what you want, it's okay to start peeking inside of it.

3 Likes

Thank you everyone!

The idea to modify the code combined with the specific means to require the file was the 1-2 punch that had me figure it out. When I required the file, I received an io error in the docker container. The filename it was looking for was Development.toml, I had development.toml. Daft mistake!

A couple of quick questions:

  1. is it a norm that MacOS was able to find the file where the linux-based container wasn't?

  2. in light of the responses everyone was confident the static value would not be set until the values it depended on were evaluated at runtime (per the docs). The config package has a feature that enables reloading the config when any of the config files change. Would I have to forgo using the lazy_static! macro if I wanted to "watch"/re-evaluate the config when the config files change? (are the two capabilities, the config "watch" and lazy_static! inherently incompatible)

Yes - Apple's filesystem defaults to being case-preserving but case-insensitive, so Development.toml and development.toml will resolve to the same file. It has a mode for case sensitivity, and macOS itself works fine with it, but it's not the default.

2 Likes

No, why would they? Their very own watch example uses lazy_static. They wrap the config value in a lock so that it can be mutated, but that's required any time you want a mutable global.

Anyway, I would switch from lazy_static to once_cell::Lazy as it doesn't require a macro and it's a candidate for inclusion into std.

1 Like

@H2CO3 conceptually I was thinking that lazy_static! stays true to the contract of a static variable - location in memory never changes, live as long as the app (defacto 'static lifetime). The only diff in relation to static, is to adjust when that contract starts.

I was imagining that the only way to maintain that contract following an update would be to reload the app; it would reload in order to reinitializing the stack with the updated static memory now hosting the new value potentially, in a new location?

That all said, without having yet read the docs for one_cell::lazy, I’m starting to guess why it might be intrinsically better suited for the task of providing global access to a single copy of memory; avoid reload following an update?

A lazy_static is a static variable (appropriately synchronized in order to make it memory-safe). There is no magic behind it. Its memory location doesn't change if you re-assign a new value to it. I don't follow why you think assignment changes the memory location, but that's most certainly not the case.

The only reason why I prefer once_cell::Lazy over lazy_static! is that it's just a regular type with a regular getter method, it's not hidden behind several layers of macro soup. A static variable typed once_cell::Lazy doesn't behave differently from a static variable created by lazy_static! or any other static for that matter, and neither of them changes its address nor do they require their address to be changed for mutation.

1 Like

Can memory location on the stack be guaranteed to remain the same if the value changes?

Assignment means "replace the value at this address with this other value". It doesn't mean "change the address". If that were the case, it would be borderline impossible to safely hand out &mut references, for instance.

Anyway, why are we talking about the stack? Statics aren't stored on the stack.

That's a core misunderstanding on my part and clarifies things. I got myopic with stack/heap. The static reference is stored in the static memory, the address to where it references is in the heap?

For some reason, I am stuck on the idea that if it's safe to mutate, you have to be on the heap. The "hangup": in the event the new value requires more space, Rust needs to have the freedom to allocate to a completely new address. Maybe clarified from a different angle: what can I safely mutate on the stack other than references?

If you have a reference of type &mut T, then you can only write to it if T: Sized, i.e., T has a single fixed size that can never change. The same applies to functions like std::mem::swap(), which also require T: Sized; otherwise, one could swap, e.g., a &mut [u8] with another &mut [u8] of a different size, which would cause all sorts of issues. The only types which allow their contents to grow (like Vec) use the borrow checker to make sure that no references are still live.

It is not. There is no "the" reference. There is a value inside the static variable, and you can take a reference to it as many times as you would like. That reference would be stored wherever you want. You could put it in a variable on the stack (or in a register), or you could even put the reference itself on the heap if you wanted to (by eg. putting it in a box or vec).

No, statics are in static memory (surprise), not in dynamic memory (the heap). They are not allocated and freed at runtime. The compiler/linker describes the size and initializing contents of every static variable at compile time, and that is baked into the executable.

The new value doesn't require more space. Types have constant sizes known at compile time. (Except unsized/dynamically-sized types, which can't be stored in variables by-value anyway.)

If you are thinking "but what about Vec?", then the answer is: Vecs always have the same size. If you put a Vec in a static, then the (pointer, length, capacity) triple will live in the static, and the pointer will point to the heap. But this has nothing to do with statics whatsoever. This is how vectors are represented, regardless of whether you put them in a local variable, a temporary expression, a static, a field in a structure, etc.

Um, anything…? As long as you don't have other references to a value (ie. no aliasing), and the value isn't declared as immutable, you can mutate it.

1 Like

That's a good reminder when T isn't a ref itself. I mentally moved away from thinking about reusing fixed-sized blocks of memory like I did with C arrays. This said, I sometimes find myself using a buffer of fixed size... But even there, in Rust, I'm convinced not to worry about it because Vec and String do a good job of figuring out the best initial dynamic (if you will) allocation on the heap (required location when the vector length can't be known at compile time). C had me so focused on arrays, in the above discussion about mutating memory, I was more focused on the exception (Vec), than the rule: enums and structs!

This I get; it only hosts information about the memory and a allocator to effect changes as needed. Vec is fixed; what it points to can change; it's a smart C array pointer with an allocator.

Yup, got it per my comment of focusing on the exception of Vec over the rule.

Go it. In the lazy_static! example, my Options struct consumes a fixed amount of space in the static memory. I can mutate the value in the static memory as long as exclusive access is assured.

Thank you!! This changed a couple of "mental models" for me:

  1. think first, the size of memory is constant; mutating that memory does not risk changing the location in memory (think enums and structs before Vec and friends with allocators; they are the exception)

  2. similarly/thus changing the value of a static value does not require rebuilding anything (the only guarantee is that that memory will exist as long as the app); it says nothing about where refs to that memory can be stored)

The patience and sharp observations present in this forum are always truly impressive. Thank you again.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.