Problem in Watching a TOML File and Updating Global State – Inconsistent Test Behavior

Hello friends,

I'm new to watching files and updating global state in Rust, and I'm facing some issues. I’d appreciate your help!

My goal is to read a TOML file from a specific path, store its data in a global state, and then watch the file for changes. Whenever the file updates, I want to modify the state so that the latest configuration is always available.

Since this is a library, and many functions will need access to this state, I decided to store it in a global variable.

Here’s the code I wrote with the help of ChatGPT (since I’m new to Rust):

use notify::{EventKind, RecommendedWatcher, RecursiveMode, Watcher};
use serde::{Deserialize, Serialize};
use std::{
    fs,
    path::Path,
    sync::mpsc::{self, Receiver, Sender},
    sync::{Arc, Mutex, OnceLock},
    thread,
};

#[derive(Debug, Deserialize, Clone, Serialize)]
pub struct Config {
    pub database_url: String,
    pub port: u16,
}

static GLOBAL_CONFIG: OnceLock<Arc<Mutex<Config>>> = OnceLock::new();

pub struct ConfigManager;

impl ConfigManager {
    pub fn init(file_path: &str) -> Receiver<()> {
        let new_config = Self::load_config(file_path);
        let config = Arc::new(Mutex::new(new_config.clone()));

        if GLOBAL_CONFIG.get().is_some() {
            *GLOBAL_CONFIG.get().unwrap().lock().unwrap() = new_config;
        } else {
            GLOBAL_CONFIG.set(config).ok().unwrap_or(());
        }

        let file_path = file_path.to_string();
        let (tx, rx) = mpsc::channel();

        let tx_clone = tx.clone();

        thread::spawn(move || {
            Self::watch_file(file_path, tx);
        });

        let _ = tx_clone.send(());

        rx
    }

    fn load_config(file_path: &str) -> Config {
        if !Path::new(file_path).exists() {
            println!(
                "Config file not found. Creating default config at: {}",
                file_path
            );
            let default_config = Config {
                database_url: "sqlite://default.db".to_string(),
                port: 8080,
            };
            let toml_string =
                toml::to_string(&default_config).expect("Failed to serialize default config");
            fs::write(file_path, toml_string).expect("Failed to write default config file");
            return default_config;
        }

        let content = fs::read_to_string(file_path).expect("Failed to read config file");
        toml::from_str(&content).expect("Invalid TOML format")
    }

    fn watch_file(file_path: String, change_notifier: Sender<()>) {
        let (tx, rx) = mpsc::channel();
        let mut watcher = RecommendedWatcher::new(tx, notify::Config::default()).unwrap();

        watcher
            .watch(Path::new(&file_path), RecursiveMode::NonRecursive)
            .unwrap();

        for event in rx {
            if let Ok(event) = event {
                if matches!(event.kind, EventKind::Modify(_)) {
                    println!("Config file changed, reloading...");
                    let new_config = Self::load_config(&file_path);

                    if let Some(config) = GLOBAL_CONFIG.get() {
                        let mut config_lock = config.lock().unwrap();
                        *config_lock = new_config;
                    }

                    let _ = change_notifier.send(());
                }
            }
        }
    }

    pub fn get_config() -> Config {
        GLOBAL_CONFIG
            .get()
            .expect("ConfigManager::init must be called first!")
            .lock()
            .unwrap()
            .clone()
    }
}

The problem I'm facing is that my tests fail inconsistently. Sometimes they pass, and sometimes they fail. I suspect that the test might not be detecting file changes correctly.

Has anyone encountered this issue before? What’s the best approach to ensure reliable testing for file change detection?

my tests

#[cfg(test)]
mod tests {
    use super::*;
    use std::fs::File;
    use std::io::Write;
    use std::time::Duration;

    #[test]
    fn test_config_initialization() {
        let test_config_path = "test_config.toml";
        let mut file = File::create(test_config_path).unwrap();
        writeln!(file, "database_url = \"sqlite://test.db\"\nport = 1234").unwrap();

        let rx = ConfigManager::init(test_config_path);

        rx.recv_timeout(Duration::from_secs(2)).unwrap();

        let mut attempts = 5;
        let mut config = ConfigManager::get_config();

        while config.database_url != "sqlite://test.db" && attempts > 0 {
            std::thread::sleep(Duration::from_millis(200));
            config = ConfigManager::get_config();
            attempts -= 1;
        }

        assert_eq!(config.database_url, "sqlite://test.db");
        assert_eq!(config.port, 1234);

        std::fs::remove_file(test_config_path).unwrap();
    }

    #[test]
    fn test_config_update_on_file_change() {
        let test_config_path = "test_config_update.toml";
        let mut file = File::create(test_config_path).unwrap();
        writeln!(file, "database_url = \"sqlite://old.db\"\nport = 1111").unwrap();

        let rx = ConfigManager::init(test_config_path);

        rx.recv_timeout(Duration::from_secs(2)).unwrap();

        let mut file = File::create(test_config_path).unwrap();
        writeln!(file, "database_url = \"sqlite://new.db\"\nport = 2222").unwrap();

        rx.recv_timeout(Duration::from_secs(2)).unwrap();

        let config = ConfigManager::get_config();
        assert_eq!(config.database_url, "sqlite://new.db");
        assert_eq!(config.port, 2222);

        std::fs::remove_file(test_config_path).unwrap();
    }
}

All the code: config.rs · GitHub

Thanks in advance!

I've not used the file watcher library in your code, but based on your description, I would guess possibly the file system cache is not being flushed.

if you want to use the write!() macro to write formatted data, try manually drop the file object. note, calling Write::flush() may not work, I don't think it actually flushes the underlying file descriptor.

let mut file = File::create(test_config_path).unwrap();
writeln!(file, "database_url = \"sqlite://new.db\"\nport = 2222").unwrap();
drop(file);

alternatively, if you don't actually need to format the content, you can simply use std::fs::write():

std::fs::write(
    test_config_path,
    "database_url = \"sqlite://new.db\"\nport = 2222"
).unwrap();

Thank you but I still have this problem sometime it passes all 2 tests and sometimes fails all the test or one of them

I do not know it is right thing I am using or not:

static GLOBAL_CONFIG: OnceLock<Arc<Mutex<Config>>> = OnceLock::new();

can you provide more details? in particular, which assertion would fail in each test case, respectively? or is it inconsistent and every assertion could potential fail?

this type is not the best choice in terms of performance overhead, since it has two layer of synchronization (OnceLock and Mutex) and an extra layer of indirection (Arc), but it is OK in terms of correcness. so I doubt this is the cause of the problem.

to reduce the overhead, I would suggest just use Mutex<Option<T>>. it can be used like this:

static GLOBAL_CONFIG: Mutex<Option<Config>> = Mutex::new(None);

fn set(config: Config) {
    *GLOBAL_CONFIG.lock().expect("failed to lock mutex") = Some(config);
}

fn get() -> Config {
    GLOBAL_CONFIG
    .lock()
    .expect("failed to lock mutex")
    .clone()
    .expect("not initialized")
}
1 Like

For example, I ran the test once, and both tests passed successfully. Now, I ran the test again, and as you can see, the first test passed successfully, but the second test failed.

running 2 tests
test config::tests::test_config_update_on_file_change ... ok
test config::tests::test_config_initialization ... FAILED

failures:

---- config::tests::test_config_initialization stdout ----

thread 'config::tests::test_config_initialization' panicked at src/config.rs:129:9:
assertion `left == right` failed
  left: "sqlite://default.db"
 right: "sqlite://test.db"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    config::tests::test_config_initialization

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 1.03s

Now, I manually ran the test multiple times again. As you can see in the terminal, this time both tests completely failed.


running 2 tests
test config::tests::test_config_update_on_file_change ... FAILED
test config::tests::test_config_initialization ... FAILED

failures:

---- config::tests::test_config_update_on_file_change stdout ----
Config file changed, reloading...
Config file changed, reloading...

thread 'config::tests::test_config_update_on_file_change' panicked at src/config.rs:157:9:
assertion `left == right` failed
  left: "sqlite://test.db"
 right: "sqlite://new.db"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- config::tests::test_config_initialization stdout ----
Config file changed, reloading...

thread 'config::tests::test_config_initialization' panicked at src/config.rs:129:9:
assertion `left == right` failed
  left: "sqlite://new.db"
 right: "sqlite://test.db"


failures:
    config::tests::test_config_initialization
    config::tests::test_config_update_on_file_change

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 9 filtered out; finished in 1.01s

error: test failed, to rerun pass `--lib`

There are a number of problems with this code. I'll point out a couple.

  1. The initialization of GLOBAL_CONFIG is not threadsafe, it has a race condition in this code:

        if GLOBAL_CONFIG.get().is_some() {
            *GLOBAL_CONFIG.get().unwrap().lock().unwrap() = new_config;
        } else {
            // <-- Other thread could have set a config at this moment,
            // after calling get() and before we call set
            GLOBAL_CONFIG.set(config).ok().unwrap_or(());
        }
  1. Since there is global state, you are likely to have nondeterministic test results. Rust by default runs unit tests concurrently in different threads. However in this case I tried cargo test -- --test-threads=1 and it was still failing, so that didn't fully solve it.

  2. Writing to the file may not be happening when you expect it to. In particular the writeln!() call may just write to an in-memory buffer in the application memory, and not be written to the file via OS system calls. You need to flush or close the file to ensure the write takes effect.

 let test_config_path = "test_config.toml";
        let mut file = File::create(test_config_path).unwrap();
        writeln!(file, "database_url = \"sqlite://test.db\"\nport = 1234").unwrap();

        let rx = ConfigManager::init(test_config_path);

I hope you continue your Rust learning journey, and try to start small and understand the language piece by piece. Generating code with AI is not a great idea unless you are already an expert in the language.

In many ways, it's harder to find problems with code and fix them than it is to write correct code in the first place.

1 Like

Here is the code with two modifications that make it possible to run the tests if you also run single threaded. So still not good code, but you hopefully can see what the problems are.

Fix some test issues so it's possible to pass. Still not good code though, it should be rewritten to be cleaner.

- close file after write (by wrapping in its own scope, when dropped the
  destructor will close and flush it) to ensure it's flushed to disk

- don't fail on rx recv timeout but ignore the timeout since there's a
  race condition in that the watcher might not see a change because it
  was reported before the watcher was listening

You must run the tests with the command `cargo test -- --test-threads=1`
since otherwise the static GLOBAL_CONFIG and the file on disk are shared
state that create a race condition that will randomly fail.

GitHub gist with these changes

1 Like

Thank you my friend, I tested your code and the code I edited, both of them worked with cargo test -- --test-threads=1, but without it they did not work. it means I have race condition in production or not safe for multi threads?

By the way, I used flush and or edited to this

std::fs::write(
            test_config_path,
            "database_url = \"sqlite://new.db\"\nport = 2222",
        )
        .unwrap();

But did not make any effect.

My new code without ARC: config.rs · GitHub

You might be able to make it threadsafe, but currently both test functions are writing to the same file, so they are going to interfere with each other through that, if not through other shared state.

I run each test on a separate file. If you mean that each test modifies the same file two or three times, then yes. This is because I want to watch whether the config file has changed so that I can bring the latest update back into the state.

For example:

let test_config_path = "test_config.toml";
// or
let test_config_path = "test_config_update.toml";

If you're getting a panic due to the unwrap on the RX wait, timing out, then it might be a race condition where the file is modified before the watcher is listening for the changes. Try ignoring the Err result instead of doing unwrap() if that's happening.

If i ignore this with Err, I am going to get this. unfortunately

---- config::tests::test_config_initialization stdout ----

thread 'config::tests::test_config_initialization' panicked at src/config.rs:147:9:
assertion `left == right` failed
  left: "sqlite://old.db"
 right: "sqlite://test.db"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- config::tests::test_config_update_on_file_change stdout ----
⚠️ Warning: Failed to receive config update: Timeout

thread 'config::tests::test_config_update_on_file_change' panicked at src/config.rs:180:9:
assertion `left == right` failed
  left: "sqlite://old.db"
 right: "sqlite://new.db"


failures:
    config::tests::test_config_initialization
    config::tests::test_config_update_on_file_change

test result: FAILED. 9 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.51s

I am still can not be able to find the last data that I should have after tracking my changed-file

From my last code that I could not be able to fix the problem: README.md · GitHub

As you can see, it got the old.db, but it is not for this test.

I think I could not handle the race condition or I have another problem!!

since it's a race condition between different test cases, why not use different file names for each test case? either make manually select some different file names, or you can just generate random names every time. for example, checkout tempfile:

Hi @nerditation,
I am testing different files, but my global state remains the same! It still sends the data to a single global state.

static GLOBAL_CONFIG: OnceLock<Arc<Mutex<Config>>> = OnceLock::new();

So, I think this is the issue I am facing.

oh I see.

you want isolated "global" states for each test case, and that's not possible if all the test cases are in the same process.

if you really don't want the --test-threads=1 approach, the only alternative I can think of is to put these test cases into separate files under the tests directory. (note, this is usually for integration tests, not unit tests).


side note

in some sense, your test cases are fundamentally flawed. they do not represent typical application requirements and use cases.

you should really think what you want to test exactly, and this might not be immediately clear for concurrent code.

for example, do you want to test if all file change notification will be received and none is lost? well, that's a test for the file watcher, not for your code, is it?

then, what does your code actually do? I don't really know, but guess from the code snippet in the post, I can break down what your code does roughtly like these:

  • users must initialize the config first before use
  • users can get (a "current" snapshot of) the configuration at any time after initialized
  • when the user initialize the config, the file is monitored for update
  • on receiving a file change event, the config file is reloaded
  • when the config is reloaded, the users shall get a notification

but with your API design, it is impossible (or at least not easy) to write unit tests for each individual functionality in isolation. so what you end up with is some kind of end to end integration tests after all.

1 Like