Mysterious resets, can't reproduce

code review
1

I have my project that I've been working on for almost two years (although, I took a sabbatical from it last year, so say little less than a year actual work?), and from very early on, I've had these .. "mysterious" resets!

I was working on the assumption that it was a hardware issue almost from the beginning - using ports that shouldn't/couldn't work together etc. This because it started almost the minute I added the second-to-last piece, the actuator controller. Maybe the actuator sent back a spike, or power drain or something when it started moving, which triggered it - they are powered from the same power supply in the end :slight_smile: .

I successfully managed LEDs, buttons, MOSFET "relays", multi-colour LEDs (Neopixel), reading and writing to the flash etc etc. All working fine, in both individual test programs just to get the hang of it, then merging that into the big program - which I call "The Big Kahuna" :slight_smile:.

But as soon as I added the actuator code (which worked flawlessly on its own), all hell broke loose!

Ok, cutting the story shorter maybe, I still have these resets, but it's now not so mysterious anymore. I've fixed almost everything, and it is rock solid, UNTIL (!) I re-enable the flash manager in part of the code! It works in five+ other places (exact same code!).

This is what stumps me - it works in four separate binaries (which uses ONLY the flash, nothing else!), plus one or two that I've written to try to reproduce the problem, and in ONE of the two functions in "The Big Kahuna"!

But in one place, it resets (as in, not crashes, as far as I can tell, I think I would see that in my DEFMT serial output) as soon as it tries to access the flash. In the other, it works just fine.

I'm new to the forum, so have a limit on how many links I can include. But this is the main link to the repo:

The problem is “somewhere” in the src/lib_config.rs:read() function (I think).

In main(), I spawn off a “core1 manager", which spins up other functions on the second core. One of them is the src/lib_actuator.rs:actuator_control() function.

So I setup the flash from main in src/lib_config.rs:init_flash(). It gives the flash Mutex handler back to main(), which gives it to src/lib_core1.rs:core1_tasks() (as a borrow), that then gives it to src/lib_actuator.rs:actuator_control() (as a borrow), which in turn calls src/lib_config.rs:DbwConfig::read(), which calls embassy_rp::flash::Blocking::blocking_read().

main → core1_tasks → actuator_control → read → blocking_read

That's where it resets!

HOWEVER, after the CORE1 tasks have been spawned, a little later in main(), I also spawn off four src/lib_buttons.rs:read_button() (which each spawn one src/lib_buttons.rs:set_led() - which seems irrelevant, but I thought I mention it). They (src/lib_buttons.rs:read_button()) also each get the flash Mutex handler (as a borrow).

main → read_button (x4) → read → blocking_read

There I do EXACTLY like I do in src/lib_actuator.rs:actuator_control() (besides do some magic regarding the buttons and LEDs), then (in both functions) I read the flash, change the values, and then write them back to the flash.

As far as I can see (and I've been spending all day looking for any nuance, or small difference, but haven't seen any), the code in src/lib_actuator.rs:actuator_control() and src/lib_buttons.rs:read_button() are identical! But in the former it (the Pico) resets (I've added some debug outputs, and it's the blocking_read() that resets it), in the latter it works flawlessly!

I would really appreciate if someone could put a second (or third :slight_smile: ) pair of eyes on this.. I'm very likely doing something wrong - please bare with me, I'm fairly new in Rust, although I'm a “fair to decent” C developer (among maybe another five script-/interpreted languages I use regularly), so ...

Don't be gentle, tell it to my face, I can take it :slight_smile: . But understand where the code comes from..

2

Try adding a noticeable sleep between flash.lock().await and DbwConfig::read(&mut flash) as another task might be faulty.

I'm also really suspicious of those static mut BUTTONS_BLOCKED, BUTTON_ENABLED which you read and write without synchronization.

3

Yeah, I didn't like those either, not even when I added them :slight_smile: .

But I could think of no other way, and then I forgot about them, because I have the stability issue to deal with.

Suggestions on how to do that instead?

Unfortunately, adding a .5s delay after every lock didn't change anything :frowning:. Nor 1.5s :frowning: :frowning: .

4

but does it crash before, during or after sleep?

5

After. I get the debug output between the timer and the read..

1 Like
6 
            trace!(":: Actuator locking flash mutex");
            let mut flash = flash.lock().await;
            Timer::after_millis(1500).await;
            trace!(":: Actuator locked flash mutex, reading config from flash");
            let mut config = match DbwConfig::read(&mut flash) {        // !!TODO: CRASHES!!
7

I'd suggest getting a shift register to free some pins. That's not an explanation to what conflict occurs here, though...

8

I have considered it, and I'm looking at alternatives.

When I first started to suspect the flash code (only two, three weeks ago!), I started digging, and I found (very conflicting!) information that you can't use "certain" (which ones depends on the site) GPIOs and/or (!?) ADCs at the same time you're using the flash.

I have another question about that here, but nothing conclusive yet.

BUT, because it works in read_button().. That don't seem plausible. Same program, just another task. And core, but it didn't work before I moved it to core1 either, so..

9

So I've got a successful response on the other thread.

Seems you have to access the flash from CORE0 only!