They certainly can happen, but they all indicate that something bad has happened to the filesystem after the program started: Either the user has done something like delete the directory, or there’s been a hardware failure.
If your program is expected to run unassisted, there’s not much you can do to recover— panicing is often the only reasonable course. On the other hand, if you have some kind of interactive session with the user, you can ask them for a different directory to switch to.