Given enough time any executable can be reverse engineered. All you can do is increase the expected amount of time it will take to reverse engineer it. There are several (non)commercial projects that can automatically obfuscate binaries. Some better than others. I don't have experience with any of them, so I can't recommend you any particular one.
I some times tell myself that the algo is complex, and the low-level code generated by the decompiler will not allow anybody to figure out the global logic behind, without several months of analysis. And before that, I will have published the source code and my paper.
I’d question how anyone could actually become “the official creator of the algo” just by publishing a paper about it faster; especially if you can easily demonstrate that the executable you published, at an earlier point, was clearly the same (hence possibly plagiarized) algorithm (which is easy to show if the source code is eventually published, and the build can be reproduced; you could also pre-publish a hash of the source code if you want to more convincingly have a proven point in time).
For source code / a written description of the algorithm this may be an option. Especially if you plan to open source your work.
Yeah. I've done that sort of thing. The only time that was especially difficult was with anti-piracy software specifically meant to interfere with a debugger.
For example, the operating system entry points cannot be removed from the executable. A hacker can easily and quickly figure out the points where your program is doing file I/O. Or network I/O. Or doing heap allocations. Those big course things allow bisecting the code until the interesting bits come into focus.
Also the nature of asymmetric crypto being what it is, you don't actually have to distribute a binary for people to test your claim of cracking the system - just ask them to submit public keys to you and give them back the private keys or forged messges. Or forge a message from a well-known public key like say, Google's.
I need feedback to improve the concept before publishing the paper.
I want people to first judge the result, and only then, the idea behind.
If I publish the source code of my project, before being known for being its original developer, many people will develop rapidly their own versions of it (not always traceable github forks). And I will just be a random guy on internet, screaming on all forums that I'm the first person to have that idea. Even if I have a proof of it, Nobody will really care of my claim. I don't need my project to be related to my real identity, but I can't even imagine that archnim (my online id) is not recognized as the author.
For something like this, I would reach out to trusted friends or colleagues in the industry. You can hopefully rely on them giving you valuable feedback without the risk of stealing your intellectual property.
You can also ask your lawyers to write up an NDA and require people to sign it before giving them access to your program - that'll give you a way to prove you were the original author.
That said, the only truly reliable way to protect your intellectual property is by making sure you are the only person that can run it... For example, by creating a server and putting all the logic behind an API. That way, nobody ever has access to a binary they can decompile to reverse-engineer your code in the first place.
As you only need proof that you wrote the algorithm trusted timestamping might be a solution for you. In short: You send a hash of your source code to a company that then sends you a signed timestamp back. With this you can proof that you owned the code at the specified time. I'm not a lawyer, but depending where you live the timestamp can be used in court.