I'm a "normal" programmer (oxymoron, I know).  I have no experience with hacking, but I need to get some information from some knowledgeable people on this subject.

I have a somewhat complex mathematical algorithm consisting of about 1000 lines of C++ code. If I compile and optimize it, how difficult will it be for someone else to reverse engineer it and discover how the algorithm works? Will it take a few days, or a few months, or a few years?

Any feedback will be greatly appreciated.

-k

Are you concerned about someone to understand how your algorithm works, or to get the actual (equivalent) code you wrote ?
From a technical point of view, you shall imagine someone with a debugger, that execute your program step-by-step and see the low level instructions as well as the changes to the data structures. The complexity of guessing how the algorithm works depends on your algorithm in this case.

Anyway I think you are more concerned about someone to use some automated process to de-compile your software and get your code or something equivalent. In that case you can consider to use some code-obfuscation technique.

I'm not saying it's a solution, i'm saying it may help.
Even if it's true that you don't deny the activity, you may have the goal to raise its complexity to the point that it doesn't worth the effort.

Anyway, if that software has to do its job on untrusted standalone machines, i don't see much more you can do about that... any ?

n0on3 - you mentioned obfuscation.  My understanding of obfuscation was that it was primarily helpful for interpreted code (Java, VB, etc.), since it mangled function and variable names.  Code that's been compiled to binary machine code should not have any meaningful names left, right?  It's all just push/pop/move/etc. statements, right?  Can an obfuscator still help?

Along those lines, can you recommend any good software protection programs?  I've tried VMProtect, but whatever it does to my program makes it look like a virus afterwards to some of the major AV packages.

First off I have to caveat this advice with the statement that I am a Reverse Engineering newbie.  I would say based on what you want accomplished "keeping someone from replacing the algorithm" and the complexity of the code in question that successful reverse engineering would depend mostly on your implementation.  Is this algorithm code going to be part of the main program executable or a separate file?  If it is included in the main exe and fully integrated into your program honestly very few people are ever going to be able to replace that algorithm by reverse engineering.  I am sure some experts would be able to but it would require significantly altering the flow of the executable using assembly code.  Now on the other hand if you put the algorithm in a separate file using a debugger it would be fairly easy to trace the program flow and change algorithms.  Of course it makes it a lot harder for you to ever upgrade your algorithm if you integrate it into your main executable.

Well that's my two cents.  Good Luck!

@Neophyte: I think the sample you posted can be reversed within a few hours - a few days, as assumed before. There were much stronger protections already reversed, so this shouldn't be too hard, though it really depends on the knowledge and experience of the reverser.