Analyzing a software for know technical principals in it or its objects or making some changes in its behavior throuth analyses of its stucture, function and operation is Reverse engineering.
Software reverse engineering involves reversing a program's machine code (the string of 0s and 1s that are sent to the logic processor) back into the that it was written in, using program language statements. Some times software reverse engineering is done to retrieve the source code of a program because the source code was lost, to study how the program performs certain operations, to improve the performance of a program, to fix a bug (correct an error in the program when the source code is not available), to identify malicious content in a program such as a virus, or to adapt a program written for use with one microprocessor for use with a differently-designed microprocessor. Reverse engineering for the sole purpose of copying or duplicating programs constitutes a copyright violation and is illegal. In some cases, the licensed use of software specifically prohibits reverse engineering.
Specially reversing software for making some change in the code named Reverse Code Engineering.
A Reverse Engineer should have a good knlowledge about assembly language. Any software is open source for him.
For now we dont want to speak about assembly language you can refer to assembly language section for see available resources about it.
Here you'll be familar with some excellent tools that commonly used to Reversing softwares:
1- first you need debugger and/or disassembler for access the assembly codes behind of EXE file. Below is the link for 3 Excellent debugger and disassembler each of them has some advantage and some disadvantage.
OllyDebug (website: www.ollydbg.de ): OllyDbg is a 32-bit assembler level analysing debugger for Microsoft® Windows®. It can trace stack frames and detect some C function calls and also able to decode MMX, 3DNow! and SSE data types and instructions. With nice monitorng features it detects most of windows api functions and as i know most of reversers use this debugger as their favorite debugger. For more information refer to website of this debugger.
IDA Pro (website: http://www.datarescue.com/idabase ): IDA Pro is the best disassembler in the world. It is a Windows or Linux hosted multi-processor disassembler and debugger. In fact i dont like it's debugger but it can decode codes for various processor and able to detect so many C liberary functions also for other languages. Ida has so many features and it is not easy to describe them all. just download it and see what is its features.
Win32Disassembler (????): This is small disassembler/debugger with nice monitorng feauture (but not as well as Olly) and ability to detect some resource id referecnces in the source. many times it speeds up the process of reaching purpose. One of the disadvantages is, this program not supported more and it crashes while going to debug mode in large EXE files. For example you can not debug Yahoo Messenger or Paltalk Messenger using this debugger. You should use olly instead of this.
2- You may want to look at User Interface ID's for looking them in the code or you may want to change their Bitmaps or rearrange Buttons, change version info and ... . Recource Editors provide you to do so with the easiest way.
Restorator (Website: http://www.bome.com/Restorator/ ): This is nice tool for grabbing and editing windows PE executable files and also .exe, .dll, .res, .rc, .dcr files, change logos and get resource id's Translate existing applications (localization) Customize the look and feel of programs Replace logos and icons (branding) Enhance control over resource files in the software development process Hack into the inner workings of applications on your computer
Resource Hacker : This is a another small program that allow you to directly change the resource in EXE, DLL, OCX & CPL files and export their recource as RC files. Easy to use but some times it fail in large Resources and no longer supported.
XN Resource Editor (website: http://www.wilsonc.demon.co.uk/d10resourceeditor.htm) : This is another Resource editor written in Delphi. It hasn't Visual dialog editor but an Open source one and that gives you the ability to add any feature you like. You can get more info and also grab the source code from it's website.