Abstract: The subject of the study is a potential  buffer overflow vulnerability in various software related to the function of the standard C / C ++ strcpy programming language library and approaches and methods for finding such vulnerabilities. The object of the study is the data of the machine code of the compilers when the program is assembled in various modes. The purpose of the study is to analyze some features of the machine code generated by various compilers for Windows and Linux in the Debug and Release modes, including, on the basis of this, a review of the buffer overflow vulnerability. Research methods. The paper reviews and develops methods for constructing algorithms for searching for buffer overflow vulnerabilities, examines the characteristics of this vulnerability at the level of machine code. This is done using the Visual C ++ compilers, Intel C ++ compilers, g ++ compilers, as well as the WinDBG, GDB debuggers. Key findings. Building programs in different modes leads to the formation of differences in the executable code, which is made from the completely same high-level programming language code; these differences manifest themselves in differences in program behavior. In the course of researching software in search of vulnerabilities, it is important to analyze computer code in order to identify hidden patterns. The novelty of the study lies in identifying differences in the machine code obtained after assembling the same high-level code, identifying compiler stamps when executing the assembly of the program in different modes. A special contribution of the author to the study of the topic is the development of methods for constructing algorithms for searching for buffer overflow vulnerabilities.

Abstract: The subject of the research is the optimization algorithms for automated dependency search on disassembled code. The object of the research is the dependent code blocks on the x64 architecture of Intel processors manufactured by the company and listings obtained as a result of reverse engineering software by compilers with different settings in Windows and Linux.Purpose of the study. The purpose of the study is to consider the possibility of using mathematical matrices to build a machine code map, and also to review possible problems for automatic analysis, to search for the paths of information flows. Research methods. We used the Visual C ++ compiler. We consider the architecture in which the transfer of information can be carried out in the following ways: register-memory, memory-register, register-register. For the analysis, the method of forming the list of functions called up to the potentially dangerous block of code under investigation, chosen for each considered path to the block under study, was chosen. Methods for implementing the matrix approach are described and developed. Novelty and key findings. Mathematical matrix method can be used to build a machine code map. However, determining the reachability paths of individual code blocks may require a significant amount of resources. In addition, the machine code can be exposed to packers and obfuscators, which also introduces additional complexity. A number of potentially dangerous functions of the standard library of the C / C ++ programming language were identified.