Reverse Engineering

Mark McGuill

The Me Slide
Virus Analyst with Symantec Corp. (Security Response

Dept.) Hobbyist Programmer from the C64 days Experienced C/C++ developer (Industrial Experience with Microsoft (Yes, The Beast)) NUI Maynooth Grad (CSSE 2004) MSc. in Security & Forensic Computing from DCU How things work (Gory Details) Perhaps a bit anal! Just another bored hacker Not a public speaker! (show me to my cave)

Introduction
"Reverse engineering is the process of
analyzing a subject system to create representations of the system at a higher level of abstraction."
Chikofsky and Cross

Introduction

Reverse Engineering is an Art Reverse Engineering is a Science Reverse Engineering is Fun! Principal: How does that work? Challenge. Remove the black magic, find bugs, crack protection schemes (only unjust ones of course) Get Closer to your machine Very Complex but Rewarding Puzzles

Overview
Not too technical (hopefully) Knowledge of C or Assembly & the x86

architecture would be nice, but well keep it simple Mostly a quick walkthrough & some demos Pique your interest! As Peter from Family Guy would say Im freakin winging this!

Overview

Why RE? Legalities A Brief History of RE Types of RE (Static / Live) Design of a Disassembler for Intel x86 (16/32 bit) (Win32/PE) RE in Action (Simple & Real World Demos) Q&A session

Why Reverse Engineer?

Fun / Challenge Compatibility / Interoperability
IBM PC Clone (Compaq BIOS) (Clean Room / Chinese Wall) OpenOffice.org / Microsoft Office File Formats (SUN Dublin) Windows File Sharing (Samba) Protocol RE

Lost Source Code (This is common) Legacy Code (Original Coder Unavailable, No Source,

Y2K) Bug Hunting (Again, no source. Popular now with malware authors. New Spl01tz!) Virus Analysis

Why Reverse Engineer?

Sony BMG DRM rootkit! (Last Week!)
Excellent Reverse Engineering work by Mark Russonovich of SysInternals First4Internet designed as DRM enforcer Will hide any file / reg key / process whose name begins with $sys$ Will screw your windows box if you delete the drivers

Clean Room (aka Chinese Wall)

Two teams operate completely independent of each

other Team A reverse engineer subject & produce spec. Spec is legally inspected for original subject leakage Spec is passed to Team B who engineer a functionally equivalent solution No Copyright / trade secret infringements. Does not hold for patents! (Independent Invention not a defence)

Legalities
US DMCA (Danger Here)
Exemption acts of reverse engineering aimed at interoperability of file formats and protocols The exception allows reverse engineering of computer programs if the reverse engineer lawfully obtains the program, seeks permission from the copyright owner, only uses the results of their efforts to create an interoperable computer program and does not publish the results". Jailed for several weeks, detained for 5 months Advanced E-book Processor (ElcomSoft Co. Ltd) Convert Adobe E-book to PDF (Removing Restrictions) Dec 2002 Jury Acquitted Elcomsoft of all charges

Dmitry Sklyarov @ DEFCON (July 2001)

Legalities

EU - COUNCIL DIRECTIVE of 14 May 1991 on the legal protection of computer programs (91/250/EEC)

Article 5 - Exceptions to the Restricted Acts

1. Alterations for where necessary for intended purpose of

program, including error correction 2. The making of a back-up copy 3. Observe, study or testto determine the ideas and principles which underlie any element of the program

Article 6 - Decompilation

Ok for purposes of Interoperability

A Brief History
First real decompilers occur in 1960s
Translation of programs from 2nd to 3rd generation computers (Transistors to ICs)

D-Neliac decompiler, 1960

Neliac is an Algol style language invented 1955

G.L. Hopwood phD dissertation, 1978

7-stage decompiler Designed for portability & documentation

A Brief History
Zebra, 1981
Designed for Portability Input ULTRA/32 Assembler Output PDP11/70 Assembler Written by J. Reuter for Vax BSD 4.2 Input Object files with symbolic information Output C-like programs

Decomp, 1988

A Brief History
Exe2c, 1990
Implements some modern features Abandoned project

DCC, 1994
Christina Cifuentes excellent decompiler, most modern features, data flow, control flow analysis, data type recognition

Types of RE (Static / Live)

Live
Simple Observation After the fact checks (netstat, directory listings etc) Monitoring Tools (Ethereal, Regmon, Filemon, TDIMon) Debuggers (Softice, Ollydbg, VS, windbg) Disassembly, Decompilation Hex Editor, Strings tool, PEDump (Imports / Exports) Phoenix Disassembler / DSM Studio! Analysis

Static (More Difficult)

Types of RE (Live) Regmon

Types of RE (Live) - Ethereal

Types of RE (Live) - Debugger

Types of RE (Static)
Strings Tool (Unix tool, also Sysinternals) PEDump (Originally by Matt Pietrek) Hiew (Hackers View) Hex Editor Disassemblers
IDA Pro (www.datarescue.com) Commercial Phoenix Disassembler / DSM Studio! (Free!) www.minds.nuim.ie/~phoenix

Disassemblers (IDA Pro)

Disassemblers (IDA Pro)

FLIRT (Fast Library Identification Recognition

Technology) PIT (Parameter Identification & Tracking) Stack Variable Tracking (Arguments & Locals) Graphing Capability Cross References (X-Refs) Debugger too Programmable

Disassemblers (DSM Studio) aka Phoenix

Disassemblers (DSM Studio)

My Pet Project Intel x86 16/32 bit code FPU / MMX / SSE / SSE2 Extensions PE File Format Only Architecture is Open & Modular C / C++ (possibly C# soon, Analysis Engine) C Code is ANSI C conformant (Have got it working under Linux!) It works!

Designing A Disassembler A Walkthrough

System
Input: Executable File (PE Portable Executable File Format) Process Output: Assembly (Nice)

Designing A Disassembler A Walkthrough

PE File Format
Now Pretty Well Documented (Google it) Based on COFF Format Magic Numbers, Headers, Tables, Directories, Sections

Simply Overlay Data with C

Structures Load File as OS Loader Would Identify Entry Points (Default & Exported)

Designing A Disassembler A Walkthrough

Intel x86 Instruction Format
Excellently Documented Extended From 8086 format Lots of Hacks, Rules & Exceptions

Tough Parsing Code (ANSI C Conformant)

Standard, FPU, MMX, SSE, SSE2 Exception Tables ModRM & SIB encoding Tables

Never want to open that file again (~3K LOC excluding tables) Lots of hard coded tables

Main Function: DisassembleSingleInstruction()

Tables allow fast lookup

Designing A Disassembler A Walkthrough

Disassembly Engine (The Driver) Basically emulate a Processor (Sort of)
Follow instruction flow in BFS manner until termination Collecting external references (CALL, JMP, Jxx conditional) Termination conditions (RET, JMP) Repeat process for all external references GOTO 10! Merge any adjacent blocks

Designing A Disassembler A Walkthrough

Analysis Engine
Scan code for thunk jumps (Call -> JMP xxx) Call j_xxx Create Symbols Identify Data Types (DWORD / String etc) Identify High Level Functions Identify Stack Parameters Still a work in progress

Designing A Disassembler A Walkthrough

Output
Simple Transform Returned Listings & Structures to strings => stdout for CLI MFC GUI similar structure to string conversion Probably C# GUI Soon

Future Plans

High Level Code Conversion

0000 0003 0009 000E 0014 0000 0003 0009 000F 0015 001B 0021 0027 mov ecx, 5 push aHello call printf loop 00000003h ... cmp ecx, 100h jnz 001Bh push aYes call printf jmp 0027h push aNo call printf ... for(int i=0;i<5;i++) { printf(Hello); }

if(x == 256) { printf(Yes); } else { printf(No); }

DSM Studio - Future Plans

Graphing (Visual aid to disassembly) Binary Differencing Cross-referencing *nix version Support for other processors Support for other file formats Open source??? (License?) Scripting Developer Components

DEMOZ

Simple Demo (1)

Source for a simple Trojan Compiled binary Disassemble Analysis Associate high-level C with assembly Greater understanding

Simple Demo (2) Trojan Source

int main(int argc, char **argv) { WSADATA wsa; SOCKET s; struct sockaddr_in name; unsigned char buf[256];
// Initialize Winsock if(WSAStartup(MAKEWORD(1,1),&wsa)) return 1; // Create Socket s = socket(AF_INET,SOCK_STREAM,0); if(INVALID_SOCKET == s) goto Error_Cleanup; name.sin_family = AF_INET; name.sin_port = htons(PORT_NUMBER); name.sin_addr.S_un.S_addr = htonl(INADDR_ANY); // Bind Socket To Local Port if(SOCKET_ERROR == bind(s,(struct sockaddr*)&name,sizeof(name))) goto Error_Cleanup;

// Wait for connection from attacker if(INVALID_SOCKET == (g_current = accept(s,NULL,0))) goto Error_Cleanup;
// Send Welcome Message! if(SOCKET_ERROR == send(g_current,kszWelcome, strlen(kszWelcome),0)) goto Error_Cleanup; // Receive Command recv(g_current,buf,255,0); // Execute Command if('v' == buf[0]) DisplayVersion(); else if('d' == buf[0]) DisplayDiskSpace(); else if('l' == buf[0]) LaunchCalculator(); // Cleanup closesocket(g_current); WSACleanup(); return 0; Error_Cleanup: WSACleanup(); return 1; }

// Set Backlog parameters if(SOCKET_ERROR == listen(s,1)) goto Error_Cleanup;

Simple Demo (3) Compiled Binary

558BEC81ECA80200008D8570FEFFFF506801010000E8A402000085C0740AB8010000 00E9470100006A006A016A02E88502000089856CFEFFFF83BD6CFEFFFFFF7505E91E 01000066C7855CFEFFFF02006800080000E8580200006689855EFEFFFF6A00E844020 000898560FEFFFF6A108D8D5CFEFFFF518B956CFEFFFF52E82302000083F8FF7505E9 D80000006A018B856CFEFFFF50E80502000083F8FF7505E9C00000006A006A008B8D 6CFEFFFF51E8E5010000A3C4304000833DC4304000FF7505E99D0000008B956CFEFF FF52E8C00100006A00A18830400050E8EF01000083C404508B0D88304000518B15C4 30400052E89601000083F8FF7502EB666A0068FF0000008D8558FDFFFF508B0DC430 400051E86F0100000FB69558FDFFFF83FA767507E856000000EB240FB68558FDFFFF8 3F8647507E8C3000000EB110FB68D58FDFFFF83F96C7505E8200100008B15C430400 052E838010000E82101000033C0EB0AE818010000B8010000008BE55DC3CCCCCCCC CCCCCCCCCCCCCCCCCC558BEC81EC04010000FF15082040008985FCFEFFFF8B85FCF EFFFF25FFFF00000FB7C8C1E9080FB6D1528B85FCFEFFFF25FFFF00000FB7C881E1FF 0000000FB6D152688C3040008D8500FFFFFF50FF153C20400083C4106A008D8D00FF FFFF51E8EB00000083C404508D9500FFFFFF52A1C430400050E8930000008BE55DC3 CC558BEC81EC0001000057A1A4304000898500FFFFFF8B0DA8304000898D04FFFFFF 668B15AC30400066899508FFFFFFB93D00000033C08DBD0AFFFFFFF3ABAA6A008D8 500FFFFFF50E88300000083C404508D8D00FFFFFF518B15C430400052E82A0000005 F8BE55DC3CCCCCCCCCCCCCC558BEC6A0168B0304000FF15042040005DC3FF258C2 04000FF2588204000FF2584204000FF2580204000FF257C204000FF2578204000FF25 74204000FF2570204000FF256C204000FF2568204000FF2564204000FF2508204000F

Simple Demo (4) - Disassembly

push ebp mov ebp, esp sub esp, 2A8h lea eax, [ebp+0FFFFFE70h] push eax push 101h call 4012BEh test eax, eax jz 401028h mov eax, 1 jmp 40116Fh push 0 push 1 push 2 call 4012B8h mov dword ptr [ebp+0FFFFFE6Ch], eax cmp dword ptr [ebp+0FFFFFE6Ch], byte 0FFh jnz 401047h jmp 401165h mov word ptr [ebp+0FFFFFE5Ch], 2 push 800h call 4012B2h mov word ptr [ebp+0FFFFFE5Eh], ax push 0 call 4012ACh mov dword ptr [ebp+0FFFFFE60h], eax push 10h lea ecx, [ebp+0FFFFFE5Ch] push ecx mov edx, [ebp+0FFFFFE6Ch] push edx call 4012A6h cmp eax, byte 0FFh jnz 40108Dh jmp 401165h push 1 mov eax, [ebp+0FFFFFE6Ch] push eax call 4012A0h cmp eax, byte 0FFh jnz 4010A5h jmp 401165h push 0 push 0 mov ecx, [ebp+0FFFFFE6Ch] push ecx call 40129Ah mov [4030C4h], eax cmp dword ptr [4030C4h], byte 0FFh jnz 4010C8h jmp 401165h mov edx, [ebp+0FFFFFE6Ch] push edx call 401294h push 0 mov eax, [403088h] push eax call 4012D0h add esp, byte 4 push eax mov ecx, [403088h] push ecx mov edx, [4030C4h] push edx call 40128Eh cmp eax, byte 0FFh jnz 4010FFh jmp 401165h push 0 push 0FFh lea eax, [ebp+0FFFFFD58h] push eax mov ecx, [4030C4h] push ecx call 401288h movzx edx, byte ptr [ebp+0FFFFFD58h] cmp edx, byte 76h jnz 40112Ch call 401180h jmp 401150h movzx eax, byte ptr [ebp+0FFFFFD58h] cmp eax, byte 64h jnz 40113Fh call 401200h jmp 401150h movzx ecx, byte ptr [ebp+0FFFFFD58h] cmp ecx, byte 6Ch jnz 401150h call 401270h mov edx, [4030C4h] push edx call 401294h call 401282h xor eax, eax jmp 40116Fh call 401282h mov eax, 1 mov esp, ebp pop ebp ret

Simple Demo (5) - Analysis

DSM Studio

Real World Demo (1)

Typical Commercial Program Standard Static Analysis with Disassembler Live Debugging (Let the program lead us) Analyse & find the bug Break out our trusty hex-editor Perform Patch

Acknowledgements (greetz!)
Dave Cahill MINDS Crew Christina Cifuentes Phrack Magazine (l33t) Windows Hackers (Pietrek, Richter et al) Mum & Dad God, Buddha, Allah, Krishna, etc

Questions & Answers

??? Later: mmcguill@gmail.com Check out DSM Studio:

http://www.minds.nuim.ie/~phoenix