Memory Dump Analysis Anthology, Volume 1, Revised Edition
Memory Dump Analysis Anthology, Volume 1, Revised Edition
About the Book
This reference volume consists of revised, edited, cross-referenced, and thematically organized articles from Software Diagnostics Institute and Software Diagnostics Library (former Crash Dump Analysis blog) written in August 2006 - December 2007. This major revision updates tool information and links with ones relevant for Windows 10 and removes obsolete references. Some articles are preserved for historical reasons, and some are updated to reflect the debugger engine changes. The output of WinDbg commands is also remastered to include color highlighting. Most of the content, especially memory analysis pattern language, is still relevant today and for the foreseeable future. Crash dump analysis pattern names are also corrected to reflect the continued expansion of the catalog.
The primary audience for Memory Dump Analysis Anthology reference volumes is: software engineers developing and maintaining products on Windows platforms, technical support, escalation, and site reliability engineers dealing with complex software issues, quality assurance engineers testing software on Windows platforms, security and vulnerability researchers, reverse engineers, malware and memory forensics analysts.
About the Author
Table of Contents
Preface 19
Acknowledgments 21
About the Author 23
PART 1: Crash Dumps for Beginners 25
Crash Dumps Depicted 25
Right Crash Dumps 26
Crashes Explained 28
Hangs Explained 31
Symbol Files Explained 34
Crashes and Hangs Differentiated 36
Proactive Crash Dumps 39
PART 2: Professional Crash Dump Analysis 43
Minidump Analysis 43
Scripts and WinDbg Commands 43
Component Identification 46
Raw Stack Data Analysis 53
Symbols and Images 63
Interrupts and Exceptions Explained 68
Exceptions Ab Initio 68
X86 Interrupts 69
X64 Interrupts 76
Interrupt Frames and Stack Reconstruction 83
Trap Command on x86 92
Trap Command on x64 100
Exceptions in User Mode 104
How to Distinguish Between 1st and 2nd Chances 109
Who Calls the Postmortem Debugger? 113
Inside Vista Error Reporting 117
Another Look at Page Faults 132
Bugchecks Depicted 135
NMI_HARDWARE_FAILURE 135
IRQL_NOT_LESS_OR_EQUAL 136
KERNEL_MODE_EXCEPTION_NOT_HANDLED 141
KMODE_EXCEPTION_NOT_HANDLED 143
SYSTEM_THREAD_EXCEPTION_NOT_HANDLED 144
CAFF 150
CF 152
Manual Stack Trace Reconstruction 157
WinDbg Tips and Tricks 167
Looking for Strings in a Dump 167
Tracing Win32 API While Debugging a Process 168
Exported NTDLL and Kernel Structures 170
Easy List Traversing 178
Suspending Threads 181
Heap Stack Traces 182
Hypertext Commands 183
Analyzing Hangs Faster 187
Triple Dereference 188
Finding a Needle in a Hay 191
Guessing Stack Trace 193
Coping with Missing Symbolic Information 199
Resolving Symbol Messages 204
The Search for Tags 206
Old Dumps, New Extensions 212
Object Names and Waiting Threads 214
Memory Dumps from Virtual Images 219
Filtering Processes 220
WinDbg Scripts 221
First Encounters 221
Yet another WinDbg Script 222
Deadlocks and Critical Sections 223
Security Problem 224
Hundreds of Crash Dumps 227
Parameterized Scripts 229
Security Issues and Scripts 230
Raw Stack Dump of All Threads (Process Dump) 231
Raw Stack Dump of All Threads (Complete Dump) 236
Case Study 241
Detecting Loops in Code 244
Crash Dump Analysis Checklist 251
Crash Dump Analysis Poster (HTML version) 254
PART 3: Crash Dump Analysis Patterns 255
Multiple Exceptions 255
Dynamic Memory Corruption 257
False Positive Dump 259
Lateral Damage 264
Optimized Code 265
Invalid Pointer 267
Inconsistent Dump 269
Hidden Exception (User Space) 271
Deadlock (Critical Sections) 276
Changed Environment 283
Incorrect Stack Trace 288
OMAP Code Optimization 294
No Component Symbols 298
Insufficient Memory (Committed Memory) 302
Spiking Thread 305
Module Variety 310
Stack Overflow (Kernel Mode) 314
Deadlock (Executive Resources) 323
Insufficient Memory (Handle Leak) 327
Managed Code Exception 331
Truncated Dump 340
Waiting Thread Time (Kernel Dumps) 343
Deadlock (Mixed Objects, User Space) 348
Memory Leak (Process Heap) 356
Missing Thread 362
Unknown Component 367
Memory Leak (.NET Heap) 371
Double Free (Process Heap) 378
Double Free (Kernel Pool) 387
Coincidental Symbolic Information 390
Stack Trace 395
Virtualized Process (WOW64) 400
Stack Trace Collection (Unmanaged Space) 409
Coupled Processes (Strong) 419
High Contention (Executive Resources) 421
Accidental Lock 423
Passive Thread (User Space) 430
Main Thread 437
Insufficient Memory (Kernel Pool) 441
Busy System 449
Historical Information 458
Object Distribution Anomaly (IRP) 459
Local Buffer Overflow 461
Passive System Thread (Kernel Space) 462
Early Crash Dump 466
Hooked Functions (User Space) 469
Custom Exception Handler (User Space) 471
Deadlock (LPC) 474
Special Stack Trace 479
Manual Dump (Kernel) 480
Wait Chain (General) 482
Manual Dump (Process) 487
Wait Chain (Critical Sections) 490
PART 4: Crash Dump Analysis AntiPatterns 493
Alien Component 493
Zippocricy 494
Word of Mouth 495
Wrong Dump 496
Fooled by Description 497
Need the Crash Dump 498
Be Language 499
Fooled by Abbreviation 500
PART 5: A Bit of Science 501
Memory Dump - A Mathematical Definition 501
Threads as Braided Strings in Abstract Space 503
What is Memory Dump Analysis? 506
Memorillion and Quadrimemorillion 507
Four Causes of Crash Dumps 508
Complexity and Memory Dumps 510
What is a Software Defect? 511
PART 6: Fun with Crash Dumps 513
Dump Analysis and Voice Recognition 513
Sending SMS Messages via Dumps 514
WinDbg as a Big Calculator 515
Dumps, Debuggers, and Virtualization 516
Musical Dumps 518
Debugging the Debugger 519
Musical Dumps: Dump2Wave 521
Dump Tomography 522
The Smallest Program 523
Voices from Process Space 526
Crash Dump Analysis Card 528
Listening to Computer Memory 529
Visualizing Memory Dumps 532
Visualizing Memory Leaks 544
Picturing Computer Memory 556
Unicode Illuminated 559
Teaching Binary to Decimal Conversion 560
Crash Dumps and Global Conspiracy 561
PART 7: WinDbg For GDB Users and Vice Versa 563
AT&T and Intel Syntax 563
Installation 565
Disassembler 568
Stack Trace (Backtrace) 573
Local Variables 581
PART 8: Software Troubleshooting 589
Four Pillars 589
Five Golden Rules 590
Critical Thinking 591
Troubleshooting as Debugging 592
PART 9: Reversing and Reconstruction 593
Pooltags 593
The List of Services 594
Reverse Engineering Component Dependencies 596
PART 10: Security 599
Memory Visualization 599
WinDbg is Privacy-Aware 600
Crash Dumps and Security 604
PART 11: The Origin of Crash Dumps 605
JIT Service Debugging 605
Local Crash Dumps in Vista 606
COM+ Crash Dumps 607
Correcting Microsoft Article about Userdump.exe 612
Where did the Crash Dump Come from? 616
Custom Postmortem Debuggers in Vista 618
Resurrecting Dr. Watson in Vista 621
Process Crash - Getting the Dump Manually 624
Upgrading Dr. Watson 627
Savedump.exe and Pagefile 628
Dumping Vista 629
Dumping Processes without Breaking Them 631
Userdump.exe on x64 632
NTSD on x64 Windows 633
Need a Dump? Common Use Cases 634
PART 12: Tools 635
Memory Dump Analysis Using Excel 635
TestDefaultDebugger.NET 636
Cons of Symbol Server 637
StressPrinters: Stressing Printer Autocreation 638
InstantDump (JIT Process Dumper) 639
TestDefaultDebugger 641
DumpAlerts 643
DumpDepends 644
Dump Monitor Suite 645
SystemDump 646
PART 13: Miscellaneous 649
What is KiFastSystemCallRet? 649
Understanding I/O Completion Ports 653
Symbol File Warnings 656
Windows Service Crash Dumps in Vista 658
The Road to Kernel Space 664
Memory Dump Analysis Interview Questions 665
Music for Debugging 666
PDBFinder 667
When a Process Dies Silently 668
ASLR: Address Space Layout Randomization 673
Process and Thread Startup in Vista 678
Race Conditions on a Uniprocessor Machine 680
Yet Another Look at Zw* and Nt* Functions 683
Programmer Universalis 686
Dr. Watson Logs Analysis 687
Post-Debugging Complications 690
The Elements of Crash Dump Analysis Style 691
Crash Dump Analysis in Visual Studio 692
32-bit Stack from 64-bit Dump 694
Asmpedia 695
How WINE Can Help in Crash Dump Analysis 696
Horrors of Debugging Legacy Code 697
UML and Device Drivers 699
Statistics: 100% CPU Spread over all Processes 702
Appendix 703
Crash Dump Analysis Portal 703
Reference Stack Traces 706
Index of WinDbg Commands 707
Cover Images 711
Other books by this author
The Leanpub 45-day 100% Happiness Guarantee
Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms
Do Well. Do Good.
Authors have earned$10,433,845writing, publishing and selling on Leanpub, earning 80% royalties while saving up to 25 million pounds of CO2 and up to 46,000 trees.
Learn more about writing on Leanpub
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers), EPUB (for phones and tablets) and MOBI (for Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
Top Books
- #1
Learning Drupal 9 as a framework
Stef Van LooverenThis course will teach you advanced concepts of drupal 9, Object-oriented PHP and Symfony components. After the course, you’ll be able to build robust and scalable software solutions of many kinds.
- #2
Continuous Delivery Pipelines
Dave FarleyThis practical handbook provides a step-by-step guide for you to get the best continuous delivery pipeline for your software.
- #3
Practical FP in Scala: A hands-on approach
Gabriel VolpeA practical book aimed for those familiar with functional programming in Scala who are yet not confident about architecting an application from scratch.
Together, we will develop a purely functional application using the best libraries in the Cats ecosystem, while learning about design patterns and best practices.
- #4
Ansible for DevOps
Jeff GeerlingAnsible is a simple, but powerful, server and configuration management tool. Learn to use Ansible effectively, whether you manage one server—or thousands.
- #5
C++20
Rainer GrimmC++20 is the next big C++ standard after C++11. As C++11 did it, C++20 changes the way we program modern C++. This change is, in particular, due to the big four of C++20: ranges, coroutines, concepts, and modules.
- #6
Introducing EventStorming
Alberto BrandoliniThe deepest tutorial and explanation about EventStorming, straight from the inventor.
- #7
Advanced Web Application Architecture
Matthias NobackThe missing manual for making your web applications future-proof
- #8
Deep Learning with PyTorch Step-by-Step
Daniel Voigt GodoyIn 2019, I published a PyTorch tutorial on Towards Data Science and I was amazed by the reaction from the readers! Their feedback motivated me to write this book to help beginners start their journey into Deep Learning and PyTorch. I hope you enjoy reading this book as much as I enjoy writing it.
- #9
Visualise, document and explore your software architecture
Simon BrownA short guide to visualising, documenting and exploring your software architecture.
- #10
R Programming for Data Science
Roger D. PengThis book brings the fundamentals of R programming to you, using the same material developed as part of the industry-leading Johns Hopkins Data Science Specialization. The skills taught in this book will lay the foundation for you to begin your journey learning data science. Printed copies of this book are available through Lulu.
- #11
C++ Best Practices
Jason TurnerLevel up your C++, get the tools working for you, eliminate common problems, and move on to more exciting things!
- #12
Functional Design and Architecture
Alexander GraninSoftware Design in Functional Programming, Design Patterns and Practices, Methodologies and Application Architectures. How to build real software in Haskell with less efforts and low risks. The first complete source of knowledge.
- #13
The Hundred-Page Machine Learning Book
Andriy BurkovEverything you really need to know in Machine Learning in a hundred pages.
- #14
CCIE SP v5.0
Łukasz Bromirski, Piotr Jablonski, and Nicholas RussoAre you striving to prepare to and pass CCIE SP lab exam? Take the opportunity and get this workbook! With the attached initial cfg files you will prepare yourself for the CCIE SP exam as well as learn SP technologies applicable to all kinds of today modern networks! This workbook covers blueprint topics and provides challenging examples.
- #15
Technical leadership and the balance with agility
Simon BrownA developer-friendly, practical and pragmatic guide to lightweight software architecture, technical leadership and the balance with agility.
- #16
The Simple Haskell Handbook
Marco SampellegriniA project-driven approach to practical Haskell development. Start from zero lines of code and finish with a working CI Server. Step by step. One type error at a time.
- #17
Atomic Kotlin
Bruce Eckel and Svetlana IsakovaFor both beginning and experienced programmers! From the author of the multi-award-winning Thinking in C++ and Thinking in Java and a Kotlin team member comes a book that breaks concepts into small, easy-to-digest "atoms," along with exercises supported by hints and solutions directly inside IntelliJ IDEA! Full support at www.AtomicKotlin.com.
- #18
Everyday Rails - RSpecによるRailsテスト入門
Junichi Ito (伊藤淳一), AKIMOTO Toshiharu, 魚振江, and Aaron SumnerRSpecを使ってRailsアプリケーションに信頼性の高いテストを書く実践的なアドバイスを提供します。詳細で丁寧な説明は本書のオリジナルコンテンツです。また、説明には実際に動かせるサンプルアプリケーションも使用します。本書は2017年版にアップデートされ、RSpec 3.6やRails 5.1といった新しい環境に対応しています！さあ、自信をもってテストできるようになりましょう！
- #19
Rector - The Power of Automated Refactoring
Matthias Noback and Tomas VotrubaLearn how to automatically and continuously upgrade and improve your PHP code base
- #20
Ansible for Kubernetes
Jeff GeerlingAnsible is a powerful infrastructure automation tool. Kubernetes is a powerful application deployment platform. Learn how to use these tools to automate massively-scalable, highly-available infrastructure.
Top Bundles
- #1
Software Architecture for Developers: Volumes 1 & 2 - Technical leadership and communication
2 Books
"Software Architecture for Developers" is a practical and pragmatic guide to modern, lightweight software architecture, specifically aimed at developers. You'll learn:The essence of software architecture.Why the software architecture role should include coding, coaching and collaboration.The things that you really need to think about before... - #2
CCIE Service Provider Ultimate Study Bundle
2 Books
Piotr Jablonski, Lukasz Bromirski, and Nick Russo have joined forces to deliver the only CCIE Service Provider training resource you'll ever need. This bundle contains a detailed and challenging collection of workbook labs, plus an extensively detailed technical reference guide. All of us have earned the CCIE Service Provider certification... - #9
"The C++ Standard Library" and "Concurrency with Modern C++"
2 Books
Get my books "The C++ Standard Library" and "Concurrency with Modern C++" in a bundle. The first book gives you the details you should know about the C++ standard library; the second one dives deeper into concurrency with modern C++. In sum, you get more than 600 pages full of modern C++ and about 250 source files presenting the standard library... - #10
EA for Programmers
2 Books
For those who want to get most of Enterprise Architects API and bejond, this bundle offers an unbeatable resource. While Scripting EA introduces the API step by step, Inside EA will dive even deeper into EA's database structure. Knowing this will let you do things that are not available through the API. Or simply you can boost the performance...
