Development and Deployment of Multiplayer Online Games, Vol. III.
Development and Deployment of Multiplayer Online Games, Vol. III.
Server Arch, Fault Tolerance, Pre-Coding
About the Book
What's the Big Idea?
The idea behind this book is to summarize the body of knowledge that already exists on multiplayer games but is not available in one single place. And quite a few of the issues discussed within this series (planned as
three nine volumes ~300 pages each), while known in the industry, have not been published at all (except for maybe in discussion forums).
Programming and deploying an MOG is a daunting task, and having a frame of reference is usually quite helpful — "hey, those guys have already tried this, and it worked for them."
Vol. I-III of the series are about MOG Architecture. In Vol. I and II, we discussed general issues (including Communication flows) and Client-Side Architecture respectively. By the end of Vol. III, we'll complete our discussion of the things related to architecting the Server-Side of your MOG, and this will conclude the ARCH part of the series. Vol. IV-VI will describe issues related to Programming, and Vol. VII-IX will be about Deployment and post-deployment tasks.
The book has got endorsements from quite a few people known in the industry, including people from Amaya, Lloyds Bank, Lionhead Studios, BlueByte (Ubisoft Studio), Lightstreamer, Plarium, and (formerly) Zynga. For the endorsements themselves please see Kickstarter campaign.
And speaking of crowdfunding - the book has already been funded on Kickstarter and Indiegogo - with over 800 backers total (and counting). BTW, here on Leanpub there is a bundle which is exactly the same as an ongoing InDemand campaign on Indiegogo - $20 for Vol. I-III together with an early access (though on Indie there is also an option to pre-order paperback or even hardcover).
Since When Are Rabbits Programming??
I cannot speak for all of rabbitkind, but this particular hare has gone alongside my writings for quite a while now. He was born as "No MT Bugs" Hare in 2010, in an Overload article titled "Single-Threading: Back to the Future?"
Therefore, there is evidence that this particular rabbit has been programming at least since 2010. Yes, he has changed his appearance, but deep inside, he is still the very same bunny with a naïve idea of making programs less buggy. BTW, the character is created by Sergey Gordeev, a guy with many animation awards, known (among other things) for being a director of some of the Animated Mr. Bean series.
Who Is this Book For?
The book is intended for those intermediary developers who want to become senior ones, and for senior developers and up. I expect that the book will also be useful for decision-makers, from PMs all the way up to CEOs — so if you by any chance know one, please make sure to give her a link to this page ;-).
On the other hand, if you are working on your first programming project, this book will probably be too difficult for you. There will be no explanation on what event-driven programming is all about, what's the difference is between optimistic locking and pessimistic locking, why you need a source control system, and so on. Instead, there will be discussions on how the concept of futures fits into event-driven programming, when the use of optimistic locking makes sense for games, and how to use source control in presence of unmergeable files.
In the same spirit, the book is not intended to be a CD to copy-paste your sample MOG from. Instead, the book aims to help you write your own code that is tailored to your own needs, and tries to advise on the available choices at most of the junctions.
What's The Plan?
Vol. III is currently in "2nd beta", with 80% of the content I'm planning, already included. Still, there will be more explanations.
As I'm working with professional editors (and vast majority of them doesn't know anything but Word :-( ) - I keep and edit my manuscript as a bunch of Word .docx files (I throw myself at the mercy of the court, and assure that it is only because of dire circumstances beyond my control ). I'm using Word to export .docx to PDF, and some_VBA_scripts+Calibre to export to .MOBI and .EPUB. However, .EPUB generated by Calibre, doesn't pass checks here on LeanPub (though it seems to show ok on iBook and on Clearview). If you want .EPUB - drop me a line, I will e-mail it to you.
Please note that current formatting is quite ugly; it will improve after being professionally formatted; it is also expected to pass all the .EPUB validity checks.
- The manuscript needs to be finalized; currently - Vol. III is about 80% text-complete.
- The manuscript will be sent to the "line editor" (that's what that Kickstarter money was for :-)).
- Then - it will be sent to the typesetter
- Then - it will be finalized here on Leanpub (and rewards for Kickstarter and Indiegogo backers will be sent out too)
- And finally, it will be published on Kindle (and on Amazon via CS). At this point, price of the book may go higher here on Leanpub (to match Kindle price).
All the comments are extremely welcome - and they DO help making the book better.
Chapter 8. Server-Side MOG Architecture
In-Memory States and Multi-Player Games
‘No Bugs’ Rule of Thumb for Multiplayer Games
Good for Us (as developers)
Exception: Stock Exchanges
NOT applicable to Single-Player Games
In-Memory Game World States: a Natural Fit for ‘No Bugs’ Rule of Thumb
On Data Consistency
In-Memory State Summary
On Scalability and Load Balancing
Scaling Up? No Way, Jose. Scaling Out!
Load Balancing of Stateful Objects. Two flavors of Load Balancing
Deployment Architectures, Take 1
Don't Do It: Naïve Game Deployment Architecture
Semi-Naïve Deployment Architecture
Web-Based Game Deployment Architecture
Web-Based Deployment Architecture (Web Stack): How It Works
Caching, More Caching, and Even More Caching
Taming DB Load: Write-Back Caches and In-Memory States
Write-Back Caches: When to Write Back?
Write-Back Caches: Locking
Caches and Transactions
Web-Based Deployment Architecture: Reactors
Web Deployment Architecture: Scaling and Load Balancing
On Specific Web Servers
Web-Based Deployment Architecture: Merits
Classical Game Deployment Architecture
Implementing Game Servers under Reactor-fest architecture
Reactors with External Interfaces
TCP Sockets Reactors and TCP Accept Reactor
Websocket-related Reactors and HTTP-related Reactors (not shown)
GPGPU Reactor (not shown)
Reactor-Fest Architecture on the Server Side: Flexibility and Deployment-Time/Run-Tim
Threads and Processes
Underlying Communication Protocol as an Implementation Detail
Moving Game Worlds Around (at the cost of client reconnect)
Online Upgrades with (Almost-)Zero Downtime
On Importance of Flexibility
Classical Deployment Architecture: Scaling and Load Balancing
Scaling Game World Servers - Natural Linear Scalability (except for seamless MMOs)
MMOGs and Seamless Game World Servers
Moving Objects Around
Scaling Matchmaking Server
Scaling Other Game Servers
Regional Datacenters to Reduce Latencies
Once again on hybrid Web-Based+Classical Architectures
Classical Game Deployment Architecture: Summary
Enter Front-End Servers
Front-End Servers as Concentrators
Front-End Servers: Benefits
Front-End Servers: Drawbacks and Issues to Solve
Latencies and Latency Differences
Discussion on Scalability of Front-End Servers and Dealing with “N-squared”. Server Gro
Front-End Servers: Implementation
Front-End Servers: Reactor-fest Implementation
Routing&Data Reactors in Game Servers and Clients
Client Load Balancing
Client-Side Random Balancing
Client-Side Random Balancing: a Law of Large Numbers, and comparison with DNS Round-R
Server-Side Load Balancer Appliances
Load Balancing Summary
Front-End Servers as a kinda-CDN
Front-End Servers + Game Servers as a kinda-CDN
Front-End Servers Summary
DB Server API
Meanwhile, at the King’s Castle… Implementing DB Server
Multi-Connection DB Server implemented as DB Reactor
Single-connection DB Server as DB Reactor
Implementing DB Server: Bottom Line
DB Server Summary
MOG Server-Side. Eternal Linux-vs-Windows Debate
New Generation Chooses Cross-Platform! Well, at least it SHOULD...
Eternal Windows-vs-Linux Debate
Other Technical Differences (scheduler, IPC, file access, etc)
Is it Enough to Decide?
Free as in “Free Beer”
On ISPs and Windows-vs-Linux Cost
Time To Market: Familiarity to your Developers
It is All about Money (
DB Server Considerations
Linux-vs-Windows: Time to Decide
Things to Keep in Mind: Windows
Things to Keep in Mind: Linux
Things to Keep in Mind: Reactors
MOG Server-Side. Programming Languages
Pros (compared to C++)
Cons (compared to C++)
Personal Preferences and Reactors
On Programming Languages as Such
Which Language is the Best? Or On Horses for Courses
On Programming Languages and Reactors
Supporting Multiple languages/compilers/JITs: Is It Worth the Trouble?
Supporting Different Programming Languages within the Same Project
Inter-Language Equivalence Testing: Yet Another Reactor Replay Benefit
Chapter 9. Fault Tolerance
Failure Modes & Effects
Game Server Failures
Containment of Game World server failures
Server Fault Tolerance: King is Dead, Long Live the King!
Clusters: HA, not FT
Fault-Tolerant Servers: Damn Expensive
Virtual Lockstep: Not Available Anymore?
Checkpoint-Based Fault Tolerance: Latencies and Even More Latencies (
DIY Fault-Tolerance in Reactor-fest architectures
DIY Virtual Lockstep
DIY N+1 Reactor-based Redundancy
DIY Fault Tolerance – Connections and IPs
DIY Fault Tolerance – What to Use
DIY Fault Tolerance in case of Almost-Determinism
DB Server Failures
It’s All about Numbers – and Nothing Else
Adding Fault Tolerance Can Make Your Numbers Worse
Exception – Stock Exchanges
/Implementing Fault Tolerance for DBs
Fault Tolerance – Failure Detection
Chapter 10. Pre-Coding Checklist: Things Everybody Hates, but Everybody Needs Them Too. From So
Git and unmergeable files
Git and 3rd-party Libraries
Protecting your Code
Reducing Chances: Firewalls & Antiviruses
Mitigating Impact from Compromises
3rd-party Libraries: Licensing
Issue Tracking System
Issue Tracking: No Bypassing Allowed
Enforcement and Static Analysis Tools
The Leanpub 45-day 100% Happiness Guarantee
Within 45 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers), EPUB (for phones and tablets) and MOBI (for Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
C++ Best PracticesJason Turner
Level up your C++, get the tools working for you, eliminate common problems, and move on to more exciting things!
Functional Design and ArchitectureAlexander Granin
Software Design in Functional Programming, Design Patterns and Practices, Methodologies and Application Architectures. How to build real software in Haskell with less efforts and low risks. The first complete source of knowledge.
OpenIntro StatisticsDavid Diez, Christopher Barr, Mine Cetinkaya-Rundel, and OpenIntro
A complete foundation for Statistics, also serving as a foundation for Data Science.
Leanpub revenue supports OpenIntro (US-based nonprofit) so we can provide free desk copies to teachers interested in using OpenIntro Statistics in the classroom and expand the project to support free textbooks in other subjects.
More resources: openintro.org.
R Programming for Data ScienceRoger D. Peng
This book brings the fundamentals of R programming to you, using the same material developed as part of the industry-leading Johns Hopkins Data Science Specialization. The skills taught in this book will lay the foundation for you to begin your journey learning data science. Printed copies of this book are available through Lulu.
C++20 is the next big C++ standard after C++11. As C++11 did it, C++20 changes the way we program modern C++. This change is, in particular, due to the big four of C++20: ranges, coroutines, concepts, and modules.
I am a Software Engineer and I am in ChargeAlexis Monville and Michael Doyle
I am a Software Engineer and I am in Charge is a real-world, practical book that helps you increase your impact and satisfaction at work no matter who you work with.
In the book, we will follow Sandrine, a fictional character who learns to think in a new way enabling her to take a different course of action.
Invest In Digital Health - The Medical Futurist's GuideDr. Bertalan Mesko
Artificial Intelligence and Digital Health are booming. In this book, we explain why now it's a good time to invest in Digital Health and give recommendations on where to invest by looking at the top 24 technological trends we find the most promising.
Atomic KotlinBruce Eckel and Svetlana Isakova
For both beginning and experienced programmers! From the author of the multi-award-winning Thinking in C++ and Thinking in Java together with a member of the Kotlin language team comes a book that breaks the concepts into small, easy-to-digest "atoms," along with exercises supported by hints and solutions directly inside IntelliJ IDEA!
The Hundred-Page Machine Learning BookAndriy Burkov
Everything you really need to know in Machine Learning in a hundred pages.
Mastering STM32Carmine Noviello
With more than 600 microcontrollers, STM32 is probably the most complete ARM Cortex-M platform on the market. This book aims to be the first guide around that introduces the reader to this exciting MCU portfolio from ST Microelectronics and its official CubeHAL.
Software Architecture for Developers: Volumes 1 & 2 - Technical leadership and communication
2 Books"Software Architecture for Developers" is a practical and pragmatic guide to modern, lightweight software architecture, specifically aimed at developers. You'll learn:The essence of software architecture.Why the software architecture role should include coding, coaching and collaboration.The things that you really need to think about before...
The Future of Digital Health
6 BooksWe put together the most popular books from The Medical Futurist to provide a clear picture about the major trends shaping the future of medicine and healthcare. Digital health technologies, artificial intelligence, the future of 20 medical specialties, big pharma, data privacy and how technology giants such as Amazon or Google want to conquer...
Cisco CCNA 200-301 Complet
4 BooksCe lot comprend les quatre volumes du guide préparation à l'examen de certification Cisco CCNA 200-301.
CCDE Practical Studies (All labs)
3 BooksCCDE lab
"The C++ Standard Library" and "Concurrency with Modern C++"
2 BooksGet my books "The C++ Standard Library" and "Concurrency with Modern C++" in a bundle. The first book gives you the details you should know about the C++ standard library; the second one dives deeper into concurrency with modern C++. In sum, you get more than 600 pages full of modern C++ and about 250 source files presenting the standard library...
Modern Management Made Easy
3 BooksRead all three Modern Management Made Easy books. Learn to manage yourself, lead and serve others, and lead the organization.
Programming with Ease
3 BooksAlle drei Bände der Serie Programming with Ease in einem Paket. Darin findest du alles, was ich dir zu den wichtigsten Phasen der Softwareentwicklung im Hinblick auf Clean Code Development für langfristig hohe Produktivität sagen kann.Im Band Slicing findest du die Anforderungsanalyse im Rahmen eines iterativ-inkrementellen Vorgehensmodells aus...
2 BooksUnveil the power of Ansible and Vagrant with this bundle at a special price. You'll have everything you need to get started with Vagrant - learn the basics and how to create your virtual development environments, using Ansible as provisioner! About Vagrant Cookbook Vagrant Cookbook is a complete guide to get started with Vagrant and create your...
2 BooksDocker and Kubernetes are taking the world by storm! These books will get you up-to-speed fast! Docker Deep Dive is over 400 pages long, and covers all objectives on the Docker Certified Associate exam.The Kubernetes Book includes everything you need to get up and running with Kubernetes!
Growing Agile: The Complete Coach's Guide
7 BooksGrowing Agile: Coach's Guide Series This bundle provides a collection of training and workshop plans for a variety of agile topics. The series is aimed at agile coaches, trainers and ScrumMasters who often find themselves needing to help teams understand agile concepts. Each book in the series provides the plans, slides, handouts and activity...