Deployment Architectures

Alvin Richards - alvin@10gen.com Technical Director, 10gen Inc. @jonnyeight

Everything is Done

Five things to think about

1. Data Protection 2. Machine Sizing: Disk and Memory 3. Load Testing and Monitoring 4. Backup and restore 5. Operations Handbook

Types of outage
Planned

Hardware upgrade O/S or le-system tuning Relocation of data to new le-system / storage Software upgrade Hardware failure Data center failure Region outage Human error Application corruption

Unplanned

Replication features
A cluster of N servers Any single node can be primary Consensus election of primary Automatic failover Automatic recovery All writes to primary Reads can be to primary (default) or a secondary

How MongoDB Replication works

Member 1 Member 3

Member 2

Set is made up of 2 or more nodes

How MongoDB Replication works

Member 1 Member 3

Member 2 PRIMARY

Election establishes the PRIMARY Data replication from PRIMARY to SECONDARY

How MongoDB Replication works

Member 1 negotiate new master Member 3

Member 2 DOWN

PRIMARY may fail Automatic election of new PRIMARY if majority

exists

How MongoDB Replication works

Member 1

Member 3 PRIMARY

Member 2 DOWN

New PRIMARY elected Replication Set re-established

How MongoDB Replication works

Member 1

Member 3 PRIMARY

RECOVERING

Member 2

Automatic recovery

How MongoDB Replication works

Member 1

Member 3 PRIMARY

Member 2

Replication Set re-established

Typical Congurations

Typical Congurations

Typical Congurations

Application CANNOT read data Application CANNOT write data

Typical Congurations

Replica Set

Typical Congurations

Replica Set

Application CAN read data Application CANNOT write data

Typical Congurations

Replica Set

Typical Congurations

Replica Set

Application CAN read data Application CAN write data

Typical Congurations
Replica Set London San Francisco New York

Typical Congurations
Replica Set London San Francisco New York

Application CAN read data Application CAN write data

Hardware Sizing

Collection 1

Index 1

Collection 1

Virtual Address Space 1

Index 1

Collection 1

Virtual Address Space 1

Index 1

This is your virtual memory size (mapped)

Collection 1

Virtual Address Space 1 Physical RAM

Index 1

Collection 1

Virtual Address Space 1 Physical RAM

Index 1

This is your resident memory size

Collection 1

Virtual Address Space 1 Physical RAM

Disk

Index 1

Collection 1

Virtual Address Space 1 Physical RAM

Disk

Index 1

Virtual Address Space 2

Collection 1

Virtual Address Space 1 Physical RAM

Disk

Index 1

= =

100 ns 10,000 ns

Sizing RAM and Disk

Working set Document Size Memory versus disk Data lifecycle patterns Long tail pure random bulk removes

Figuring out working Set

> db.blogs.stats() { "ns" : "test.blogs", "count" : 1338330, "size" : 46915928, "avgObjSize" : 35.05557523181876, "storageSize" : 86092032, "numExtents" : 12, "nindexes" : 2, "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, "totalIndexSize" : 99860480, "indexSizes" : { "_id_" : 55877632, "name_1" : 43982848 }, Size of data Average document size Size on disk (and in memory!)

Size of all indexes

Size of each index

Disk congurations
Single Disk
~200 seeks / second

Disk congurations
Single Disk
~200 seeks / second

RAID 0

~200 seeks / second

~200 seeks / second

~200 seeks / second

Disk congurations
Single Disk
~200 seeks / second

RAID 0

~200 seeks / second

~200 seeks / second

~200 seeks / second

RAID 10

~400 seeks / second

~400 seeks / second

~400 seeks / second

SSD?
Seek time of 0.1ms vs 5ms
(200 seeks / sec => 10000 seeks / sec)

But expensive

Key Points
Know how important page faults are If you want low latency, avoid page faults Size memory appropriately To avoid page faults, t everything in RAM Collection Data + Index Data Provision disk appropriately RAID10 is recommended SSDs are fast, if you can afford them

Monitoring is your friend!

Monitoring Tools
mongostat -

MMS! - http://mms.10gen.com munin, cacti, nagios, zabix http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics

Load Testing!
Understand what you think the system should do

Load and test your hypothesis Use the DB profiler Use a trending monitoring tool to analyze MMS, munin, etc.

Backups!
mongodump versus snapshot

Restore a member versus whole rep set

versus whole cluster system

Don't forget your config dbs in a sharded

Plan for the worst

Not everything will go to plan

Have a Operations handbook Practice basic procedures backup & restore failing over node rolling upgrade

download at mongodb.org
alvin@10gen.com

conferences, appearances, and meetups

http://www.10gen.com/events

http://bit.ly/mongoN

Facebook | Twitter | LinkedIn

@mongodb

http://linkd.in/joinmongo