Shortcuts Around The Mistakes I've Made Trying To Scale MongoDB

SHORTCUTS AROUND THE MISTAKES IVE MADE SCALING MONGODB
Theo, Chief Architect at

onsdag 21 september 11
What we do
We want to revolutionize the digital advertising industry by showing that there is more to ad analytics than click through rates.
Ads
Data
Assembling sessions
exposure ping ping
ping event ping event ping
ping
session
Crunching
session session session session session session
session
session
session
42
session
session session
session
Reports
What we do
Track ads, make pretty reports.
That doesnt sound so hard

We dont know when sessions end

We dont know when sessions end Theres a lot of data

We dont know when sessions end Theres a lot of data Its all done in (close to) real time
Numbers
Numbers
40 Gb data
Numbers
40 Gb data 50 million documents
Numbers
40 Gb data 50 million documents per day
How we use MongoDB
How we use MongoDB

Virtual memory to ofoad data while we wait for sessions to nish
How we use MongoDB

Virtual memory to ofoad data while we wait for sessions to nish Short time storage (<48 hours) for batch jobs
How we use MongoDB

Virtual memory to ofoad data while we wait for sessions to nish Short time storage (<48 hours) for batch jobs Metrics storage
Why we use MongoDB
Why we use MongoDB

Schemalessness makes things so much easier, the data we collect changes as we come up with new ideas
Why we use MongoDB

Schemalessness makes things so much easier, the data we collect changes as we come up with new ideas Sharding makes it possible to scale writes
Why we use MongoDB

Schemalessness makes things so much easier, the data we collect changes as we come up with new ideas Sharding makes it possible to scale writes Secondary indexes and rich query language are great features (for the metrics store)
Why we use MongoDB

Schemalessness makes things so much easier, the data we collect changes as we come up with new ideas Sharding makes it possible to scale writes Secondary indexes and rich query language are great features (for the metrics store) Its just nice
Btw.
Btw.
We use JRuby, its awesome
A story in 7 iterations
1st iteration
secondary indexes and updates
1st iteration
secondary indexes and updates
One document per session, update as new data comes along Outcome: 1000% write lock
#1
Everything is about working around the
GLOBAL WRITE LOCK
MongoDB 2.0.0
db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)
db.coll.update({_id: "abc"}, {$push: {x: ...}}, true)

MongoDB 1.8.1
db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)
db.coll.update({_id: "abc"}, {$push: {x: ...}}, true)

2nd iteration
using scans for two step assembling
Instead of updating, save each fragment, then scan over _id to assemble sessions
2nd iteration
using scans for two step assembling
Outcome: not as much lock, but still not great performance. We also realised we couldnt remove data fast enough
#2
GLOBAL WRITE LOCK
#3
Give a lot of thought to your
PRIMARY KEY
3rd iteration
partitioning
3rd iteration
partitioning
We came up with the idea of partitioning the data by writing to a new collection every hour
3rd iteration
partitioning
We came up with the idea of partitioning the data by writing to a new collection every hour Outcome: lots of complicated code, lots of bugs, but we didnt have to care about removing data
#4
Make sure you can
REMOVE OLD DATA
4th iteration
sharding
4th iteration
sharding
To get around the global write lock and get higher write performance we moved to a sharded cluster.
4th iteration
sharding
To get around the global write lock and get higher write performance we moved to a sharded cluster. Outcome: higher write performance, lots of problems, lots of ops time spent debugging
#5
GLOBAL WRITE LOCK
#6 SHARDING IS NOT A SILVER BULLET

and its buggy, if you can, avoid it
#7 IT WILL FAIL
design for it
5th iteration
moving things to separate clusters
5th iteration
We saw very different loads on the shards and realised we had databases with very different usage patterns, some that made autosharding not work. We moved these off the cluster.
5th iteration
We saw very different loads on the shards and realised we had databases with very different usage patterns, some that made autosharding not work. We moved these off the cluster. Outcome: a more balanced and stable cluster
#8
GLOBAL WRITE LOCK
#9 ONE DATABASE
with one usage pattern
PER CLUSTER
#10 MONITOR EVERYTHING

look at your health graphs daily
6th iteration
monster machines
6th iteration
monster machines
We got new problems removing data and needed some room to breathe and think
6th iteration
monster machines
We got new problems removing data and needed some room to breathe and think Solution: upgraded the servers to HighMemory Quadruple Extra Large (with cheese).
6th iteration
monster machines
We got new problems removing data and needed some room to breathe and think Solution: upgraded the servers to HighMemory Quadruple Extra Large (with cheese).
I
#11
Dont try to scale up
SCALE OUT
#12
When youre out of ideas
CALL THE EXPERTS
7th iteration
partitioning (again) and pre-chunking
7th iteration
We rewrote the database layer to write to a new database each day, and we created all chunks in advance. We also decreased the size of our documents by a lot.
7th iteration
We rewrote the database layer to write to a new database each day, and we created all chunks in advance. We also decreased the size of our documents by a lot. Outcome: no more problems removing data.
#13
Smaller objects means a smaller database, and a smaller database means
LESS RAM NEEDED
#14
Give a lot of thought to your
PRIMARY KEY
#15
GLOBAL WRITE LOCK
#16
GLOBAL WRITE LOCK
KTHXBAI
@iconara architecturalatrocities.com burtcorp.com
Since we got time
Tips
Safe mode
Tips
Safe mode
Run every Nth insert in safe mode
Tips
Safe mode
Run every Nth insert in safe mode This will give you warnings when bad things happen; like failovers
Tips
Avoid bulk inserts
Tips
Avoid bulk inserts
Very dangerous if theres a possibility of duplicate key errors
Tips
EC2
Tips
EC2
You have three copies of your data, do you really need EBS?
Tips
EC2
You have three copies of your data, do you really need EBS? Instance store disks are included in the price and they have predictable performance.
Tips
EC2
You have three copies of your data, do you really need EBS? Instance store disks are included in the price and they have predictable performance. m1.xlarge comes with 1.7 TB of storage.

Shortcuts Around The Mistakes I've Made Trying To Scale MongoDB

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Shortcuts Around The Mistakes I've Made Trying To Scale MongoDB

Caricato da

Copyright:

Formati disponibili

SHORTCUTS AROUND THE MISTAKES IVE MADE SCALING MONGODB

Theo, Chief Architect at

ping event ping event ping

That doesnt sound so hard

That doesnt sound so hard

That doesnt sound so hard

That doesnt sound so hard

How we use MongoDB

How we use MongoDB

How we use MongoDB

How we use MongoDB

Why we use MongoDB

Why we use MongoDB

Why we use MongoDB

Why we use MongoDB

Why we use MongoDB

GLOBAL WRITE LOCK

db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)

db.coll.update({_id: "abc"}, {$push: {x: ...}}, true)

db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)

db.coll.update({_id: "abc"}, {$push: {x: ...}}, true)

GLOBAL WRITE LOCK

REMOVE OLD DATA

GLOBAL WRITE LOCK

#6 SHARDING IS NOT A SILVER BULLET

GLOBAL WRITE LOCK

#10 MONITOR EVERYTHING

CALL THE EXPERTS

LESS RAM NEEDED

GLOBAL WRITE LOCK

GLOBAL WRITE LOCK

@iconara architecturalatrocities.com burtcorp.com

Since we got time

Potrebbero piacerti anche