Bigger

Drupal + Mongo:
is Better?

by Forest Mars

Drupal (FTW!)
Drupal: Aspect-oriented (modular) social-publishing framework, written in php (pdo) that allows easy creation and integration of multiple data-rich social networking sites and robust web applications & services. Used on large high-performance websites. Roadmap anticipates future web (rdf, ggg) & emerging technologies (MongoDB!)

The Problem with SQL* Webapps

> Once you have related data, you need joins > Indexing on joins does not work that well ...So you: > Introduce denormalisation > Build extra tables *(RDBMS)

The Problem with Schema

> Changes broadly lock data, requiring downtime or putting excessive, short-term load on the system.
> The data integrity constraints dont quite support application integrity constraints. For example, theres no standard way in SQL to require a column to contain only valid URLs. > Coordinating schema changes with application code changes is difficult, largely because schema changes lack convenient coupling with code.

In theory, theory in practice are exactly the same.

In practice, they're completely different.

Theory Clean abstractions Strong semantics Smart-proof

Practice WiP (works in practice) Scales well Future-proof

ACID / BASE

Acid Atomic

Base Basically Available

Consistent
Isolated

Scales well
Eventually consistant

Durable

(Why Mongo?) Highly Available Easily Scalable Partition Tolerant

(Why Mongo?)
Tableless Queriless Schemaless

Blazingly fast
Faster Development times Nicer learning curves Code is trimmer Future-proof

Performance / Scaling
Whitehouse.gov / direct engagement
15K/day contact requests 2M records in db

4GB db: replication risks

MongoDB: 180M+ documents in 1 collection

Writing Performant Queries

Sort column must be the last column used in the index.
Range query must also be the last column in an index, Only use a range query or sort on one column. Conserve indexes by re-ordering columns used in straight = queries Never use Mongo's $ne or $nin operator's Never use Mongo's $exists operator
http://jira.mongodb.org/browse/WEBSITE-12

Install Mongo
public array authenticate (string $username, string $password ) public array command ( array $data ) __construct ( Mongo $conn , string $name ) public MongoCollection createCollection ( string $name [, bool $capped = FALSE [, int $size = 0 [, int $max = 0 ]]] ) public array createDBRef ( string $collection , mixed $a ) public array drop ( void ) public array dropCollection ( mixed $coll ) public array execute ( mixed $code [, array $args = array() ] ) public bool forceError ( void ) public MongoCollection __get ( string $name ) public array getDBRef ( array $ref ) public MongoGridFS getGridFS ([ string $prefix = "fs" ] ) public int getProfilingLevel ( void ) public array lastError ( void )

Real World Example

list the nodes of a user ordered by comment count uid is stored in the node table and the comment count is in node_comment_statistics > thus query cannot be indexed (Comparison of dissimilar columns may prevent use of indexes if values cannot be compared directly without conversion.)

What's already in Drupal

* mongodb: support library for the other modules (D7/D6)

* mongodb_block: Store block information in mongodb. Very close to the core block API. * * * mongodb_cache: Store cache items in mongodb. mongodb_session: Store sessions in mongodb. mongodb_watchdog: Store watchdog messages in mongodb

* mongodb_queue: DrupalQueueInterface implementation using mongodb. * mongodb_field_storage: Store the fields in mongodb.

Mongo Watchdog

mongodb_watchdog

mongo> db.watchdog.drop(); mongo> db.createCollection("watchdog", {capped:true, size:1000000, max:10000} );

mongodb_cache

$conf['page_cache_without_data base'] = TRUE;

mongodb_sessions

$conf['session_inc'] = 'sites/all/modules/mongodb/ mongodb_session/mongodb_session.inc' ;

mongodb_sessions
function mongodb_session_user_update($edit, $account) { if (!module_exists('mongodb_field_storage')) { $roles = _mongodb_session_get_roles($account); $save = (array) $account + array( '_id' => (int) $account->uid, '@bundle' => 'user', '@fields' => array(), 'roles' => $roles, ); foreach (array('uid', 'created', 'access', 'login', 'status', 'picture') as $key) { $save[$key] = (int) $save[$key]; } mongodb_collection('fields_current', 'user')>save($save); } return $roles; }

mongodb_sessions
* * * * * * * The user-level session storage handlers: - _drupal_session_open() - _drupal_session_close() - _drupal_session_read() - _drupal_session_write() - _drupal_session_destroy() - _drupal_session_garbage_collection()

assigned by session_set_save_handler() in bootstrap.inc

mongodb_block
function hook_block_view_alter(&$data, $block) { // Remove the contextual links on all blocks that provide them. if (is_array($data['content']) && isset($data['content']['#contextual_links' ])) {
unset($data['content']['#contextual_links' ]); } // Add a theme wrapper function defined by the current module to all blocks // provided by the "somemodule" module. if (is_array($data['content']) && $block->module == 'somemodule') { $data['content']['#theme_wrappers'][] = 'mymodule_special_block';

Block rebuild
Notice: Undefined variable: block_html_id in include() (line 4 of /var/www/Drupal/drupal-7.0alpha4/themes/garland/block.tpl.php). Notice: Undefined variable: block_html_id in include() (line 4 of /var/www/Drupal/drupal-7.0alpha4/themes/garland/block.tpl.php). Notice: Undefined variable: block_html_id in include() (line 4 of /var/www/Drupal/drupal-7.0alpha4/themes/garland/block.tpl.php). Notice: Undefined variable: block_html_id in include() (line 4 of /var/www/Drupal/drupal-7.0alpha4/themes/garland/block.tpl.php). Notice: Undefined variable: block_html_id in include() (line 4 of /var/www/Drupal/drupal-7.0alpha4/themes/garland/block.tpl.php).

Render main content block

function mongodb_block_theme() { 'block' => array( 'render element' => 'elements', 'template' => 'block', 'path' => drupal_get_path('module', 'block'), ), } function mongodb_block_mongodb_block_info_alter(&$blocks) { // Enable the main content block. $blocks['system_main']['region'] = 'content'; $blocks['system_main']['weight'] = 0; $blocks['system_main']['status'] = 1; } function mongodb_block_rehash($redirect = FALSE) { $collection = mongodb_collection('block'); $theme = variable_get('theme_default', 'garland');

mongodb_field_storage
don't: variable_set('field_storage_default', 'mongodb_field_storage'); instead: $conf['field_storage_default'] = 'mongodb_field_storage'; in settings.php ESP. for session/caching backends

Drupal 7

Everything In MongoDB*
*(some restrictions may apply)

Import all Nodes > MongoDB*

// Connect $mongo = new Mongo();

(* in 14 l.o.c.)

// Get the database (it is created automatically) $db = $mongo->testDatabase; // Get the collection for nodes (it is created automatically) $collection = $db->nodes; // Get a listing of all of the node IDs $r = db_query('SELECT nid FROM {node}'); // Loop through all of the nodes... while($row = db_fetch_object($r)) { print "Writing node $row->nid\n"; // Load each node and convert it to an array. $node = (array)node_load($row->nid);

// Store the node in MongoDB $collection->save($node);

Import all Nodes > MongoDB*

(* in 14 l.o.c.)

# drush script mongoimport.php

# use testDatabase; # db.nodes.find( {title: /about/i} , {title: true}).limit(4);

Import all Nodes > MongoDB*

<?php // Connect $mongo = new Mongo(); (* in 14 l.o.c.)

// Write our search filter (same as shell example above) $filter = array( 'title' => new MongoRegex('/about/i'), ); // Run the query, getting only 5 results. $res = $mongo->quiddity->nodes>find($filter)->limit(5); // Loop through and print the title of each article. foreach ($res as $row) { print $row['title'] . PHP_EOL; } ?>

What's Next?
Multiple DB servers Data Persistance
Query logging - Devel support Query builder Views integration DBTNG Full DB Abstraction MongoDB API

Query Logging

Extend Mongo collection class Pass instance back from mongodb_collection

Implement all collection methods

Full DBTNG Impementation

DO NOT USE !!!

awesomesauce
page callback => 'drupal_json' $items['node/%node/json'] = array('page callback' => 'drupal_json', 'page arguments' => array(1), 'type' => MENU_CALLBACK);

Where Mongo Won't Work

Joining across Entities ex. return birthday from profile belonging to author of current node

Thanks!
Comments & questions to: ForestMars @gmail.com ForestMars @googlewave.com Facebook, LinkedIn, etc.

twitter: @elvetica (identica@forest)