build your environments

in the cloud with pallet

@tbatchelli
a.k.a. @disclojure

Oct 28, 2010

Thursday, October 28, 2010

 what

a provisioning and
configuration tool for the
cloud
(and other places too)

Thursday, October 28, 2010

 how
• jclouds by Adrian Cole - Create/destroy
nodes/services in the cloud.
- http://www.jclouds.org

• pallet by Hugo Duncan - Manage and

configure your nodes in a sane way.
- http://hugoduncan.github.com/pallet/

Thursday, October 28, 2010

 pallet’s goals
• Use the same configuration of compute
nodes from development to production
• Reuse of configuration artifacts by
composition and reuse (as opposed to copy
+modify)
• Leverage SCM systems to store the history
of setups
• Bring software engineering practices to
Thursday, October 28, 2010
 ✓ No central server
✓ No agents on nodes
✓ Works with standard (or custom) images
✓ First-class support for complex multi-
service configurations
✓ All node configuration done remotely via
generated scripts over SSH
✓ DSL for generating scripts, resources
✓ Testable

Thursday, October 28, 2010

 crates
• Pallet’s basic configuration unit
• A crate is a grouping of functions
• Functions usually:
- manipulate/create internal representations
- result in resources deployed or scripts being run
in the target nodes

Thursday, October 28, 2010

 proxied

Custom Crates

tomcat reverse
bootstrap tomcat
deploy proxy

Standard Crates

automated haproxy
admin user

java tomcat

Thursday, October 28, 2010

 proxied node
(core/defnode  proxied
   "Basic  web  app,  served  by  tomcat"
   {:os-­‐family  :ubuntu  
     :os-­‐description-­‐matches  "10.04"
     :inbound-­‐ports  [8080  22]}  
   :bootstrap  (resource/phase  (crates/bootstrap))
   :configure  (resource/phase  (crates/tomcat))
   :deploy  (resource/phase  
(crates/tomcat-­‐deploy  "webapp.war"))
   :restart-­‐tomcat  (resource/phase
                                     (service/service  "tomcat6"  
             :action  :restart)))

Thursday, October 28, 2010

 proxied node
(core/defnode  proxied
   "Basic  web  app,  served  by  tomcat"
   {:os-­‐family  :ubuntu  
     :os-­‐description-­‐matches  "10.04"
     :inbound-­‐ports  [8080  22]}  
   :bootstrap  (resource/phase  (crates/bootstrap))
   :configure  (resource/phase  (crates/tomcat))
   :deploy  (resource/phase  
(crates/tomcat-­‐deploy  "webapp.war"))
   :restart-­‐tomcat  (resource/phase
                                     (service/service  "tomcat6"  
             :action  :restart)))

Thursday, October 28, 2010

 proxied node
(core/defnode  proxied
   "Basic  web  app,  served  by  tomcat"
   {:os-­‐family  :ubuntu  
     :os-­‐description-­‐matches  "10.04"
     :inbound-­‐ports  [8080  22]}  
   :bootstrap  (resource/phase  (crates/bootstrap))
   :configure  (resource/phase  (crates/tomcat))
   :deploy  (resource/phase  
(crates/tomcat-­‐deploy  "webapp.war"))
   :restart-­‐tomcat  (resource/phase
                                     (service/service  "tomcat6"  
             :action  :restart)))

Thursday, October 28, 2010

 (converge  {proxied  1}  
                     :compute  service  
                     :phase  :deploy)

• jclouds instantiates one node (if needed)

• Pallet:
- Sets up an admin user, and exchanges SSH keys so that
you can login via SSH
- Installs tomcat, and all the necessary dependencies (e.g.
java)
- Deploys the designated webapp on tomcat
- Starts tomcat

Thursday, October 28, 2010

 How to deal with
multi-service setups?

Thursday, October 28, 2010

 we’ll use this example
proxy proxied

haproxy tomcat

webapp

http://github.com/hugoduncan/pallet-examples

Thursday, October 28, 2010

 multi-service config

(pallet.core/converge  
{proxied  10  haproxy  1}  
:compute  service  
:phase  :deploy)

Thursday, October 28, 2010

 two-phase model
1. Invoke: for each phase (bootstrap,
configure, deploy, etc...), on all nodes of
each tag, call the crates. This generates:
- An intermediate representation of both
the particular and the global configuration
- A list of the tasks to perform to setup the
nodes up to that configuration level needed

2. Execute all listed tasks based on the final

configuration created in the previous step

Thursday, October 28, 2010

 the invoke step
• haproxy crate
- Schedules the creation its configuration based on the
info about the proxied nodes found in the intermediate
model.
• Notice that at this point the proxied info might not yet known

• Notice that the actual creation of the configuration will be

done in the next step

• proxied crate
- for every proxied node it will create an entry in the
intermediate model

Thursday, October 28, 2010

 the execute step

• builds the actual files and scripts to be

created and executed, based on the
intermediate model
• Executes the scripts on the node via ssh,
move files, etc...

Thursday, October 28, 2010

 { { debconf-set-selections <<EOF
debconf debconf/frontend select noninteractive
debconf debconf/frontend seen false
EOF
} && aptitude install -q -y haproxy && aptitude show haproxy; } || { echo
Packages failed ; exit 1 ; } >&2
echo "...done"
echo "remote-file /etc/haproxy/haproxy.cfg..."
{ { cat > /etc/haproxy/haproxy.cfg.new <<'EOFpallet'
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
user haproxy
...
retries 3
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen app1 0.0.0.0:80
server QytL9K0WhWk9-qlzmUNWzA 184.72.80.35:8080 maxconn 50 weight 1 check
server 4_5jtxMF5P0k-1q97bFbAg 174.129.114.224:8080 maxconn 50 weight 1
check
balance roundrobin

EOFpallet
........

Thursday, October 28, 2010

 reuse crates in all
cycles of your app

Thursday, October 28, 2010

 development

haproxy

proxied

DB

Thursday, October 28, 2010

 QA

haproxy haproxy

proxied proxied

DB

Thursday, October 28, 2010

 integration tests
haproxy

proxied proxied

DB
DB

Thursday, October 28, 2010

 production
haproxy
haproxy

proxied
proxied
proxied
proxied
proxied
proxied

DB
DB

Thursday, October 28, 2010

 thank you
please visit
- www.pallet.org
- www.jclouds.org
- #pallet on IRC
- #jclouds on IRC
- @hugoduncan on twitter
- @jclouds on twitter
- and of course you can find everyting clojure in
www.disclojure.org

Thursday, October 28, 2010