Sei sulla pagina 1di 37

RED HAT CEPH STORAGE

Lab Guide – Ceph Object Gateway & Ceph Pools

Lab Description

Welcome to the third installment in the Red Hat Ceph Storage Test Drive series.

In this lab we will be focusing on the installa?on and configura?on of:

Ceph Object Storage Gateway Replicated Storage Pool Erasure Coded Storage Pool

of: •   Ceph Object Storage Gateway •   Replicated Storage Pool •   Erasure Coded

AGENDA

Register for Red Hat Storage Test Drive

Launch Lab : Red Hat Ceph Object Gateway and Ceph Pools

AWS EC2 Lab Deployment Architecture Overview

Accessing Servers

Lab Setup Commands

Setup/Configure Ceph Object Store Gateway

Setup/Configure Replicated Storage Pool

Generate/View object(s)

 
 

Snapshot pool

 
 

Restore from Snapshot

Setup/Configure Erasure Coded Storage Pool

pool     •   Restore from Snapshot •   Setup/Configure Erasure Coded Storage Pool

Register for Red Hat Storage Test Drive

Go to the URL: h+ps://engage.redhat.com/aws-test-drive-201308271223

Register for Red Hat Storage Test Drive Go to the URL: h+ps://engage.redhat.com/aws-test-drive-201308271223
Register for Red Hat Storage Test Drive Go to the URL: h+ps://engage.redhat.com/aws-test-drive-201308271223

Class Selection

Once logged in you will see that courses available to you. In this Test Drive we are focusing on the Ceph Training modules. Select Ceph Training (highlighted) to get started.

In this Test Drive we are focusing on the Ceph Training modules. Select Ceph Training (highlighted)
In this Test Drive we are focusing on the Ceph Training modules. Select Ceph Training (highlighted)

Lab Launch Page

Select the Lab you wish to work on by selecNng Start Lab bu+on.

Lab Launch Page Select the Lab you wish to work on by selecNng Start Lab bu+on.
Lab Launch Page Select the Lab you wish to work on by selecNng Start Lab bu+on.

Lab - Initializing

Lab - Initializing
Lab - Initializing

Lab - Tabs

Each tab has specific informaNon relaNng to the lab you are taking

Ceph Concepts : This tab contains detail on the components that make up a Ceph storage cluster

Lab Instruc?ons: In this tab you will find instrucNons on how to set up and run the lab

Connec?on : The SSH key pairs can be found in this tab. Only needed if you have connecNon problems

ADDL. INFO : IP Addresses of the nodes in the lab

can be found in this tab. Only needed if you have connecNon problems ADDL. INFO :
can be found in this tab. Only needed if you have connecNon problems ADDL. INFO :

LAB IP INFORMATION

Under ADDL INFO tab, you should see the IP addresses of all the nodes. Note: The ManagementNodePublicIP in your lab will likely be different than appears here.

IP addresses of all the nodes. Note: The ManagementNodePublicIP in your lab will likely be different
IP addresses of all the nodes. Note: The ManagementNodePublicIP in your lab will likely be different

Lab Concepts

Ceph Calamari: Calamari is a graphical management and monitoring system for Ceph storage cluster.

Ceph Monitor: A Ceph monitor maintains a master copy of the Ceph storage cluster map with the current state of the storage cluster.

Ceph OSD (Object Storage Device): Ceph OSDs store data on behalf of Ceph clients. AddiNonally Ceph nodes perform data replicaNon, rebalancing, recovery, monitoring and reporNng funcNons.

Ceph PG (Placement Groups): PGs are logical containers for objects.

Ceph Pools: A Pool is a logical parNNon for storing objects via the PGs assigned to a pool.

Ceph Object Storage Gateway (RADOSGW): Object storage gateway supporNng S3 and SwiZ compaNbility built on top of librados.

Replicated Storage Pool: Ceph pool that establishes the number of object copies to be stored (default 3).

Erasure Coded Storage Pool: Ceph pool where objects are divided into chunks to be stored. The chunks(n) are then wri+en to mulNple OSDs using K data chunks and M coded chunks (where n = k + m ).

For more details see the Ceph Concepts guide located at the main lab page.

and M coded chunks (where n = k + m ). For more details see the

AWS resources launched as part of the Lab: Red Hat Ceph Storage – Fundamentals 1

Administra?on / Calamari - mgmt 10.100.0.72/External IP PUBLIC NETWORK mon1 mon2 mon3 client1 10.100.2.11
Administra?on / Calamari - mgmt
10.100.0.72/External IP
PUBLIC NETWORK
mon1
mon2
mon3
client1
10.100.2.11
10.100.2.12
10.100.2.13
10.100.2.14
CLUSTER NETWORK
OSD node1
OSD node2
OSD node3
OSD node4
10.100.1.11
10.100.1.12
10.100.1.13
10.100.1.14
SSD
SSD
SSD
SSD
HDD
HDD
HDD
HDD
HDD
HDD
HDD
HDD
HDD
HDD
HDD
HDD
10.100.1.12 10.100.1.13 10.100.1.14 SSD SSD SSD SSD HDD HDD HDD HDD HDD HDD HDD HDD HDD

Logical view of Red Hat Ceph Storage cluster components

RADOSGW RBD LIBRADOS Client Interface Layer Ceph Diagram Here RADOS Obj Obj Obj Obj Obj
RADOSGW
RBD
LIBRADOS
Client Interface Layer
Ceph Diagram Here
RADOS
Obj
Obj
Obj
Obj
Obj
Obj
Obj
Obj
Obj
Obj
Objects in Pools
Obj
Obj
Obj
Obj
Obj
Obj
Pool ID. (Hash(Object)
%Num of PGs)
Pool ID. (Hash(Object)
%Num of PGs)
CRUSH Ruleset
Placement Groups
PG PG PG PG PG PG PG PG PG PG PG PG
PG PG PG PG PG PG PG PG PG PG PG PG
PG PG PG PG PG PG PG PG PG PG PG PG
PG PG PG PG PG PG PG PG PG PG PG PG
CRUSH Map
OSD1 OSD2
OSD3
OSD4
OSD5 OSD6
Ceph Nodes:
Mon1 Mon2 Mon3
OSD Hosts
Monitors ( mons)
PG PG PG PG PG PG CRUSH Map OSD1 OSD2 OSD3 OSD4 OSD5 OSD6 Ceph Nodes:

Lab Setup

This is the reference host file for this lab

127.0.0.1

localhost localhost.localdomain localhost4 localhost4.localdomain4 !

::1

localhost localhost.localdomain localhost6 localhost6.localdomain6 !

# !

10.100.0.72 mgmt.ec2.internal mgmt ip-10-100-0-72.ec2.internal ip-10-100-0-72 !

10.100.0.80 gateway.ec2.internal gateway ip-10-100-0-80.ec2.internal ip-10-100-0-80!

# !

10.100.1.11 node1.ec2.internal node1 ip-10-100-1-11.ec2.internal ip-10-100-1-11 !

10.100.1.12 node2.ec2.internal node2 ip-10-100-1-12.ec2.internal ip-10-100-1-12 !

10.100.1.13 node3.ec2.internal node3 ip-10-100-1-13.ec2.internal ip-10-100-1-13 !

10.100.1.14 node4.ec2.internal node4 ip-10-100-1-14.ec2.internal ip-10-100-1-14 !

# !

10.100.2.11 mon1.ec2.internal mon1 ip-10-100-2-11.ec2.internal ip-10-100-2-11 !

10.100.2.12 mon2.ec2.internal mon2 ip-10-100-2-12.ec2.internal ip-10-100-2-12 !

10.100.2.13 mon3.ec2.internal mon3 ip-10-100-2-13.ec2.internal ip-10-100-2-13 !

# !

10.100.2.14 client1.ec2.internal client1 ip-10-100-2-14.ec2.internal ip-10-100-2-14 !

A note about networking This lab uses a very simple networking schema to be+er facilitate the learning acNviNes. Your corporate networking setup will likely differ. Make sure the Admin node has connecNvity to the server nodes (Mons, OSD’s).

networking setup will likely differ. Make sure the Admin node has connecNvity to the server nodes

Lab Setup

First we need to InstanNate the lab environment.

Use your SSH client of choice to access the management node via the public ip address (found at the bo+om of the ADDL. INFO tab on the launch page):

ssh ceph@ManagementNodePublicIP // password = Redhat12 !

Example: ssh ceph@54.88.26.3 !

You can login to each individual node using:

User – ceph (or root) !

Password – Redhat12!

Note: If your SSH connecNon fails, do the following

- Select the CONNECTION tab from the main lab page

- Select the Download PEM/PPK bu+on

- Select Download PEM (note the download locaNon of the PEM file)

- Change to the locaNon of the downloaded PEM file and execute the following:

ssh -i name-of-file.pem ceph@ip-address-of-ManagementNodePublicIP

Example: ssh -i qwikLABS-L12-1652.pem ceph@54.88.26.3 (your keypair and IP will differ)!

Example: ssh -i qwikLABS-L12-1652.pem ceph@54.88.26.3 (your keypair and IP will differ) !

Add OSDs

Login to the mgmt node as user ceph with password Redhat12 and change directory to ceph-config. Execute the mgmt1.sh script under /home/ceph/scripts directory to add eight OSDs to the Ceph cluster. You will be adding the disks xvdg and xvdh for the four nodes.

!

[ceph@mgmt ~]$ cd /home/ceph/ceph-config! [ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt1.sh !

! [ ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt1.sh ! Virtual disk(s) on EC2 Bonus: Add “ xvdi ” to

Virtual disk(s) on EC2

Bonus: Add “ xvdi ” to all 4 nodes to increase the number of OSDs in the cluster to twelve.

! Virtual disk(s) on EC2 Bonus: Add “ xvdi ” to all 4 nodes to increase

Verify Lab

Using a browser go to the URL of the management node. h+p://ManagementNodePublicIP

Username – root Password – Redhat12

Using a browser go to the URL of the management node. h+p://ManagementNodePublicIP Username – root Password
Using a browser go to the URL of the management node. h+p://ManagementNodePublicIP Username – root Password

Verify Lab - Add Hosts to Calamari

Select the Add bu+on (highlighted) to establish the Ceph cluster and setup the Monitors

to establish the Ceph cluster and setup the Monitors Once the process completes, return to the

Once the process completes, return to the command prompt and enter:

[ceph@mgmt ceph-config]$ sudo salt '*' state.highstate !

completes, return to the command prompt and enter: [ceph@mgmt ceph-config] $ sudo salt '*' state.highstate !

Ceph Management/Monitoring – CLI

Type the following administraNon commands to check the status of the cluster, lisNng of the OSDs & pools and the status of quorum among the monitors.

[ceph@mgmt ~]$ ceph health !

Three possible values: HEALTH_OK, HEALTH_WARN, and HEALTH_ERR

[ceph@mgmt ~]$ ceph –w

A more concise view of what is occurring in the cluster (use ctrl-c to end)

[ceph@mgmt ~]$ ceph osd tree !

A tree lisNng of the OSD’s and their status

[ceph@mgmt ~]$ ceph osd lspools !

Displays the pools configured in the cluster

[ceph@mgmt ~]$ ceph quorum_status --format json-pretty !

Displays the quorum status of the Monitors

Note: Try the previous “ quorum_status” command without “ json-pretty” formanng and compare the two

[ceph@mgmt ~]$ ceph quorum_status !

quorum_status ” command without “ json-pretty ” formanng and compare the two [ceph@mgmt ~]$ ceph quorum_status

Flow Diagram for configuring a RadosGW

Servers

[mgmt] [gateway] [client 1] mgmt1.sh gateway1.-h+p.sh Or gateway1.-h+p.sh mgmt2.sh mgmt3.sh gateway2.-h+p.sh
[mgmt]
[gateway]
[client 1]
mgmt1.sh
gateway1.-h+p.sh
Or gateway1.-h+p.sh
mgmt2.sh
mgmt3.sh
gateway2.-h+p.sh
gateway3.-h+p.sh
gateway4.-h+p.sh
gateway5.-h+p.sh
Sequence of execuNon
client1.sh
boto -scripts/*

Configure Ceph Object Gateway (RADOSGW)

Install and configure the Ceph Object Gateway.

SSH to the gateway node as ceph user. You can SSH using the public IP address for the Gateway node or SSH from the mgmt node. [ceph@mgmt ~]$ ssh ceph@gateway! Enter password – Redhat12 Execute the gateway1-h]p.sh script under /home/ceph/scripts directory. This script will install Apache, copy the configuraNon file and start the service for non-secure access. The Rados Gateway packages are also installed by the script. Answer y when prompted when prompted by YUM.

Follow instrucNons on next slide if you want to enable secure access of HTTP using SSL. [ceph@gateway ~]$ /home/ceph/scripts/gateway1-http.sh !

slide if you want to enable secure access of HTTP using SSL. [ ceph@gateway ~]$ /home/ceph/scripts/gateway1-http.sh

!

slide if you want to enable secure access of HTTP using SSL. [ ceph@gateway ~]$ /home/ceph/scripts/gateway1-http.sh

Configure Ceph Object Gateway (RADOSGW) contd.

Skip this slide if you do not require secure access of HTTP using SSL.

SSH to the gateway node as ceph user. You can SSH using the public IP address for the Gateway node or SSH from the mgmt node. [ceph@mgmt ~]$ ssh ceph@gateway! Enter password – Redhat12 Execute the gateway1-h]ps.sh script under /home/ceph/scripts directory. This script will install Apache, install SSL packages, copy the configuraNon file, generate x509 cerNficate and start the service for secure access. The Rados Gateway packages are also installed by the script. Answer y when prompted when prompted by YUM. [ceph@gateway ~]$ /home/ceph/scripts/gateway1-https.sh !

the script. Answer y when prompted when prompted by YUM. [ ceph@gateway ~]$ /home/ceph/scripts/gateway1-https.sh ! !

!

Configure Ceph Object Gateway (RADOSGW) contd.

Return to the mgmt node from the gateway node and make sure you are logged in as ceph user on the mgmt node. [root@gateway ~]# exit ! Execute the mgmt2.sh script under /home/ceph/scripts directory. [ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt2.sh !

The script uses the mgmt node to create a keyring. Then it creates a client user name and key which is then added to the Ceph storage cluster. Finally, the key ring is distributed to the gateway node.

name and key which is then added to the Ceph storage cluster. Finally, the key ring
name and key which is then added to the Ceph storage cluster. Finally, the key ring

Configure Ceph Object Gateway (RADOSGW) contd.

On the mgmt node as ceph user execute the mgmt3.sh script under /home/ceph/scripts directory. [ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt3.sh !

The script adds the Ceph Object Gateway configuraNon to the Ceph configuraNon file ( ceph.conf) on the mgmt node. The configuraNon informaNon includes hostname of the gateway server, a keyring, socket path for FastCGI and a log file. Pull the updated ceph.conf to the ceph-config directory on the mgmt node and then push out the ceph.conf file to all the nodes in the cluster including the gateway node.

on the mgmt node and then push out the ceph.conf file to all the nodes in
on the mgmt node and then push out the ceph.conf file to all the nodes in

Configure Ceph Object Gateway (RADOSGW) contd.

Log in to the gateway node as user ceph. On the gateway node as ceph user execute the gateway2.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway2.sh !

The script copies a pre-built CGI wrapper script that provides the interface between the webserver and the radosgw process. The CGI wrapper script should be in a web accessible locaNon and should be executable. The helper script creates the data directory for the radosgw daemon on the gateway node, sets the correct directory permission, log file ownership and starts the radosgw service.

daemon on the gateway node, sets the correct directory permission, log file ownership and starts the
daemon on the gateway node, sets the correct directory permission, log file ownership and starts the

Configure Ceph Object Gateway (RADOSGW) contd.

On the gateway node as ceph user execute the gateway3.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway3.sh !

The script copies a pre-built Ceph Gateway configuraNon file rgw.conf that is needed for the radosgw service to the /etc /h]pd/conf.d directory. The script then restarts the h+pd service to accept the new configuraNon.

to the / etc / h]pd / conf.d directory. The script then restarts the h+pd service
to the / etc / h]pd / conf.d directory. The script then restarts the h+pd service

Configure Ceph Object Gateway (RADOSGW) contd.

On the gateway node as ceph user execute the gateway4.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway4.sh !

The script creates a Ceph Object Gateway user for the S3 interface. Note down the access_key and secret_key values from the output as they will be used by the Python example code for access validaNon.

! ! ! ! ! ! ! !
!
!
!
!
!
!
!
!
values from the output as they will be used by the Python example code for access

Configure Ceph Object Gateway (RADOSGW) contd.

Please log into the client1 node using login ceph and password Redhat12 in order to test the Rados Gateway access using the newly created S3 user. On client1 node, execute the script client1.sh script under /home/ceph/scripts directory. [ceph@client1 ~]$ /home/ceph/scripts/client1.sh !

! The script installs Boto which is a Python package that provides interfaces to Amazon
!
The script installs Boto which is a Python package that provides interfaces to Amazon Web Services. The
script also install all the packages needed for the SwiZ access.
!
!
!
!
!
!
!
!
!
!
!
!
!
!
On client1 node, under boto-scripts directory, you will find sample Python test scripts for verifying S3
access to the Rados Gateway. The test scripts will connect to the radosgw service, create a new bucket,
create an object within the bucket and list all buckets and the objects it contains.
!
service, create a new bucket, create an object within the bucket and list all buckets and

Configure Ceph Object Gateway (RADOSGW) contd.

In order to test SwiZ access to the Rados Gateway, we need to log back into the gateway node as ceph user. On gateway node, execute the script gateway5.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway5.sh !

!

The script creates a SwiZ subuser in a two step process. The first step is to create the user, and the second step is to create the secret key. Note the secret_key corresponding to testuser:swi` in the output from the second key generaNon step. The secret key will be used to get access to radosgw. !

! ! ! ! ! ! ! !
!
!
!
!
!
!
!
!
! You can now log into the client1 node to test the SwiZ access using
!
You can now log into the client1 node to test the SwiZ access using the following command:
!
[ceph@client1 ~]$ swift -A http://10.100.0.80/auth/v1.0 !
-U testuser:swift -K '<secret_key>' list !
!
[ceph@client1 ~]$ swift -A http://10.100.0.80/auth/v1.0 ! -U testuser:swift -K '<secret_key>' list ! !

Replicated Storage Pool - CLI

Create Replicated Storage Pool – CLI

CreaNng a Ceph storage pool requires a pool name, PG/PGP values, and a pool type; replicated or erasure coded (replicated is the default type). We will create a replicated pool rep-pool-cli with 64 PG/PGP and 4 replicas.

First here is a view ways to view the exisNng pools

[ceph@mgmt ~]$ ceph osd lspools ! [ceph@mgmt ~]$ rados lspools ! [ceph@mgmt ~]$ ceph osd dump | grep -i pool !

Create replicated pool

[ceph@mgmt ~]$ ceph osd pool create rep-pool-cli 64 64 !

Again run ceph osd dump | grep -i pool and locate the rep-pool-cli pool.

How many replicas did the rep-pool-cli pool default too? ! Modify exisNng replica count to 4

[ceph@mgmt ~]$ ceph osd pool set rep-pool-cli size 4 !

Verify the replicated size is now 4

[ceph@mgmt ~]$ ceph osd dump | grep -i pool !

osd pool set rep-pool-cli size 4 ! Verify the replicated size is now 4 [ceph@mgmt ~]

Replicated Storage Pool - CLI

Create Objects and take snapshot of the newly created pool

Create an object rep-obj in rep-pool-cli

[ceph@mgmt ~]$ rados -p rep-pool-cli put rep-obj /etc/hosts ! [ceph@mgmt ~]$ rados lspools !

Verify the object in the storage cluster

[ceph@mgmt ~]$ rados -p rep-pool-cli ls !

Snapshot the pool and verify

[ceph@mgmt ~]$ rados mksnap snapshot01 -p rep-pool-cli ! [ceph@mgmt ~]$ rados -p rep-pool-cli ls !

Remove the object

[ceph@mgmt ~]$ rados -p rep-pool-cli rm rep-obj ! [ceph@mgmt ~]$ rados -p rep-pool-cli ls !

Rollback the pool to the snapshot

[ceph@mgmt ~]$ rados -p rep-pool-cli listsnaps rep-obj ! [ceph@mgmt ~]$ rados rollback -p rep-pool-cli rep-obj snapshot01 !

Verify the object is once again available

[ceph@mgmt ~]$ rados -p rep-pool-cli ls !

rep-pool-cli rep-obj snapshot01 ! Verify the object is once again available [ceph@mgmt ~] $ rados -p

Replicated Storage Pool - Calamari

Now lets create a Replicated Storage Pool using the Calamari management interface

Bring up Calamari and navigate to Manage > Pools. Select the Add bu+on (highlighted)

the Calamari management interface Bring up Calamari and navigate to Manage > Pools. Select the Add
the Calamari management interface Bring up Calamari and navigate to Manage > Pools. Select the Add

Replicated Storage Pool - Calamari

Create Replicated Storage Pool – Calamari

Add pool NAME, REPLICAS, PGs, CRUSH RULESET. Press the Add bu+on to commit

Create Replicated Storage Pool – Calamari Add pool NAME, REPLICAS, PGs, CRUSH RULESET. Press the Add
Create Replicated Storage Pool – Calamari Add pool NAME, REPLICAS, PGs, CRUSH RULESET. Press the Add

Erasure Coded Storage Pool

Create Erasure Coded Storage Pool

CreaNng a Ceph storage pool requires a pool name, PG/PGP numbers, and a pool type; replicated or erasure code (replicated is the default type). We will create an Erasure Coded pool named ec-pool with senngs k=3,m=2 (5 different OSDs will be used to store data and coded objects).

First view the exisNng pools

[ceph@mgmt ~]$ ceph osd lspools ! [ceph@mgmt ~]$ rados lspools ! [ceph@mgmt ~]$ ceph osd dump | grep -i pool !

[ceph@mgmt ~] $ ceph osd lspools ! [ceph@mgmt ~] $ rados lspools ! [ceph@mgmt ~] $

Erasure Coded Storage Pool

Erasure Code is defined by a profile. The profile dictates the number of data chunks (represented as k ) and the number of coding chunks (represented by m ), along with the erasure code plugin in use and coding technique used.

Default Profile:

ceph osd erasure-code-profile get default directory=/ usr/lib64/ ceph/erasure-code

k=2

m=1

plugin= jerasure technique=reed_sol_van

View exisNng erasure code profile

[ceph@mgmt ~]$ ceph osd erasure-code-profile get default !

Create a new erasure code profile

[ceph@mgmt ~]$ ceph osd erasure-code-profile set ec-profile ruleset-failure- domain=osd k=3 m=2 !

Verify the new profile

[ceph@mgmt ~]$ ceph osd erasure-code-profile ls !

View new profile contents

[ceph@mgmt ~]$ ceph osd erasure-code-profile get ec-profile !

osd erasure-code-profile ls ! View new profile contents [ ceph@mgmt ~] $ ceph osd erasure-code-profile get

Erasure Coded Storage Pool

Create Erasure Coded Storage Pool - CLI

Create a Ceph erasure coded pool using the profile we just created

[ceph@mgmt ~]$ ceph osd pool create ec-pool 64 64 erasure ec-profile !

Verify status/details of the newly created pool

[ceph@mgmt ~]$ ceph osd dump | grep -i pool !

For extra credit: Add data to and take a snapshot of the new erasure coded pool (hint:

use the same procedures described in the Replicated Pool – CLI secNon.

a snapshot of the new erasure coded pool (hint: use the same procedures described in the

End Lab

If you have completed all the lab exercises and wish to end your session, please select the End bu+on from the launch page.

Thank You!

completed all the lab exercises and wish to end your session, please select the End bu+on