Sei sulla pagina 1di 90

HP AppSystem for Microsoft SQL Server 2012

Parallel Data Warehouse


Getting Started Guide

Abstract
This document provides information about configuring and using HP AppSystem for Microsoft SQL Server 2012 Parallel Data Warehouse.

Part Number: 745824-001


September 2013
Edition: 1
© Copyright 2013 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall
not be liable for technical or editorial errors or omissions contained herein.

Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government
under vendor’s standard commercial license.

Microsoft®, Windows®, and Windows Server® are U.S. registered trademarks of Microsoft Corporation.
Contents

Introduction .................................................................................................................................. 5
Executive summary .................................................................................................................................... 5
Solution overview ...................................................................................................................................... 5

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console........................................ 12
Adding new logins (user IDs) .................................................................................................................... 16
Admin Console features ........................................................................................................................... 17
Overview ..................................................................................................................................... 17
Home screen................................................................................................................................. 17
Sessions screen ............................................................................................................................. 18
Queries screen .............................................................................................................................. 19
Loads screen ................................................................................................................................. 20
Backups/Restores screen ................................................................................................................ 21
Health screen ................................................................................................................................ 22
Resources screen ........................................................................................................................... 23
Storage screen .............................................................................................................................. 23
Performance monitor screen ............................................................................................................ 25

SQL Server 2012 Parallel Data Warehouse Client Tools ................................................................. 27


Client Tools and connection strings ............................................................................................................ 27
Supported query tools .............................................................................................................................. 28
Client tool connectivity - port 17001 .......................................................................................................... 28
PolyBase Hadoop integration .................................................................................................................... 29

Data management ...................................................................................................................... 31


Data loading prerequisites........................................................................................................................ 31
Best practices for performance .................................................................................................................. 31
Distributed and replicated tables ............................................................................................................... 32
xVelocity in-memory analytics engine ......................................................................................................... 32
Data loading tools ................................................................................................................................... 32
Copying the Client Tools installer ..................................................................................................... 33
Installing Client Tools ..................................................................................................................... 34
Creating a staging database .......................................................................................................... 36
Loading data using dwloader.......................................................................................................... 37
Loading data using fastappend ....................................................................................................... 38

Password management ................................................................................................................ 41


Appliance Password Reset screen .............................................................................................................. 41
Changing individual device passwords ...................................................................................................... 41
Finding device IP addresses ............................................................................................................ 42
iLO passwords .............................................................................................................................. 42
Ethernet switch passwords .............................................................................................................. 44
iPDU passwords ............................................................................................................................ 47
FDR InfiniBand switch passwords ..................................................................................................... 52

Server management .................................................................................................................... 56


Starting Hyper-V Manager ........................................................................................................................ 56
Adding nodes ......................................................................................................................................... 57

Contents 3
Using Failover Cluster Manager ................................................................................................................ 59
Aligning servers and VMs......................................................................................................................... 62
Recognizing misaligned VMs .......................................................................................................... 62
Realigning VMs ............................................................................................................................. 63

Appliance management............................................................................................................... 65
Power up the appliance ........................................................................................................................... 65
Backing up and restoring SQL Server 2012 Parallel Data Warehouse data .................................................... 66
Backup and restore syntax .............................................................................................................. 66
Moving backup data to a corporate-wide backup system .................................................................... 68
Power down the appliance ....................................................................................................................... 68
Starting the VMs...................................................................................................................................... 69
Starting AD VM ............................................................................................................................. 69
Starting the remaining VMs on HST01.............................................................................................. 70
Starting the VMs on the HSA nodes ................................................................................................. 70
Starting the appliance software ................................................................................................................. 72
Configuring the appliance ........................................................................................................................ 72
Running Configuration Manager ..................................................................................................... 73
Configuring Windows Server Update Services .................................................................................. 79

Appendix................................................................................................................................... 80
HP Data Integration Platform ..................................................................................................................... 80
InfiniBand Data Integration Platform connectivity ................................................................................ 80
10 Gb Ethernet connectivity ............................................................................................................ 80
HP Insight Remote Support software and HP AppSystem for Parallel Data Warehouse ...................................... 81

Support and other resources ........................................................................................................ 82


HP AppSystem Parallel Data Warehouse Support Pack ................................................................................ 82
Download and installation .............................................................................................................. 82
HP AppSystem Parallel Data Warehouse support recommendations ............................................................... 83
Contacting support for HP AppSystem Parallel Data Warehouse ................................................................... 83
HP contact information ............................................................................................................................. 84

Acronyms and abbreviations ........................................................................................................ 85


Documentation feedback ............................................................................................................. 88
Index ......................................................................................................................................... 89

Contents 4
Introduction

Executive summary
The HP AppSystem for Microsoft SQL Server 2012 Parallel Data Warehouse is a purpose-built solution
developed through collaborative engineering efforts between HP and Microsoft. Designed to simplify the
implementation and management of an enterprise data warehouse environment and optimized for Microsoft
SQL Server 2008 R2 (and later, Microsoft SQL 2012 Parallel Data Warehouse Edition), HP AppSystem for
Parallel Data Warehouse provides excellent performance through MPP.
HP AppSystem for Parallel Data Warehouse is a massively parallel, scale-out, shared-nothing appliance
architecture that operates as a single entity. The approach to servicing the unit, including hardware
replacement or driver and firmware updates, contrasts with scale-up, SMP server architecture maintenance
methodologies.

IMPORTANT: HP supports only the HP approved firmware and driver updates contained in the
HP AppSystem Parallel Data Warehouse Support Pack (see, "HP AppSystem Parallel Data
Warehouse Support Pack (on page 82)."
HP AppSystem for SQL 2012 Parallel data Warehouse appliance uses a very tightly controlled
composition of firmware and drivers. HP has completed extensive testing on this appliance with
the specific firmware and drivers delivered from the factory, as well as those delivered in the
support pack.

For additional information about HP AppSystem for Parallel Data Warehouse, see HP AppSystem for
Microsoft SQL Server 2012 for Parallel Data Warehouse QuickSpec, available from the HP website
(http://h18000.www1.hp.com/products/quickspecs/ProductBulletin.html).

Solution overview
HP AppSystem for Parallel Data Warehouse replaces the previous generation concepts of Control Rack and
Data Rack with a virtualized cluster architecture. This section provides information on the available servers,
as well as diagrams to help you locate the servers in your configuration.
Base Scale Unit
The Base Scale Unit is the smallest HP AppSystem for Parallel Data Warehouse configuration available and
includes the following:
• Two FDR InfiniBand switches
• Two HP 5120-24G EI switches
• Two passive servers, HST01and HST02 (An optional, third passive server, HST03, is available in a
single rack only.)
o HST01 runs four virtual machines:
— CTL01—Runs MPP engine, controls DMS on all nodes, is the location where client applications
connect, and is a central point for all hardware monitoring.

Introduction 5
— MAD—Provides a staging location for software upgrades and patch deployment, holds
software images in case a node needs re-imaging, and holds Active Directory.
— AD—Special-purpose database designed by Microsoft to handle a large number of read and
search operations and a significantly smaller number of changes and updates.
— VMM—Used during HP AppSystem Parallel Data Warehouse installations and upgrades and to
provision empty servers. Also referred to as VMM.

Introduction 6
o HST02 and the optional HST03 are standby servers for all other servers in the cluster.

The base rack contains a server at the top of the rack (HST01) that runs the corralling service and four VMs.
The next slot holds a second server (HST02), which provides failover services. If present, the optional, third

Introduction 7
server (HST03) occupies the third slot. HST02 and HST03 act as backup for any node in the appliance, even
in a multi-rack cluster.
The base rack also includes two HSA servers, located at the bottom of the rack (HSA01 and HSA02). These
servers provide data storage.
Users can add scale units within each rack. Each scale unit contains two HSA servers and a respective
D6000 storage array. For high availability (failover) purposes, multi-rack appliances must have three HST
servers in the Base rack and each additional rack must have only one passive HST server.
For more information on Ethernet and InfiniBand connections and ports, see "HP Data Integration Platform
(on page 80)."

Introduction 8
The following table provides an example of single rack scaling:

Introduction 9
The following table provides an example of second rack scaling (showing optional passive servers in
expansion racks):

Introduction 10
The following graphic shows an example of a multi-rack HP AppSystem for Parallel Data Warehouse system.
Environments vary and can contain different quantities of scale units and passive servers. HP AppSystem for
Parallel Data Warehouse scales up to seven racks containing a maximum of 64 active and passive server
nodes.

HP racks and configures HP AppSystem for Parallel Data Warehouse in the factory, and installs Microsoft
Windows 2012, Microsoft SQL Server 2012, and the Microsoft SQL Server 2012 Parallel Data Warehouse
software.
All D6000s and disk sizes within the D6000s must be the same. If they are different, you can only use the
amount of space available on the smallest D6000 disk on the appliance.
HP AppSystem for Parallel Data Warehouse architecture enables simultaneous query execution, and each
individual query can execute on all the servers and all the D6000 disks storing user data, in parallel. This
massively parallel architecture gives HP AppSystem for Parallel Data Warehouse the speed to provide user
queries with significantly higher levels of I/O throughput than is available on a scale-up SMP system.
For more information about the appliance architecture, see “Understanding SQL Server PDW (SQL Server
PDW)” in the Microsoft SQLServerPDW.chm file located on the Control Virtual Machine (CTL01 VM) in the
C:\PDWINST\ClientTools directory.

For information on the Ethernet and InfiniBand connections and available ports, see "HP Data Integration
Platform (on page 80)."

Introduction 11
Accessing the SQL Server 2012 Parallel Data
Warehouse Admin Console

Any user with security access can open the Admin Console.
1. Determine your Node Cluster IP address:
a. Run dwconfig to start the Configuration Manager.
b. Click Network Configuration in the left navigation pane.
c. Locate and select <domain_name> CTL01in the list under Nodes.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 12
d. Make note of the Node Cluster IP address listed for <domain_name> CTL01.

2. Enter the Node Cluster IP address into a browser using an HTTPS secure connection from any node that
has access to the appliance, such as the Management nodes, Control nodes, Landing Zone, or Backup
Server. For example:
https://172.56.252.5/
HP Services configured the Node Cluster IP address for your appliance at the time of installation.
The browser might return an error regarding a security certificate for this device.

To avoid this error, replace the default certificate with a trusted certificate. For more information, see the
"Appliance Configuration Tasks (SQL Server PDW)" in the SQLServerPDW.chm.
3. Click Continue to the website (not recommended).

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 13
Your system might return the following security alert. Click OK.

o Your system might return the following pop-up, indicating that you need to add the IP address as a
trusted site.

i. Check the box next to Continue to prompt when website content is blocked, and then click Add.
ii. Verify the IP address, and then click Add.

iii. Click Close.


After you are connected, the Admin Console prompts you for a user name and password.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 14
4. Enter sa in the Login field, and then enter your password in the Password field.

The Admin Console Landing Page appears.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 15
Adding new logins (user IDs)
Add new SQL Server 2012 Parallel Data Warehouse logins using the CREATE LOGIN SQL statement. For
more information, see "SQL Reference, Data Definition Statements, CREATE LOGIN" in the
SQLServerPDW.chm.

When creating the login, you can set the password as part of the command. The user must change the
password on the first login. A new login cannot connect or be used to load data until the user has logged in
and reset the password.
If the login already exists, you can reset the password using the ALTER LOGIN statement.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 16
Admin Console features
Overview
Use the Admin Console to monitor and review appliance activities.
To access and log on to the Admin Console, see "Accessing the SQL Server 2012 Parallel Data Warehouse
Admin Console (on page 12)."
After you log on to the Control Node Cluster, the SQL Server 2012 Parallel Data Warehouse Admin Console
Home screen (on page 17) appears.

Home screen
The Home screen provides a starting point for viewing and changing appliance configuration information
including:
• Sessions—Access individual session information to review activities performed during that session.
• Queries—Review and analyze queries.
• Loads—Check Loads status and detail.
• Backups/restores—Review current and previous backup/restore operations.
• Health—Check overall appliance operation.
• Resources—Check resource status including locks or waits.
• Storage—Review allocation of storage across the appliance.
• Performance monitor—Monitor performance in real-time.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 17
Sessions screen
Use the Sessions screen to view session information:
1. Click SESSIONS in the left navigation pane.
2. Click an SID number in the SESSION ID column to see details about that session.

3. To view Queries, Locks, or Waits for a particular session, click the individual word, and then click a
number in the ID column (in the example, session ID #: sid2776).
You can view details about work performed during that session, as well as the IP address of the client
that performed the work (in the example, 172.16.255.37).

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 18
For example, if you selected Queries, and then the ID QID2287, the system displays information about
the query text and query plan for that session.

You can also view Locks and Waits information from the Resources ("Resources screen" on page 23) screen.

Queries screen
The Queries screen provides an additional method for accessing information about specific queries.
1. Select QUERIES in the left navigation pane.
2. Click a QID number in the ID column.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 19
For example, if you selected the ID QID2287, the system displays information about the query text and
query plan for that session.

Loads screen
The Loads screen provides an additional method for accessing Loads information about specific queries.
1. Select LOADS in the left navigation pane.
2. Click a number in the ID column.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 20
For example, if you selected the ID 626, the system displays the actual query used for that data load,
as well as the details associated with the load.

Backups/Restores screen
The Backup/Restores screen provides a quick view of backup and restore operations.
1. Select BACKUPS/RESTORES in the left navigation pane.
2. Choose the ID associated with the type of activity you want to review (backup or restore).
3. Click the number in the ID column for that activity type.

The system returns the backup or restore details associated with the ID you selected.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 21
As shown in the following example, multiple backup processes for all compute nodes can occur in parallel.
When you issue a backup or restore command, the SQL Server 2012 Parallel Data Warehouse software
executes multiple operations in parallel.

Health screen
The Health status screen provides access to the status of the various nodes.
1. Select HEALTH in the left navigation pane.
2. Click STATUS in the command line. This presents the overall status of the individual nodes.

3. Click Alerts, All Alerts, or Errors to see any alerts or errors for all nodes.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 22
4. Click an individual node name to see additional details about the health status of that node.

Resources screen
The Resources screen provides information about Locks and Waits, which can effect the HP AppSystem for
Parallel Data Warehouse appliance resources and workload performance.
1. Select RESOURCES in the left navigation pane.
2. Click either LOCKS or WAITS to see the reports available for either.

Storage screen
The Storage screen provides a report showing the allocation of disk storage across various databases,
including tempdb.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 23
1. Select STORAGE in the left navigation pane. The system provides information about the overall space
utilization of the appliance, including the OS and databases, and lists the databases by size
(DATA(GB)).

2. Click a database name to view detailed information about that database.


The following example shows a database with data evenly distributed across all eight compute nodes
(CMP01 - CMP08).

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 24
Performance monitor screen
Use the Performance Monitor screen to view the current performance of the appliance in real time.
1. Select PERFORMANCE MONITOR in the left navigation pane.
o Performance monitor requires the use of Microsoft Silverlight. If your server does not have Silverlight
installed, the system prompts you to install it.
o If the HP AppSystem Parallel Data Warehouse has Internet access, you can download Silverlight
directly from Microsoft. Otherwise, you can put Silverlight on a USB memory stick and install it on
the HP AppSystem Parallel Data Warehouse host.

2. After you have installed Silverlight, click PERFORMANCE MONITOR to see the Parallel Data
Warehouse performance in real time.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 25
Installing Microsoft Silverlight
To view the current performance of the appliance in real time, install Microsoft Silverlight on the Admin
server.
If the Parallel Data Warehouse has Internet access, you can download Silverlight directly from Microsoft from
within the Performance Monitor screen. Otherwise, you can put Silverlight on a USB stick and install it on the
HP AppSystem Parallel Data Warehouse host.
You can get the Silverlight download from the Microsoft webpage (http://www.microsoft.com/silverlight).
1. Click Silverlight_x64.exe to launch the installation program.

2. Click Install now.


3. Wait for the installation to finish, and then close and reopen your browser.
After you have installed Silverlight, log onto the Admin console ("Overview" on page 17) and return to the
Performance monitor ("Performance monitor screen" on page 25) screen.

Accessing the SQL Server 2012 Parallel Data Warehouse Admin Console 26
SQL Server 2012 Parallel Data Warehouse Client
Tools

Client Tools and connection strings


HP AppSystem for Parallel Data Warehouse supports Microsoft SQL Server 2012 Parallel Data Warehouse
Client Tools for the following Data Access APIs:
• ADO.NET
• OLE DB
• ODBC
• JDBC
For connection string information, see SQLServerPDW.chm in the C:\PDWINST\ClientTools.

SQL Server 2012 Parallel Data Warehouse Client Tools 27


Supported query tools
Previous versions of HP AppSystem for Parallel Data Warehouse used Nexus client software to query and
manage the database; however, Microsoft no longer includes Nexus with SQL Server 2012 Parallel Data
Warehouse. Instead, Microsoft provides and recommends SQL Server Data Tools as the query tool. For more
information about SQL Server Data Tools, see "Install SQL Server Data Tools for Visual Studio" in
SQLServerPDW.chm.

Client tool connectivity - port 17001


In this release, ODBC and OLE DB connections use SQL Server Native Client. ADO connections use .NET
Framework Data Provider for SQL Server (SqlClient). All of these connections use port 17001, because SQL
Server 2012 Parallel Data Warehouse only listens to port 17001.To avoid connection failures, append all
connection strings with a comma and the port number 17001. For example, change 10.192.54.48 to
10.192.54.48,17001.
You do not need to rebuild existing SSIS loading packages in order to update the port to 17001. For more
information, see "Breaking Changes" in the SQLServerPDW.chm.

SQL Server 2012 Parallel Data Warehouse Client Tools 28


PolyBase Hadoop integration
For SQL Server 2012 Parallel Data Warehouse, Microsoft has included PolyBase as the integrated query
across Hadoop and relational data. PolyBase replaces Sqoop, and all scripts based on the Microsoft SQL
Server SQOOP Connector for Hadoop no longer work with SQL Server 2012 Parallel Data Warehouse.
Use PolyBase to query Hadoop data using SQL, import data from Hadoop for persistent storage in SQL
Server 2012 Parallel Data Warehouse as a distributed or replicated table, and export data from SQL Server
2012 Parallel Data Warehouse into Hadoop.
To use PolyBase, replace Sqoop import statements with CREATE TABLE AS SELECT, and replace Sqoop
export statements with CREATE EXTERNAL TABLE AS SELECT.
For more information, see "PolyBase for Hadoop Integration" in SQLServerPDW.chm.

SQL Server 2012 Parallel Data Warehouse Client Tools 29


SQLServerPDW.chm also contains detailed information on configuring Hadoop for SQL Server 2012
Parallel Data Warehouse connectivity.

SQL Server 2012 Parallel Data Warehouse Client Tools 30


Data management

Data loading prerequisites


With the hardware and software up and running, you can load data into the system. Before loading data,
see "Distributed and Replicated Tables" in SQLServerPDW.chm. This documentation provides information
about SQL Server 2012 Parallel Data Warehouse terminology regarding distributed tables and replicated
tables, and provides best practices for designing a data warehouse for the appliance.

In addition, HP recommends that you have a good logical and physical database design. The following
sections provide insight into design considerations. Contact HP or Microsoft Services for additional
information regarding optimal SQL Server 2012 Parallel Data Warehouse database design techniques.

Best practices for performance


For the best performance from SQL Server 2012 Parallel Data Warehouse, try eliminating indexes. Using
indexes encourages disk head movement, and excessive disk head movement slows data streaming because
of the excessive seek time required.
Load large fact table data so that the data is stored sequentially on a disk, followed by dimension tables.
Designing your system with minimal indexing (Index-lite) or loading data using SQL Server 2012 Parallel
Data Warehouse best practices is beyond the scope of this document; however, HP or Microsoft services can
assist with data load optimization.

Data management 31
Distributed and replicated tables
A critical issue with SQL Server 2012 Parallel Data Warehouse database design involves defining replicated
tables versus distributed tables. Tables default to replicated tables unless you define a distribution key. In
general, large tables (such as fact tables) should be distributed, and small tables (such as dimension tables)
should be replicated.
HP AppSystem for Parallel Data Warehouse running SQL Server 2012 Parallel Data Warehouse enables
parallel execution of queries. The combination of hardware and software provides high levels of I/O
throughput while maintaining consistent and predictable response times, which is useful when executing
ad-hoc queries.

xVelocity in-memory analytics engine


xVelocity is part of the Microsoft family of in-memory and memory-optimized data management technologies
in SQL Server 2012. The xVelocity in-memory analytics engine and the xVelocity memory-optimized
columnstore index feature are the first two members of this family.
The xVelocity columnstore index feature enables high performance query processing in SQL Server data
warehouses and data marts. For example, business intelligence end-users tend to submit queries that select
a relatively small number of columns (hot columns) in the fact tables or dimension tables. This implies that,
rather than having a database retrieve all the columns in a row, it makes sense to structure the database in
a columnar fashion. Columnstore table structures only need to access, and potentially cache, the small
percentage of columns that a majority of users request.
Efficient compression is a side effect of a columnstore design in data warehousing. Database compression
rates for data warehouses and data marts tend to be significantly greater when data is stored by column
versus storing data by row. This is especially noticeable when a column has low cardinality. For example, if
each row includes a column named "Gender" and a column named "State," rather than repeating this low
cardinality data for each row, you only need the number or entries in each row that match the cardinality. So,
if the State column references the 50 United States, that column contains only 50 entries, and the Gender
column has, potentially, only two entries.
By reorienting data storage in a column rather than a row, organizations can see HP AppSystem for Parallel
Data Warehouse performance improve by reducing the query time from hours to seconds and compression
ratios for data. In addition, Microsoft Parallel Data Warehouse xVelocity enables updates to the HP
AppSystem for Parallel Data Warehouse xVelocity columnstore table

Data loading tools


HP Enterprise Data Warehouse Appliance Optimized for Microsoft SQL Server 2008 Parallel Data
Warehouse included a Landing Zone server that provided a pre-staging location for loading data into the
database. In addition, the Landing Zone could run ETL software such as SSIS.
HP AppSystem for Parallel Data Warehouse does not require the purchase of a Landing Zone server. Instead,
HP offers an HP Data Integration Platform. The Data Integration Platform is an optional component that HP
has sized and validated for use with ETL and dwloader, HP Insight Remote Support, landing zone
functionality, and backup/restore operations. Because most site requirements do not promote backing up HP
Parallel Data Warehouse in tandem with a data upload, combining ETL and landing zone functionality on
the same physical hardware as the backup makes sense.

Data management 32
The HP AppSystem Parallel Data Warehouse still requires landing zone functionality, and the optional Data
Integration Platform supports this function, as well as supporting backup, or you can use your own servers to
run ETL, dwloader, and landing zone functions.
HP Parallel Data Warehouse allows for multiple servers running landing zone functionality, including
dwloader, to speed up the data loading process. Similarly, HP Parallel Data Warehouse allows for multiple
backup nodes.
Be sure to consider network connectivity when connecting a server for landing zone functions. For maximum
throughput, HP recommends InfiniBand, followed by 10 Gb Ethernet. The slowest connection is 1 Gb
Ethernet connectivity between the landing zone/backup node or Data Integration Platform and HP Parallel
Data Warehouse. See "HP Data Integration Platform (on page 80)"for information about available ports on
the Ethernet and InfiniBand switches.
There are two ways to load data into the SQL Server 2012 Parallel Data Warehouse database:
• Load data directly from SSIS.
If SSIS ETL software executes on an external, non-HP Data Integration Platform tier, the Data Integration
Platform disks might contain stored, cleansed data before the actual load into SQL Server 2012 Parallel
Data Warehouse tables. However, if you run SSIS on the Data Integration Platform, your start-to-end ETL
batch window completes faster. This is because SSIS can load data directly from the Data Integration
Platform into the staging database and into SQL Server 2012 Parallel Data Warehouse tables. This
procedure of loading data directly from SSIS into the appliance saves disk I/O.
• Load data using dwloader (command line tool) from a cleansed and pre-formatted flat file.
If you are loading data from a cleansed flat file residing on the Data Integration Platform or any external
server, dwloader performs faster than SSIS.

Copying the Client Tools installer


HP installs and configures the Data Integration Platform at the factory. If you are using your own server, use
this procedure to load and configure the Client Tools.
Before loading any data, copy the ClientTools-amd64.msi file to the servers that will load the data.
You can find this file on the Parallel Data Warehouse Control VM (CTL01) in the C:\PDWINST\ClientTools
directory.
Use either Failover Cluster Manager to access CTL01, or you can connect directly to CTL01 using Windows
Remote Desktop. Windows Remote Desktop supports copy and paste functionality, so you can copy the
appropriate Windows Installer files to the server that will run dwloader.
Otherwise, use Failover Cluster Manager to access CTL01:
1. Log onto Failover Cluster Manager. If this is your first time using Failover Cluster Manager, see Using
Failover Cluster Manager (on page 59).
2. Expand Roles, and then Nodes in the left navigation pane.

Data management 33
3. Right-click CTL01, and then select Connect.

4. Navigate to the C:\PDWINST\ClientTools directory, and then locate the ClientTools-amd64.msi


file.

5. Copy the file to the server that will run dwloader.

Installing Client Tools


The following procedure uses the ClientTools-amd64.msi file to install Client Tools onto a stand-alone
server.

Data management 34
Launch the Client Tools installation.
1. To launch the installer, double-click the appropriate Windows Installer file on your data source server.
2. Click Next.

3. Read and accept the terms in license Agreement to continue with the installation.
4. Click Next.

5. Choose your installation type, and then click Next.

Data management 35
6. Click Install. The installation takes a few minutes, and then displays the following screen.

7. Click Finish, and then continue with the next section to load your data.

Creating a staging database


Microsoft recommends creating and using only one staging database for optimal SQL Server 2012 Parallel
Data Warehouse performance.

IMPORTANT: Do not create tables in the staging database. When you run the dwloader tool, it
creates tables for you.

SQL Server 2012 Parallel Data Warehouse includes the dwsys database for internal use by the software to
store temporary tables on local disks of the Control node. As a best practice, avoid using dwsys. Instead,
create your own production databases for your environment.
The following example of a command file shows how you can create a database and the associated tables
as the destination for the loaded data.
// Getting Started Guide - Sample DB and table creation
// Create target database for load test
CREATE DATABASE db100gb
WITH (
AUTOGROW = ON,
REPLICATED_SIZE = 50,
DISTRIBUTED_SIZE = 100,
LOG_SIZE = 5
)
;
// Create staging database (Refer to the “Create the Staging Database (SQL
Server PDW)” in the SQLServerPDW.chm.
SQL Server 2012 Parallel Data Warehouse uses a staging database to store data temporarily during the
load process. By default, Parallel Data Warehouse uses the destination database as the staging database,
which can cause table fragmentation.

Data management 36
To reduce table fragmentation, create a user-defined staging database. If rollback from a load failure is not
a concern, you can use the fastappend loading mode to improve performance. The fastappend loading
mode skips the temporary staging table and loads directly into the destination table.

CREATE DATABASE staging


WITH (
AUTOGROW = ON,
REPLICATED_SIZE = 40,
DISTRIBUTED_SIZE = 90,
LOG_SIZE = 4
);

USE db100gb;

// Create partsupp table


// partsupp table will be distributed using ps_partkey column

CREATE TABLE partsupp


( ps_partkey bigint,
ps_suppkey bigint,
ps_availqty integer,
ps_supplycost decimal(15,2),
ps_comment varchar(199))
WITH (Distribution = hash(ps_partkey));

Loading data using dwloader


After creating the databases and tables, you can load your data.
HP AppSystem Parallel Data Warehouse supports either ETL or ELT data loading techniques. In either case,
load your data into the HP AppSystem for Parallel Data Warehouse from a server that contains the input data
file. If you have not installed the Client Tools, see "Installing Client Tools (on page 34)" before downloading
any data.

IMPORTANT: HP AppSystem Parallel Data Warehouse allows dwloader to execute from multiple
servers when those servers act as data loading sources. Data loading timelines improve when ETL
software executes on multiple servers in parallel.

Initiate your data load using dwloader. You can find dwloader at the following location:
%SystemDrive%\Program Files\Microsoft SQL Server Parallel Data
Warehouse\100\dwloader.exe
The following SQL command is an example of a load command using the common parameters listed below.

• -S—IP address of the HP AppSystem Parallel Data Warehouse Control VM.


• -U—User ID (in our example, sa).
• -P—System password.

Data management 37
• -d—Name of the staging database.

IMPORTANT: Microsoft recommends using a staging database for optimal performance. A


staging database helps Parallel Data Warehouse maintain sequential storage, which reduces
disk seek times.
The -d stage parameter specifies the use of the staging database. If you omit this parameter,
data loads directly into the destination tables without using the staging database, which can
reduce overall database performance.

• -m—Commit loads in parallel. Required option for -M fastappend. Performs much faster than the default
loading mode by bypassing the staging database. Has no rollback function, so any recovery from a
failed or aborted load must be handled by your own load process.
• -t "|"—Delimit each field (column) in the row.
• -r 0x7c0x0d0x0a—Delimit row.
• -T—Target database.
• -i—Source data location (input).
• -R—File where you can find the rows which failed to load.
• -rv—Number or percentage of row rejections to allow before halting the load (in this example, 100).
• -b—Batch size; the number of rows to load per transaction. Defaults to 10,000.
For more information about this command and its parameters, see "Tools Reference\dwloader command-line
loader" in SQLServerPDW.chm.
After loading your data, you can use Client Tools that support OLE DB, ODBC, or ADO.NET to query your
data. The most common tools used to query the database are Reporting Services, SQL Server Analysis
Services (OLAP cubes), DWSQL (the Parallel Data Warehouse utility), and SSIS.
SSIS is useful when you need to perform a table look-up during ETL processing. SQL Server Analysis Services
translate MDX queries into SQL queries, which send the request to the Parallel Data Warehouse Control node
for query processing.

Loading data using fastappend


When you need to complete a data load in a short time frame, and you don't need to rely on rollback, you
can use fastappend. The fastappend command skips the temporary staging table and loads data directly
into the destination table. In fastappend mode, the loader appends rows directly to the end of existing rows
in the destination table without using a temporary table.
You can neither specify a staging database when using fastappend nor rely on rollback, so you must
handle any recovery from a failed or aborted load as part of your own load process. However, fastappend
speeds up loading, which is useful when you are under time constraints, such as having to complete a data
load in two hours.
Using fastappend requires using the multi-transaction (–m) option.
Load data using fastappend:
1. Connect to a database using sqlcmd. The following example connects to a database called tpch1tb.
sqlcmd -S "172.16.255.5,17001" -U sa -P password –I –d tpch1tb
2. Create a table named partsupp.
CREATE TABLE partsupp
( ps_partkey bigint,

Data management 38
ps_suppkey bigint,
ps_availqty integer,
ps_supplycost decimal(15,2),
ps_comment varchar(199))
WITH (
Distribution = hash(ps_partkey));
The hashed distribution key, ps_partkey, distributes the data across all of the Compute VMs on the
physical HSAxx servers.

3. Execute dwloader on the servers that contain the source data, such as the optional Data Integration
Platform.
dwloader -S 172.16.254.5 -U sa -P AppSys4PDW -M fastappend -m -t "|" -r
0x7c0x0d0x0a -T tpch1tb..partsupp -i d:\flatfiles.1TB\1\partsupp.tbl -R
partsupp.out -rv 100 -b 100000
Parameters include:
o -S—IP address of the HP AppSystem Parallel Data Warehouse Control VM.
o -U—User ID (in our example, sa).
o -P—System password.
o -M fastappend—Append rows directly to the end of existing rows in the destination table.
Requires the multi-transaction (–m) option.
o -m—Commit loads in parallel. Required option for -M fastappend. Performs much faster than the
default loading mode by bypassing the staging database. Has no rollback function, so any recovery
from a failed or aborted load must be handled by your own load process.
o -t "|"—Delimit each field (column) in the row.
o -r 0x7c0x0d0x0a—Delimit row.
o -T—Target database.
o -i—Source data location (input).
o -R—File where you can find the rows which failed to load.
o -rv—Number or percentage of row rejections to allow before halting the load (in this example,
100).
o -b—Batch size; the number of rows to load per transaction. Defaults to 10,000.

Data management 39
The following example shows the beginning and end of fastappend command output.

Data management 40
Password management

Appliance Password Reset screen


As part of the HP AppSystem for Parallel Data Warehouse installation and startup service, HP Support
changes the factory default passwords to site-specific passwords. You can change passwords at any time
using the Password Reset screen.
Passwords must meet the following requirements:
• 8 to 15 characters
• Lower case or upper case characters
• Numbers 0-9
• No special characters
Use the menu to change the associated passwords on the appliance. HP recommends using the same
password for all components in the appliance (see "Changing individual device passwords (on page 41)").

Changing individual device passwords


During the installation of the appliance, the HP support team reset all passwords according to site
requirements. Use the following procedures if you need to change the passwords.

IMPORTANT: HP strongly recommends using the same password throughout the Parallel Data
Warehouse.

Changing passwords requires the IP addresses of the various system components, including iPDUs, iLO, and
InfiniBand switches.

Password management 41
Finding device IP addresses
Find the device IP addresses.
1. Log on to HST01 (physical server, not the VM).
2. Open the C:\PDWINST\Media folder.
3. Locate and open HPApplianceDetails.xml in a text editor. The file lists the IP addresses for the
system components, as shown in the following example:

iLO passwords
To change the iLO password you need to know the iLO IP addresses. You can find all server iLO IP addresses
on HST01 in the BmcAddress file under C:\PDWINST\Media\HPApplianceDetails.xml.
1. Open a browser window.
2. Enter one iLO IP address into the browser, and then press Enter.
The browser might return an error regarding a security certificate for this device.

To avoid this error, replace the default certificate with a trusted certificate. For more information, see the
"Appliance Configuration Tasks (SQL Server PDW)" in the SQLServerPDW.chm.

Password management 42
3. Click Continue to website (not recommended) to launch the iLO HP ProLiant log in screen.

4. Log in to the iLO 4 HP ProLiant as Administrator using the current password.


5. In the iLO Overview screen, click Administration to expand the list, and then select User Administration.

Password management 43
6. Under Local Users, select the box next to Administrator, and then click Edit.

7. Enter the new password information, and then click Save.

8. Repeat this procedure for the iLO on each server.

Ethernet switch passwords


Enabling Ethernet switch browser access
Requirements at your site might include disabling browser access to the Ethernet switches for security
reasons. You can enable browser access to simplify any necessary password changes. Disabling browser
access after you complete the password changes is optional.
1. Log on to the Ethernet switch as admin.
2. Enter system-view at the system prompt, and then press Enter.
3. Enter ip http enable, and then press Enter.
4. Enter save force, and then press Enter.
o Wait while the system validates the file.
o Continue with the next step when you see "Configuration is saved to device successfully."
5. Enter local-user admin, and then press Enter.
6. Enter service-type web at the luseradmin prompt, and then press Enter.
7. Enter save force, and then press Enter.
o Wait while the system validates the file.
o Continue when you see "Configuration is saved to device successfully."

Password management 44
Verify that the change was successful.
1. Enter display current-configuration at the system prompt, and then press Enter.
2. Scroll through the data returned by the command to local-user admin.
3. Verify that service-type web appears in the list as shown below.
#
local-user admin
password cipher $c$3$lNkvR8QxIYQAqdTqDdoewEMVl0NrXQqFt8XpDe4p
authorization-attribute level 3
service-type lan-access
service-type ssh terminal
service-type portal
service-type web
4. Continue with Ethernet switch passwords (on page 45) to change the Ethernet switch password using a
browser.

Ethernet switch passwords


To change the Ethernet switch password you need to know the Ethernet switch IP addresses. To find them, see
"Finding device IP addresses (on page 42)." You might also need to enable browser access to the Ethernet
switch (see "Enabling Ethernet switch browser access (on page 44)").
To change the Ethernet switch password:
1. Open a browser window.
Enter one Ethernet switch IP address into the browser, and then press Enter.
The browser might return an error regarding a security certificate for this device.

To avoid this error, replace the default certificate with a trusted certificate. For more information, see the
"Appliance Configuration Tasks (SQL Server PDW)" in the SQLServerPDW.chm.
2. Click Continue to website (not recommended) to launch the Web User Login.
3. Log into the switch as admin using the current password.

Password management 45
4. Click Summary, and then click Device in the left navigation pane.

5. Select Users from the Device list, and then click the admin username.
6. Click the Modify tab.

Password management 46
7. At the bottom of the screen, select the box next to Modify Password, and then enter the old and new
passwords.

8. Click Apply, and then click Logout in the upper right corner of the screen to exit the switch.
9. Repeat this procedure for each Ethernet switch in the configuration.

iPDU passwords
Using Telnet to change iPDU passwords
To change the iPDU password you need to know the iPDU IP addresses. To find them, see "Finding device IP
addresses (on page 42)."
If you have disabled web access on the iPDU for security reasons, use a serial cable to connect a laptop to
the iPDU. HP has provided an extension serial cable to simplify this laptop connection. The serial cable
provided is attached to the iPDU.
If you are able to use a browser, see "Using a browser to change iPDU passwords (on page 50)."
To change the iPDU password using PuTTY as the Telnet interface:
1. Attach your serial cable to the extension serial cable, which is strapped to the rear door of the
appliance, and then use PuTTY to access the iPDU. Do not use ssh.
2. Log on as admin using your current password.

Password management 47
3. At the prompt, enter 3 (User Accounts), and then press Enter.

4. Enter 1 (admin), and then press Enter.

Password management 48
5. Enter 2 (Change Password), and then press Enter.

6. Enter the new password information, and then press Enter.


7. Enter S (Save New Changes and Restart), and then press Enter.

8. Repeat this procedure for each iPDU in the configuration.

Password management 49
Using a browser to change iPDU passwords
To change the iPDU password you need to know the iPDU IP addresses. To find them, see "Finding device IP
addresses (on page 42)."
If you are unable to use a browser, see "Using Telnet to change iPDU passwords (on page 47)."
Change the iPDU password using a browser:
1. Open a browser window.
2. Enter one iPDU IP address into the browser, and then press Enter.
The browser might return an error regarding a security certificate for this device.

To avoid this error, replace the default certificate with a trusted certificate. For more information, see the
"Appliance Configuration Tasks (SQL Server PDW)" in the SQLServerPDW.chm.
3. Click Continue to website (not recommended) to launch the HP Intelligent Modular PDU software.
4. Log into the switch as Admin using the current password.

Password management 50
5. From the Home screen, select the Setup tab.

6. Click User Accounts in the left navigation pane.


7. Enter the new password in the Password field, and then enter the same password into the Verify
Password field.

8. Click Save Settings.

9. Click OK, and then click sign out in the upper right corner of the HP Intelligent Modular PDU screen.
10. Repeat this procedure for each Ethernet switch in the configuration.

Password management 51
FDR InfiniBand switch passwords
Using Telnet to change FDR InfiniBand switch passwords
To change the FDR InfiniBand passwords you need to know the FDR InfiniBand IP addresses. To find them,
see "Finding device IP addresses (on page 42)."
If you have disabled web access on the InfiniBand switches for security reasons, you can use a serial cable
to connect a laptop to the switch.
If you are able to use a browser, see "Using a browser to change FDR InfiniBand passwords ("Using a
browser to change FDR InfiniBand switch passwords" on page 52)."
To change the InfiniBand password using PuTTY as the Telnet interface:
1. Connect your laptop to the InfiniBand switch, and then use PuTTY to access the switch.
2. Log on as admin using your current password.
3. At the command prompt, enter Enable, and then press Enter.
4. Enter Configure terminal, and then press Enter.

5. Enter the new password at the Password prompt, and the press Enter.
6. Enter the new password again at the Confirm prompt, and then press Enter.
7. Enter Exit, and then press Enter.
8. Enter Exit, and then press Enter.
9. Repeat this procedure for each FDR InfiniBand switch in the configuration.

Using a browser to change FDR InfiniBand switch passwords


To change the FDR InfiniBand password you need to know the FDR InfiniBand IP addresses. To find them, see
"Finding device IP addresses (on page 42)."
If you are unable to use a browser, see "Using Telnet to change FDR InfiniBand passwords ("Using Telnet to
change FDR InfiniBand switch passwords" on page 52)."
Change the System Administrator password using a browser:
1. Open a browser window.
2. Enter one InifinBand IP address into the browser, and then press Enter.

Password management 52
The browser might return an error regarding a security certificate for this device.

To avoid this error, replace the default certificate with a trusted certificate. For more information, see the
"Appliance Configuration Tasks (SQL Server PDW)" in the SQLServerPDW.chm.
3. Click Continue to website (not recommended) to launch the Mellanox MLNX-OS Management Console.
4. Log into the switch as admin using the current password.

5. From the Summary screen, select the Security tab.

Password management 53
6. Click Edit next to the admin Username under User Accounts.

7. Enter the new password in the New password field, and then enter the same password into the Confirm
new password field.
8. Click Apply, and then click OK.

9. Click Save in the lower right corner.

Password management 54
Change the System Monitor password:
1. Click Edit on the System Monitor line under User Accounts.

2. Enter the new password in the New password field, and then enter the same password into the Confirm
new password field.
3. Click Apply, and then click OK.

4. Click Logout in the upper right corner of the MLNS-OS Management Console to log off of the switch,
and then log on again to verify the new passwords.
5. Repeat this procedure for each FDR InfiniBand switch in the configuration.

Password management 55
Server management

Starting Hyper-V Manager


Hyper-V Manager has many uses in HP AppSystem for Parallel Data Warehouse management. Consider
creating a desktop shortcut for easy access to this commonly used utility.
Start Hyper-V Manager:
1. Logon to HST01 using the PDW domain administrator account and password.
2. From the Server Manager Dashboard, click Tools, and then select Hyper-V Manager.

You can also log on to HST01 using the Windows 8 Administrative Tools:

Server management 56
Adding nodes
Hyper-V Manager should list every server in your configuration. In a single-rack configuration, for example,
you should see the following nodes:
<Fabric name>-HST01
<Fabric name>-HST02
<Fabric name>-HST03 (optional server)
<fabric name>-HSA01
<fabric name>-HSA02
<fabric name>-HSA03
<fabric name>-HSA04
<fabric name>-HSA05
<fabric name>-HSA06
<fabric name>-HSA07
<fabric name>-HSA08
Multiple-rack installations include a similar list for each rack. If Hyper-V Manager does not list all of the nodes
for your configuration, you can add them using the following procedure.
1. Logon to HST01 and launch Hyper-V Manager.
2. Right-click Hyper-V Manager in the left navigation, and then click Connect to server...

3. Select Another Computer, and then enter a server name into the text box.

Server management 57
You can also click Browse to locate the servers you want to add, and then click OK.

4. After connecting to all the servers, your Hyper-V Manger screen should appear similar to the following:

5. Verify that each HSA has one compute node VM and one iSCSI VM:
o HSA01 = CMP01 VM and ISCSI01
o HSA02 = CMP02 VM and ISCSI02
o HSA03 = CMP03 VM and ISCSI03
o HSA04 = CMP04 VM and ISCSI04
o HSA05 = CMP05 VM and ISCSI05
o HSA06 = CMP06 VM and ISCSI06
o HSA07 = CMP07 VM and ISCSI07
o HSA08 = CMP08 VM and ISCSI08
6. Check the State column for each compute node VM and each ISCSI VM.
7. Start any VMs that are not shown as Running:
a. Highlight the server in the left navigation pane.

Server management 58
b. Right-click the VM in the center pane, and then click Start.

If you find that any of the HSA servers are not aligned with the respective VMs (HSA01 with ISCI01and
CMP01), see "Aligning servers and VMs (on page 62)" to realign the VMs.

Using Failover Cluster Manager


The Failover Cluster Manager has many uses in HP AppSystem for Parallel Data Warehouse management.
Consider creating a desktop shortcut for easy access to this commonly used tool.
To access the Failover Cluster Manager:
1. Log on to HST01 using your domain administrator account and password.
2. From the Server Manager Dashboard, click Tools, and then select Failover Cluster Manager.

Server management 59
You can also access Failover Cluster Manager using the Windows 8 Administrative Tools.

3. Right-click Failover Cluster Manager in the left navigation pane, and then select Connect to Cluster...

4. Click the arrow next to the cluster to expand the cluster view.

Server management 60
I
In the list of nodes shown in the left panel of the Failover Cluster Manager, each server has a green server
icon, which indicates that the server is running.

Server management 61
Aligning servers and VMs
Recognizing misaligned VMs
Under normal operation, HST01 runs five roles: the Corralling Service Controller and the four virtual
machines, AD, VMM, MAD, and CTL. The remaining HSTs exist for backup purposes in case of a system
failure.

Occasionally, a VM becomes associated with an incorrect HST, as shown below where CTL01 VM is running
in HST02. CTL01 is a misaligned VM that you must move back to HST01 so that it is aligned with the
appropriate server and exists with the four Parallel Data Warehouse VMs on HST01.

In a configuration where HST01 and HST02 are the only passive servers available, this misalignment can
cause a reduction in availability levels because no passive servers are available if a compute node fails. This
occurs because Parallel Data Warehouse resource requirements do not allow two separate Compute VMs to
exist on the same passive server.
Because this misalignment can cause a system failure, regularly verify that all four VMs are properly aligned
on HST01 and take steps to realign the VMs whenever necessary.

Server management 62
Realigning VMs
HP AppSystem for Parallel Data Warehouse includes a Corralling Service Controller designed to keep the
HST01 VMs together on one node. If a VM becomes misaligned, you can move a VM from one node to
another. Be sure to stop the Corralling Service Controller first so that the service does not interfere with your
activities.

Shutting down the corralling service


Shut down the Corralling Service Controller:
1. From the Failover Cluster Manager, log onto the <appliance name>-HST01 server using your domain
account.
2. Under Nodes, select HST01.
3. Under Roles, right-click Corralling Service Controller, and then select Stop Role.
4. Wait for the Corralling Service Controller to stop.

Moving a VM
After you have shut down the Corralling Service Controller, you can move VMs from one location to another.
1. Under Nodes in the left navigation, right-click HST02.
2. Select Move > Quick Migration > Select Node.

Server management 63
3. In the Move Virtual Machine window, select HST01, and then click OK. Wait a few minutes for the
operation to complete.

Server management 64
Appliance management

Power up the appliance


The HP support team installs, connects, and powers up the Parallel Data Warehouse as part of the installation
process. Use the following procedure to power up the appliance if required.

IMPORTANT: When powering on the appliance, follow all steps in the exact order listed. Allow
each step to complete before starting the next, unless otherwise noted. Performing steps out of
order or without waiting for each step to complete could result in startup errors.

1. Power on all non-server components in this order:


a. PDUs—Make sure there are no error lights. Visually check power connections.
b. Switches (InfiniBand, Ethernet)—Make sure there are no error lights. Visually check connections.
c. Storage (D6000s).
Wait for all components to start. This takes a few minutes.
2. Power up the HSAxx servers:
a. Connect a monitor to each server as you power it up to watch each server boot (optional).
b. Start at the bottom of the rack, and power up the first HSAxx compute node server.
c. Wait approximately three minutes for the server to completely start before powering up the next
compute node server.
d. When the HSA compute node server is running, power up the next server above it in the rack.
e. Wait approximately three minutes for the server to completely start before powering up the next
compute node server.
f. Repeat steps d and e until you have powered up all of the HSA compute node servers.
g. After you have powered up all the HSAxx servers, let them run for about 15 minutes to make sure
that each server is running properly.
h. Verify that communication is available with the D6000 storage prior to booting the HSTxx servers
(on the top of the rack).
3. Check all components to verify that there are no errors. If errors exist, contact support (see, "Support
and other resources (on page 82)").
4. Power up the HSTxx servers starting from the top of the first rack:
a. Power up HST01, the top server in the rack.
b. Wait approximately five minutes for the server to start.
c. Power up HST02, and then wait approximately five minutes for the server to start.
d. If the optional HST03 server exists in the configuration, power up HST03, and then wait
approximately 10 minutes for the server to start.
5. See "Activating the software ("Starting the appliance software" on page 72)" to start the software and
VMs.

Appliance management 65
Backing up and restoring SQL Server 2012 Parallel
Data Warehouse data
Microsoft created backup and restore syntax for SQL Server 2012 Parallel Data Warehouse to capitalize on
the parallel nature of the appliance. For more information about backup operations, see "Backup and
Loading Hardware" in SQLServerPDW.chm.

To provide for more flexible disaster recovery architectures, HP AppSystem for Parallel Data Warehouse
enables you to connect multiple backup nodes to the appliance. For detailed information about backup and
restore software, see "Backup and Restore" in SQLServerPDW.chm.

Backup and restore syntax


Backing up HP AppSystem for Parallel Data Warehouse requires access to dwsql.exe. Initiate this
command session from the backup destination node, such as the optional Data Integration Platform. Add the
HP AppSystem Parallel Data Warehouse Cluster node CLUSTER IP address to the sqlcmd command:
sqlcmd -S "172.16.255.5,17001" -U sa -P password -I -d pdwdatabase
Parameters included in the example are defined as follows:

Appliance management 66
• -S—IP address of the Control node CLUSTER that the Admin Console, operation commands, and users
use for access.
• -U—User ID (in the example, sa).
• -P—System password.
• -d—Name of the database to back up.
This example assumes the database name of pdwdatabase and the backup folder name of
pdwdatabasebak.
Initiate the backup process.

IMPORTANT: When saving commands to a text file for reuse, be sure to change the encoding to
UTF8. This saves the single-quotations appropriately so you can avoid syntax errors when reusing
the command from the file.

The following examples show syntax for commonly used Backup and Restore options, extracted from "SQL
Reference\Data Definition Statements Backup Database and Restore Database" of SQLServerPDW.chm.
Backup syntax
Backup database to disk:
BACKUP DATABASE Invoices TO DISK =
'\\10.192.63.147\backups\yearly\Invoices2013Full';
Create a full backup of a user database or the master database:
BACKUP DATABASE database_name
TO DISK = '\\UNC_path\backup_directory'
[ WITH [ ( ] <with_options> [ ,...n ] [ ) ] ]
[;]
Create a differential backup of a user database:
BACKUP DATABASE database_name
TO DISK = '\\UNC_path\backup_directory'
WITH [ ( ] DIFFERENTIAL
[ , <with_options> [ ,...n ] [ ) ]
[;]

<with_options> ::=
DESCRIPTION = 'text'
| NAME = backup_name
Restore syntax
Restore a full database backup or dwsys metadata backup:
RESTORE DATABASE database_name
FROM full_or_metadata_backup_directory

Appliance management 67
[;]
Restore a full database backup, and then a differential backup:
RESTORE DATABASE database_name
FROM differential_backup_directory
WITH [ ( ] BASE = full_backup_directory [ ) ]
[;]
Restore header information from a database backup:
RESTORE HEADERONLY FROM backup_directory
[;]

Moving backup data to a corporate-wide backup system


Copy the backup directory files to a corporate-wide backup or tape library for disaster recovery purposes.
The system stores a backup as a set of files written to a directory under G:\backups on the Backup node.
After completing the backup, you can copy the backup files from the Backup node to another location using
the Windows file system. The restore operation uses the data in the G:\backups directory to restore the
data to the appliance.
Because the default storage location uses a disk configuration that optimizes backup and restore
performance, moving a backup set off the Backup node can negatively impact restore performance, even
when you move files to a different logical disk configuration. Whenever possible, perform restore operations
from the backup files as they were originally stored on the Backup node. Only restore from files stored in a
corporate-wide backup or tape library if the files on the Backup node are no longer available.

Power down the appliance


1. Connect to the HST01 server using the Domain Administrator account.
2. Open the Failover Cluster Manager from HST01.
3. Connect to the MAD01 virtual machine
4. Launch DWConfig.exe from the MAD01 VM:
C:\Program Files\Microsoft SQL Server Parallel Data
Warehouse\100\dwconfig.exe
5. Stop the appliance using the DWConfig.exe utility:
a. Click Service Status in the left navigation pane.
b. Click Stop Appliance.
6. From the Failover Cluster Manager:
a. Stop the Corralling Service Controller.
b. Stop all VMs.
7. Power down individual servers.
8. Power off individual storage servers.
9. Power off iPDUs.

Appliance management 68
Starting the VMs
To configure HP AppSystem Parallel Data Warehouse, make sure you are logged in as the PDW Fabric
Administrator and not as PDW Domain Administrator. Both the PDW Fabric Domain and the PDW Domain
Administrator name were provided in the Parallel Data Warehouse Site Preparation survey.
• PDW Fabric Administrator—Tied to the Active Directory Domain used for the Parallel Data Warehouse
hardware components.
• PDW Domain Administrator—Related to the Active Directory domain and used for the Parallel Data
Warehouse virtual components.
If you are unsure of your login status, enter whoami at a command prompt.

IMPORTANT: To avoid any startup issues, start the Virtual Machines in the order documented.

Starting AD VM
1. Log on to HST01 using Hyper-V Manager (see "Starting Hyper-V Manager (on page 56)").
Hyper-V Manager should list every server in your configuration. In a single-rack configuration, for
example, you should see the following nodes:
<Fabric name>-HST01
<Fabric name>-HST02
<Fabric name>-HST03 (Customer optional server)
<fabric name>-HSA01
<fabric name>-HSA02
<fabric name>-HSA03
<fabric name>-HSA04
<fabric name>-HSA05
<fabric name>-HSA06
<fabric name>-HSA07
<fabric name>-HSA08
2. Select HST01. You should see the following list of available roles:

3. Select <fabric name>-AD:

Appliance management 69
o If you are unable to see <fabric name>-AD on HST01, check each HST server to locate it, and then
move it to HST01 (see, "Aligning servers and VMs (on page 62)").
o If you are still unable to find <fabric name>-AD, reboot HST01 and check again.
o When all VMs are aligned on HST01, continue with the next step.
4. Right-click <fabric name>-AD, and then click Start.
Wait at least three minutes for it to fully start. The VM is fully started when the computer icon next to it
turns green.

Starting the remaining VMs on HST01


1. Log on to Failover Cluster Manager.
2. Right click Corralling Service Controller, and then click Start Role. to automatically start the VMM,
MAD01, and CTL01 VMs.
If the VMs do not start, start each one manually using the following procedure:
o Right-click <fabricdomain>-VMM, and then click Start. Wait for it to fully start.
o Right-click <pdwdomain>-MAD01, and then click Start. Wait for it to fully start.
o Right-click <pdwdomain>-CTL01, and then click Start. Wait for it to fully start.
Each VM takes approximately three minutes to fully start. A green computer icon next to each VM
indicates that the VM is fully started.
3. Click HST01 under Nodes, and then check the center pane to verify the presence and status of each of
the four VMs (AD, VMM, MAD01, CTL01) and the Corralling Service Controller.

Starting the VMs on the HSA nodes


Each HSA node has one compute VM (CMPxx) and one iSCSI VM (iSCSIxx), similar to the following list:
• HSA01 = CMP01 VM and ISCSI01
• HSA02 = CMP02 VM and ISCSI02
• HSA03 = CMP03 VM and ISCSI03
• HSA04 = CMP04 VM and ISCSI04

Appliance management 70
• HSA05 = CMP05 VM and ISCSI05
• HSA06 = CMP06 VM and ISCSI06
• HSA07 = CMP07 VM and ISCSI07
• HSA08 = CMP08 VM and ISCSI08
Like the HST VMs, the HSA VMs can become misaligned. Use the following procedure to check the alignment
of each VM, and then start it. If you need to realign any VMs, see "Aligning servers and VMs (on page 62)."
1. Log on to the Failover Cluster Manager.
2. Click an individual HSA node to make sure that it is running its respective iSCSI and CMP VMs.

3. Check the Status column for each VM.


o If the Status of both VMs is Running, repeat steps 1 and 2 for the next HSA node.
o If the Status of either VM is not Running continue with the next step.
4. Right click the CMP VM (CMPxx), and then click Start.
5. Wait about 10 minutes for CMPxx to start running. A green computer icon next to the VM name
indicates that it has started. You can also see a started VM in the bottom ribbon of the Failover Cluster
Manager screen. Click the down-arrow to expand the view for more detail about that VM.

6. Right-click the iSCSI VM (iSCSIxx), and then click Start.


7. Wait several minutes for the iSCSI VM to start. A green computer icon next to the VM name indicates
that it has started.

Appliance management 71
8. Repeat this procedure for the VMs on each HSA node in the configuration.

Starting the appliance software


With the HP AppSystem for Parallel Data Warehouse hardware powered on, start the software.
Activate the SQL Server 2012 Parallel Data Warehouse cluster and services:
1. Use Windows Remote Desktop Connection to access the primary Management node (MAD01). If you
do not know the IP address for MAD01, see "Finding device IP addresses (on page 42)."

2. On the remote server, run the SQL Server 2012 Parallel Data Warehouse from following path:
C:\Program Files\Microsoft SQL Server Parallel Data
Warehouse\100\dwconfig.exe
3. Create a shortcut to dwconfig on the MAD01 desktop.
4. Click Service Status in the left navigation pane.
5. Click Start Appliance.

Configuring the appliance


This section describes the SQL Server 2012 Parallel Data Warehouse Configuration Manager screens and
functions. For more information, see “Tools Reference\SQL Server 2012 Parallel Data Warehouse
Configuration Manager” in the SQLServerPDW.chm, available on each server in HP AppSystem for Parallel
Data Warehouse.

Appliance management 72
HP Services performs the tasks in this section during the appliance installation. HP recommends reviewing
each function within the Configuration Manager to be sure that the parameters are set to the appropriate
values, and to verify the IP address firewall and security settings that allow the HP AppSystem for Parallel
Data Warehouse access to and from your corporate network.
Always use SQL Server 2012 Parallel Data Warehouse Configuration Manager to start and stop services on
the HP AppSystem for Parallel Data Warehouse.

Running Configuration Manager


After logging onto MAD01 VM (see “Starting the appliance software (on page 72)”), you can manage many
Parallel Data Warehouse functions using dwconfig. Consider creating a shortcut to dwconfig on the desktop
for easier access to this commonly used utility.
1. Run the SQL Server 2012 Parallel Data Warehouse Configuration Manager (dwconfig) from the
following path on the MAD01 VM:
C:\Program Files\Microsoft SQL Server Parallel Data
Warehouse\100\dwconfig.exe

Appliance management 73
Appliance Topology screen
When you execute dwconfig, the Appliance Topology screen appears and lists all of the nodes available on
the Appliance (physical servers and VMs).

From this location, you can view and modify many of the HP AppSystem for Parallel Data Warehouse
configuration settings. On the main page you can see the Appliance Domain Name, and then scroll down
to see each individual node name.

Appliance Password Reset screen


As part of the HP AppSystem for Parallel Data Warehouse installation and startup service, HP Support
changes the factory default passwords to site-specific passwords. You can change passwords at any time
using the Password Reset screen.
Passwords must meet the following requirements:
• 8 to 15 characters
• Lower case or upper case characters
• Numbers 0-9
• No special characters

Appliance management 74
Use the menu to change the associated passwords on the appliance. HP recommends using the same
password for all components in the appliance (see "Changing individual device passwords (on page 41)").

Appliance Time Zone Configuration screen


Use this screen to verify the time zone for the appliance, and to enable the system to adjust for Daylight
Saving Time.

Appliance Network Configuration screen


The Network Configuration tab presents the IP configuration information for various HP AppSystem for
Parallel Data Warehouse components. Click the arrow next to the Node name to see a menu of the available
components on that node.

Appliance management 75
You can also use this screen to change IP addresses for the Parallel Data Warehouse servers and Virtual
Machines.

Appliance Certificate Provisioning screen


Use this screen to provision certificates for establishing secure connections to HP AppSystem for Parallel Data
Warehouse. You can obtain certificates locally or from a server within your corporate network.

Appliance Firewall Configuration screen


Use this screen to enable or disable firewall rules that allow or prevent access to specific ports on HP
AppSystem for Parallel Data Warehouse.

IMPORTANT: Do not change the firewall settings that are pre-configured in the HP factory. You
can add firewall entries to enable the SQL Server 2012 Parallel Data Warehouse software to
connect to your internal corporate network.

Appliance management 76
Appliance Services Status screen
Use this screen to view the status of various SQL Server 2012 Parallel Data Warehouse services.
This screen provides the buttons used to start or stop the appliance services.
• When shutting down, click Stop Appliance to take the cluster offline and shut down the SQL Server
2012 Parallel Data Warehouse services. Then power-off the hardware (servers, storage, and switches).
• When starting up, click Start Appliance to start the cluster and start the SQL Server 2012 Parallel Data
Warehouse services.

Appliance management 77
Privileges (Instant File Initialization) screen
Use this screen to provide Instant File Initialization, which reclaims used disk space without filling space with
zeros. Instant File Initialization overwrites disk content with the new data written to the files. Log files cannot
be initialized instantaneously.
When enabled, data file operations can execute more quickly. Enable this feature after you have granted the
Network Service account SE_MANAGE_VOLUME_NAME privilege.

Restore Master Database screen


The Restore Master Database screen enables you to specify the master backup folder on a network drive. This
network drive must be available when performing HP AppSystem for Parallel Data Warehouse backup and
restore operations.

Appliance management 78
Configuring Windows Server Update Services
Configure Windows Server Update Services (WSUS). For more information, see "System
Administration/Windows Server Update Services" in the SQLServerPDW.chm help file.

Appliance management 79
Appendix

HP Data Integration Platform


An HP AppSystem for Parallel Data Warehouse configuration might include the optional Data Integration
Platform for use as a landing zone and backup server. The information in this section provides details about
the optional Data Integration Platform server.

InfiniBand Data Integration Platform connectivity


When connecting external servers to HP AppSystem for Parallel Data Warehouse, available ports typically
include ports 19 through 30. However, if the site uses InfiniBand to connect to the network, it is likely that port
19 was used for that purpose.
The following table shows the available ports. HP recommends starting at port 20 and working up.

10 Gb Ethernet connectivity
10 Gb Ethernet provides an alternative to InfiniBand for connecting the Data Integration Platform to HP
AppSystem for Parallel Data Warehouse. If your configuration uses 10 Gb Ethernet, the XG E2/2 port of the
topmost Ethernet switch is not available for HP AppSystem for Parallel Data Warehouse.
Although you can use either of the ports indicated in the following table for the Data Integration Platform
connection, HP recommends using the XG E2/2 port on the bottom Ethernet switch.

Appendix 80
If the site has more than one landing zone or backup node, you can use the same ports in the switches on
another rack.

1 Gb Ethernet connectivity
Because it is not a high speed connection, you will most likely not use port 24 or 25 for 1 Gb Ethernet
connectivity. If you do, you may use either port for Data Integration Platform connectivity or to connect to
other landing zone or backup servers in the environment.

HP Insight Remote Support software and HP


AppSystem for Parallel Data Warehouse
HP Insight Remote Support is a key component of HP Proactive Care support from HP. This software provides
continuous proactive monitoring of the hardware components within the Parallel Data Warehouse
environment. HP installs Insight Remote Support on the optional Data Integration Platform, which is Windows
Server 2008 R2 compliant. Insight RS 7.0.5 is not compatible with Windows Server 2012. To install Insight
RS on your own Windows Server 2008 R2 server, and to configure Insight RS for use with HP AppSystem for
Parallel Data Warehouse and Microsoft SQL Server 2012 Parallel Data Warehouse, see "Installing and
Configuring HP Insight Remote Support for the HP AppSystem for SQL 2012 Parallel Data Warehouse
Appliance" available from the HP website
(http://h20565.www2.hp.com/portal/site/hpsc/public/psi/home/?sp4ts.oid=4346326).
For additional information about Insight RS, see the HP website
(http://www.hp.com/go/insightremotesupport).

Appendix 81
Support and other resources

HP AppSystem Parallel Data Warehouse Support


Pack
Keeping HP AppSystem for Parallel Data Warehouse running at maximum performance is critical and
requires updating firmware and drivers to maintain the overall health of servers, storage arrays, switches,
and all components included in the hardware stack. Developing a methodology for maintaining firmware on
a complex hardware stack like the multi-server, MPP appliance powered by SQL Server 2012 Parallel Data
Warehouse Edition, and HP AppSystem for Parallel Data Warehouse, requires a number of considerations.
HP AppSystem for Parallel Data Warehouse requires a specific, tested firmware and driver software stack
that uses consistent firmware and driver versions across all similar nodes and subsystems throughout the
appliance. Designed to help you simplify installation and maintenance tasks, the HP AppSystem for Parallel
Data Warehouse Support Pack utility suite for HP AppSystem for Parallel Data Warehouse provides a unique
set of tools and a collection of solution tested HP firmware and drivers to maintain and support the correct
MRA configuration level. All appliance maintenance procedures require careful coordination with support.
For more information see, "HP AppSystem for Parallel Data Warehouse support recommendations ("HP
AppSystem Parallel Data Warehouse support recommendations" on page 83)."
HP support customers can download the HP AppSystem for Parallel Data Warehouse Support Pack from the
HP Software Updates website (http://www.hp.com/go/hpsoftwareupdatesupport).
See HP AppSystem for Microsoft SQL Server 2012 Parallel Data Warehouse appliance Support Pack
documentation, available from the HP website
(http://h20565.www2.hp.com/portal/site/hpsc/public/psi/home/?sp4ts.oid=4346326).

Download and installation


1. Download the latest Support Pack from the HP website
(http://www.hp.com/go/hpsoftwareupdatesupport).
2. Log in using HP Passport.
3. Enter the appropriate customer information under Directly Enter SAID.
4. Review and accept the Terms and Conditions.
5. Click View available products. If you have used this site before, click View previously selected products,
and then skip the following step.
6. Enter PDW in the Search for products in all categories / product centers field.
7. Click Get software updates.
8. Select the latest version of the Complete Tool, which includes all of the tools in the latest toolkit, or select
an individual Support Pack tool from the same location by choosing a tool from the list.
9. Copy the Support Pack zip file to the management node:
<appliance name>-MAD01
10. Double-click the zip file, and then click Extract all files.

Support and other resources 82


NOTE: Each zip file contains two additional zip files, each specific to the base hardware
platform (D6000 or P2000). For example, Complete Tool.zip includes
CompleteTool_v1hardware.zip for P2000-based hardware and
CompleteTool_v2hardware.zip for D6000- based hardware.

11. In the Files will be extracted to this folder text box, enter c:\HP PDW Support Pack\, and then click
Extract.
o If you have downloaded the Complete Tool, the file unzips to c:\HP PDW Support
Pack\CompleteTool\CompleteTool\
o If you have downloaded an individual tool, the file unzips to c:\HP PDW Support
Pack\<toolname>\<toolname>
12. Unzip the appropriate hardware-based folder. Make note of the directory location of the files
associated with the support pack so that you can run the various scripts provided with the tool.

HP AppSystem Parallel Data Warehouse support


recommendations
HP AppSystem for Parallel Data Warehouse requires a minimum of HP Proactive Care support to help
proactively prevent issues, minimize downtime, and mitigate risk in converged, virtualized environments.
Alternatively, HP provides HP Critical Service to enhance the support experience. HP and Microsoft have
worked together to build a coordinated support process with a common goal of providing a joint solution
support experience for HP AppSystem for Microsoft SQL Server 2012 Parallel Data Warehouse customers.
HP and Microsoft collaborate together to resolve issues. You are encouraged, but not required, to contact
Microsoft first for issues with HP AppSystem for Parallel Data Warehouse.

Contacting support for HP AppSystem Parallel Data


Warehouse
To contact HP Support for HP AppSystem for Parallel Data Warehouse, use the phone number provided with
your HP service contract documentation. Your specific support phone number is tied to your service contract
and the specific level of support purchased.
Alternatively, you may submit an HP support case online using the HP Support Case Manager. When
submitting an HP support case online for HP AppSystem for Parallel Data Warehouse, use pin code# 33900
to ensure proper routing of the case to an HP AppSystem for Parallel Data Warehouse Support Engineer.
When calling HP Support or Microsoft Support for HP AppSystem for Parallel Data Warehouse issues,
provide your appliance Solution ID. You can find the Solution ID label at the top rear underside of the HP
AppSystem for Parallel Data Warehouse base rack. The label contains a unique combination of the product
SKU number and serial number.
Be sure to have the following information available before you call HP Support:
• HP AppSystem for Parallel Data Warehouse Solution Identifier (Solution ID)
• Applicable error messages
• Operating system type and revision level

Support and other resources 83


• HP AppSystem for Parallel Data Warehouse MRA version

HP contact information
For United States and worldwide contact information, see the Contact HP website
(http://www.hp.com/go/assistance).
In the United States:
• To contact HP by phone, call 1-800-334-5144. For continuous quality improvement, calls may be
recorded or monitored.
• If you have purchased a Care Pack (service upgrade), see the Support & Drivers website
(http://www8.hp.com/us/en/support-drivers.html). If the problem cannot be resolved at the website,
call 1-800-633-3600. For more information about Care Packs, see the HP website
(http://pro-aq-sama.houston.hp.com/services/cache/10950-0-0-225-121.html).

Support and other resources 84


Acronyms and abbreviations

AD
Active Directory

ADO
ActiveX Data Objects

CTL
Control VM

DBA
Database Administrator

DVD
digital video disc

ELT
extract, load, and transform

ETL
extract, transformation, and load

FC
Fibre Channel

FDR
fourteen data rate

HTA
host storage attached

HTTPS
hypertext transfer protocol secure sockets

IB
InfiniBand

Acronyms and abbreviations 85


iLO
Integrated Lights-Out

JDBC
Java database connectivity

LFF
large form factor

MDX
multidimensional expression

MDX
Management VM holding active directory

MPP
massively parallel processing

MSA
Modular Smart Array

ODBC
open database connectivity

OLAP
on line analytical processing

OLE DB
object linking and embedding database

PDU
power distribution unit

SAS
serial attached SCSI

SATA
serial ATA

SFF
small form factor

Acronyms and abbreviations 86


SMP
Server Migration Pack

SSIS
SQL Server Integration Services

USB
universal serial bus

VM
Virtual Machine

VMM
Virtual Machine Manager

VMM
data movement service

WSUS
Windows Server Update Services

Acronyms and abbreviations 87


Documentation feedback

HP is committed to providing documentation that meets your needs. To help us improve the documentation,
send any errors, suggestions, or comments to Documentation Feedback (mailto:docsfeedback@hp.com).
Include the document title and part number, version number, or the URL when submitting your feedback.

Index 88
Index

A F
accessing the Admin Console 17 features 5
accessing the appliance 12 firewall settings, configuring 72, 76, See Ethernet
adding new users 16 switches
Admin Console, accessing 12
Admin Console, features 17 H
Admin Console, overview 17
help resources 84
appliance information, viewing 5, 18, 19, 56, 59,
HP contact information 84
77
HP Insight Remote Support software 80, 81
authorized reseller 82, 84
HP technical support 82, 84
HP website 84
B
backing up data 21, 66, 68, 78 I
backup information, viewing 21
iLO network 42
backup syntax 21, 66
InfiniBand 52, 80
before you contact HP 84
introduction 5
IP address, external 28, 42, 75
C
certificates 76 L
changing the password 41, 44, 47, 52
load task information, viewing 20
client software, installing 27
loading data, prerequisites 38
client utilities 26, 27, 28, 33, 34
loading data, tools 32, 37, 38
columnstore, xVelocity 32
locks, viewing 23
Configuration Manager, accessing 59, 72, 73
configuring network settings 45, 75
configuring the appliance 34, 72, 75 M
connection strings 27 management tools 59, 63, 73
contact information 84
contacting HP 84 N

D network settings 28, 44, 50, 75, 80


node details 57, 69, 70, 74
data loading prerequisites 31, 36 node setup 57
data loading tools 38 node status 56, 70
distributed tables 32 nodes, viewing 70, 74
documentation feedback 88
O
E
overview 5
Ethernet switches 44, 45, 80, 81 overwriting data 78

Index 89
P W
password, changing 41, 44, 45, 47, 50, 52 waits, investigating 23
performance monitoring 20, 22, 23, 25, 26, 31 website, HP 84
phone numbers 84 Windows Updates 79
ports 80
powering up the components 65
protocol support 27

Q
query information, viewing 19, 28, 29

R
Remote Support, configuration 81
replicated tables 32
resources 23, 82
restore information, viewing 21
restoring data 66, 78

S
secure web access 72, 76
starting the service 72, 77
status, node 74
stopping the service 68, 77
storage allocation 23, 78
strings, connection 27
support 82, 83, 84
support and other resources 82

T
tables 32
technical support 82, 84
telephone numbers 82, 84
time zones 75
topologies 74
troubleshooting 20, 25, 62, 63, 78

U
updating the firmware 82
user account, adding 16

V
viewing session information 18
virtual machine host, registration 69, 70
virtual machine migration (VMM) 63

Index 90

Potrebbero piacerti anche