The NVMe Management Interface (NVMe-MI):

Overview and New Developments

Peter Onufryk
Microsemi

NVMe Developer Days 2018

San Diego, CA 1
 Agenda

 NVMe-MI 1.0a Overview

 New Features in NVMe-MI 1.1
 NVMe-MI Status
 Summary

NVMe Developer Days 2018

San Diego, CA 2
 NVM Express, Inc.
120 Companies Defining NVMe Together

NVMe Developer Days 2018

San Diego, CA 3
 What is the NVMe Management Interface
1.0a?

A
A programmable
programmable interface
interface that
that allows
allows out-of-band
out-of-band management
management of
of
an
an NVMe
NVMe Storage
Storage Device
Device Field
Field Replaceable
Replaceable Unit
Unit

NVMe Developer Days 2018

San Diego, CA 4
 NVMe Storage Device Management
Server
Server Caching
Caching Server
Server Storage
Storage External
External Storage
Storage

Root
Root Root
Root Root Root
Complex NVMe
NVMe Complex Controller Root Root
Controller
Complex Complex Complex
Complex Complex
Complex
A B
x16 x16 x16 x16
PCIe
PCIe PCIe
SAS PCIe SAS
PCIe
PCIe PCIe/PCI
PCIe/PCI SAS Switch
Switch Switch
Switch
SAS
Switch
Switch e
e RAID
RAID
x4 x4 NVMe
NVMe
NVMeNVMe
NVMe
NVMe
NVMe
NVMe
NVMe
NVMe NVMe
NVMe NVMe
NVMe NVMe
NVMe NVMe
NVMe NVMe
NVMe

SAS
SAS
HDD
HDD

 Example Pre-boot Management

• Inventory, Power Budgeting, Configuration, Firmware Update
 Example Out-of-Band Management During System Operation
• Health Monitoring, Power/Thermal Management, Firmware Update, Configuration

Flash Memory Summit 2014

Santa Clara, CA 5
 Why Standardize
NVMe Storage Device Management?
 Reduces Cost and Broadens Adoption
• Allows OEMs to source storage devices from multiple suppliers
• Eliminates need for NVMe storage device suppliers to develop custom OEM specific
management features

 Consistent Feature Set

• All storage devices that implement management implement a common baseline feature set
• Optional features are implemented in a consistent manner

 Industry Ecosystem
• Compliance tests / program
• Development tools

Flash Memory Summit 2014

Santa Clara, CA 6
 NVMe Architecture
NVM Subsystem - one or more controllers, one or more namespaces, one or more
PCI Express ports, a non-volatile memory storage medium, and an interface
between the controller(s) and non-volatile memory storage medium

NVM Subsystem with NVM Subsystem with

One Controller and One Port Two Controllers and Two Ports
NVMe Developer Days 2018
San Diego, CA 7
 NVMe Storage Device in NVMe-MI 1.0a

NVMe Storage Device – One NVM Subsystem with one or more ports, optional FRU
Information Device, and an optional SMBus/I2C interface
NVMe Developer Days 2018
San Diego, CA 8
 Vital Product Data (VPD)
• Utilizes IPMI Platform Management FRU Information Storage Definition with NVMe-MI
extensions
• The VPD may be accessed using two methods
– NVMe-MI commands over MCTP
– SMBus/I2C interface using I2C operations as defined by IMPI Platform Management FRU
Information Storage Definition
VPD Elements
Common Header
Product Info Area (optional)
NVMe MultiRecord Area
NVMe PCIe Port MultiRecord Area
Internal Use Area (optional)
Chassis Info Area (optional)
Board Info Area (optional)
NVMe Developer Days 2018
San Diego, CA 9
 Out-of-Band Management and NVMe-MI
• Out-of-Band Management – Management that operates with hardware
resources and components that are independent of the operation system
control
• NVMe Out-of-Band Management
Interfaces
– SMBus/I2C
– PCIe Vendor Defined Messages (VDM)
– IPMI FRU Data (VPD) accessed
over SMBus/I2C

NVMe Developer Days 2018

San Diego, CA 10
 NVMe-MI Protocol Layering

NVMe Developer Days 2018

San Diego, CA 11
 NVMe-MI 1.0a Command Set Overview
Command Type Command Command Type Command
Read NVMe-MI Data Structure Firmware Activate/Commit
NVM Subsystem Health Status Poll Firmware Image Download
Controller Health Status Poll Format NVM
NVMe
Management Configuration Get Get Features
Interface Configuration Set Get Log Page
Specific VPD Read NVMe Identify
Commands Admin
VPD Write Namespace Management
Commands
Reset Namespace Attachment
Vendor Specific Security Send
PCIe Configuration Read Security Receive
PCIe Configuration write Set Features
PCIe I/O Read Vendor Specific
PCIe
PCIe I/O Write
Command
PCIe Memory Read
PCIe Memory Write
Vendor Specific

NVMe Developer Days 2018

San Diego, CA 12
 New Features in NVMe-MI 1.1

 In-Band NVMe-MI
 Enclosure Management
 NVMe Storage Device Extensions

NVMe Developer Days 2018

San Diego, CA 13
 In-Band Management and NVMe-MI

• In-band mechanism allows application to tunnel

NVMe-MI commands through NVMe driver
– Two new NVMe Admin commands
– NVMe-MI Send
– NVMe-MI Receive
• Benefits
– Provides management capabilities not
available in-band via NVMe commands
– Efficient NVM Subsystem health status
reporting
– Ability to manage NVMe at a FRU level
– Vital Product Data (VPD) access
– Enclosure management

NVMe Developer Days 2018

San Diego, CA 14
 NVMe JBOFs

Facebook Lightning PCIe NVMe JBOF

NVMe Developer Days 2018

San Diego, CA 15
 NVMe Enclosure Management
• Native PCIe Enclosure Management (NPEM)
• Submitted to the PCI-SIG Protocol Workgroup (PWG) on behalf of
the NVMe™ Management Interface (NVMe-MI™) Workgroup
• Approved by PCI-SIG on August 10th, 2017
• Transport specific basic enclosure management

• SCSI Enclosure Services (SES) Based Enclosure Management

• Technical proposal developed in the NVMe-MI workgroup
• While the NVMe and SCSI architectures differ, the elements of an
enclosure and capabilities to manage them are the same
• Example enclosure elements: power supplies, fans, display or indicators,
locks, temperature sensors, current sensors, voltage sensors, and ports
• Comprehensive enclosure management for NVMe that leverages
(SES), a standard developed by T10 for management of enclosures
using the SCSI architecture
NVMe Developer Days 2018
San Diego, CA 16
 NVMe Storage Device in NVMe-MI 1.0a

NVMe Storage Device – One NVM Subsystem with one or more ports, optional FRU
Information Device, and an optional SMBus/I2C interface
NVMe Developer Days 2018
San Diego, CA 17
 NVMe Storage Devices in NVMe-MI 1.1

M.2 Carrier Board from Amfeltec ANA Carrier Board from Facebook

NVMe Developer Days 2018

San Diego, CA 18
 NVMe-MI 1.1 Storage Device Extensions
NVMe-MI 1.0a NVMe-MI 1.1
NVMe Storage Device An NVMe Storage Device consists of one NVM An NVMe Storage Device contains at least one NVM
Definition Subsystem with one or more ports, optional FRU subsystem or Expansion Connector and at least one
Information Device, and an optional SMBus/I2C Upstream Connector. It may contain additional
interface elements such as: FRU Information Devices, PCIe
switches, and SMBus/I2C Muxes.
Supports Multiple NVM No Yes
subsystems

Supports Carriers No Yes

Elements NVM Subsystem and FRU Information Device NVM Subsystem, Expansion Connector, Upstream
Connector, FRU Information Device, PCIe Switch,
and SMBus/I2C Mux
VPD Describes basic characteristics of NVMe Storage Describes characteristics of NVMe Storage Device,
Device elements that make up the NVMe Storage Device,
and the element topology

NVMe Developer Days 2018

San Diego, CA 19
 NVM Express Roadmap

NVMe Developer Days 2018

San Diego, CA 20
 NVMe-MI Ecosystem

 Commercial test equipment and conformance tests exist for

NVMe-MI
 NVMe-MI 1.0a compliance testing program exists
• Compliance testing started in the May 2017 NVMe Plugfest
conducted by the University of New Hampshire Interoperability
Laboratory (UNH-IOL)
• Multiple devices from multiple vendors have passed compliance
testing and are on the NVMe-MI Integrators List
 Servers are shipping that support NVMe-MI

NVMe Developer Days 2018

San Diego, CA 21
 Summary

 NVMe-MI 1.0a has been released

• Focused on managing NVMe Storage Devices (e.g., SSDs)
• SSDs and systems are shipping that support NVMe-MI 1.0a
 NVMe-MI 1.1 is nearing completion
• Technical work is completed and a ratified specification is expected
in Q1’19
• Key new features in NVMe-MI 1.1
– In-band NVMe-MI
– Enclosure Management
– NVMe Storage Device Extensions

NVMe Developer Days 2018

San Diego, CA 22