Getting Started with Product Analytics March 19, 2017 (Sunday)

UpGrad Xchange, Bangalore

A Workshop for Beginner & Aspiring Product Managers
What is all the fuss about?
Making a case for product analytics & feedback loops

In God we You cannot Without data,

trust. Everyone improve upon youre just
else must bring something you another person
along data. cant measure. with an opinion.
Proctor & Gamble operates on this Pretty obvious no? You dont want to fight on opinions!

Plan - Design - Spec - Build - Launch - Measure - Iterate

Typical product cycle for PMs
PMs and data (hence,
analytics) are interrelated

Data is all pervasive!

Lets take a PM walk through the how products

are developed to understand why you cannot fly
blind when building a product or it features.

Associated Costs with Product Development

Time to build & grow
Money to build & grow
Resources to build & grow
Opportunity Cost (!)
often ignored
Meet the Empathy Debt
A gauge of whether you understand your users and how they use your product

Let customers do the Let numbers do the

talking for you talking for you
Customer research, interviews, ethnography Product & Marketing Analytics, Deductions
(External Feedback Loop) (Internal Feedback Loop)
Getting it right
Why is it absolutely critical to get your product analytics implementation right?

Measurement vs Analytics

Data is the new oil.

Its only useful
when its refined.
Jess Greenwood, Contagious
Real-World Examples
How analytics data is being used to drive product & business decisions
Some essentials before we begin
Lets all get on the same page
What will we NOT discuss about?
Data Analytics Techniques
Data Mining & Manipulation
Data Analytics Algorithms
and so on
What will we discuss about?
Product Analytics
Product Metrics
User-level vs aggregate
Marketing & Campaign Metrics
Implementing product analytics
What should you keep in mind?
Datapoints - Identifier, Attributes,
Derived Metrics, Dimensions
Definition Sessions, Events, User
Paradigms Segments, Cohorts,
Funnels, Experiments
Reporting and Visualization
Analysis, Hypothesis Verification
Analytics Tools
Correlation between Metrics
Misleading Metrics
Vanity vs Real/Actionable metrics
Quiz Time
Lets put on our thinking caps!

What do you think of these?

Are these vanity or insightful metrics?

# of Downloads
App Install Base
Total Revenue
# of Daily App Opens
Session Interval
Session Length
# of Daily Users
Implementing Product Analytics
Startups (or SMBs) vs Established Enterprises

STARTUP / SMB ENTERPRISES

Starved of capital, analysts and engineering Abundance of capital and resources, hence
resources, hence prefer an out-of-box 3P solution prefer in-house, custom built solutions also
owing to IP protection & tight data-
Focus is on integration to hit the ground running sharing/privacy policies

Often different analytics tools each plugged into Often uniform analytics platform powering
different properties different product lines
Implementing Product Analytics
Learning through a Group Activity

Implementation Pipeline

> > > >

Plan Design Spec Execute Rollout
Implementing Product Analytics
Learning through a Group Activity

Our Case Companions

Industry: Ecommerce, Shopping Industry: Real-time Communication

Product: Physical (Groceries) Product: Virtual, Software
#1 Business Goal: Purchase #1 Business Goal: Usage (+Conversion)
Implementing Product Analytics
Learning through a Group Activity

Some pointers towards what to measure?

Acquisition: Downloads, Clicks, Visits etc. Acquisition: Downloads, Clicks etc.

Activation: Onboarding, Login/Register
Engagement: category & product views, size select
Retention: Loyalty/stickiness, repeat visit/purchase
Referral: Viral growth, Share, Invitation
Monetization: Conversion, Subscription
Activation: Onboarding, Register
Engagement: Opens, # of chats, calls etc.
Retention: Frequency of users coming back, duration etc.
Referral: Viral growth, Share, Invitation
Monetization: Subscription, In-app purchases
Planning Your Product Analytics
Commandment #1

How does analytics tie up with larger business goals?

What business
decisions and actions

should your analytics
implementation aid?
Planning Your Product Analytics
Commandment #1 (contd.)

Examples of Business Goals / Decisions

Product Usage
What business Effectiveness of product discovery properties (CTRs etc.)
User flow between screens
decisions and actions
should your analytics
implementation aid?

Efficacy of Traffic & Marketing Channels
Traffic Sources effectiveness and quality
Attribution of a sale to a traffic source

Merchandise Performance
Product impressions, views/clicks, conversions
Product categories performance

Communication & Campaign Management

Single tool vs different tools for marketing campaigns
Real-time performance tracking of marketing campaigns
Planning Your Product Analytics
Commandment #2

Why does it matter?

Product requirements
Does your analytics

Delivery timelines
Selection of analytics tools
implementation span Infrastructure

across platforms or is
limited only to one?





Program management
Engineering responsibilities
Single vs multiple code bases
Sync between platform-specific PMs
Sync in terms of data points, definitions, terminology, event attributes,
reporting dashboards etc.
Unified Campaign Management
Adoption costs by stakeholders

Concept of Earned Differences

Youd most likely not be revisiting
this fundamental consideration.
Planning Your Product Analytics
Commandment #3

Become a Business PM at this stage - Each stakeholder will use

your implementation differently as per their needs.

Double down on
Product team
Engineering team
Business Analysts
gathering business Data Scientists
Business / Revenue team
requirements from
different functions or





Marketing team
Digital and Growth team
Operations team
Category / Merchandise team
Customer Support team
stakeholders.

Management & Executive Leadership team

Gather business requirements NOT data points to empower stakeholders.

PM needs to understand needs of each function to the determine:

Listing down their requirements Data points to be collected to fulfil requirements
should come naturally to you. Aggregation, reporting and visualization to be done thereafter
Planning Your Product Analytics
Commandment #4

What should you keep in mind?

Native Platform Support

Build vs Buy: Choose Designed to play equally well across platforms?
Support for unifying data across platforms?
the analytics tool and
infrastructure best
suited to satisfy your
Ease of Instrumentation
How easy is it to integrate in your codebase?
Is basic out-of-box reporting possible with minimal effort?
Is the implementation scalable as your platform expands?

requirements. Will the tool support your present & future scale?
Tier-pricing
Processing scale

Huge decision - once implemented, youll

stick with the tool for at least a few years!
Planning Your Product Analytics
Commandment #4 (contd.)

Does it address your data processing needs?

Events and attributes are hygiene
Does it support user-level profiles?
Build vs Buy: Choose How many default and custom dimensions?

the analytics tool and Does it unify data signals and profiles across platforms?

infrastructure best
suited to satisfy your
Reporting & Visualization Support
How easy is it to setup reporting in the tool?
What kind of reports are supported?

requirements. What technical expertise is required to operate the tool?

How sharply can you segment your reports?
What level of pivoting and aggregation is supported?
Does it support private/public reports/dashboards?
Does it support smart dashboards (eg: derived metrics?)
Does it support exporting data for offline consumption?
Planning Your Product Analytics
Commandment #4 (contd.)

Action Orientation
Can you configure marketing interventions, campaigns,
communication (eg: emails, push notifications etc.)?
Build vs Buy: Choose What behavioural and profile conditions can be used?

the analytics tool and Can the action be taken in real-time?

Are there nuances to the data on which youll take action?
infrastructure best
suited to satisfy your
Support for Ingesting External Data
Does the tool allow you to upload data sitting elsewhere?

requirements. Particularly useful if youve historic data

Also if youre enriching campaigns from outside info

User Management
Support for user management & different access rights?
You would not want your employees and vendors both to
have access to the same view of data!
Planning Your Product Analytics
Commandment #4 (contd.)
Build vs Buy: Choose
the analytics tool and
infrastructure best
suited to satisfy your
Raw Data Access
requirements.
Automation
Does the tool support putting conditional alerts?
Can it send an automatic summary of metrics & reports?
Programmability and API Support
Does it support APIs to access data & run campaigns?
Does it let you consume full raw data eg: clickstreams?
Is it provided in real-time, near real-time or deferred?
Integration and Technical Support
Will your provider handhold you during integration?
How responsive will support be? Is there team in India?
What level of support if free vs paid?
Planning Your Product Analytics
Commandment #5

Back it up!

At least 2 tools (maybe one free and another paid) so that:

Always back up your
primary analytics

You can compare and cross-check data
You can troubleshoot any discrepancies
implementation with
a secondary one!


Tools stay available to fight downtime/outages
You can compensate for missing features in each tool (for eg:
visualization capabilities vs real-time data collection
capabilities)

Never ever put all your eggs in only

a single basket.
Design, Spec & Execute the Implementation
Commandment #6

I can guarantee that reading documentation, user guides is worth

a PMs time to avoid getting caught unaware down the road.

Know the tools youve

chosen inside out. Do
not assume or leave
any doubt about how

they work.

Yes, it means reading the dev docs

and user guides as each tool
internally behaves little different.
Design, Spec & Execute the Implementation
Commandment #6 (contd.)

Its worth investing ones time to understand the following:

Definitions of Common Terms

Know the tools youve Tools always promise a very brief integration period

chosen inside out. Do

Its essential to understand (and not assume!) what the
metrics mean in the context of your tool
not assume or leave
any doubt about how
Sessions, Users, Browsers, Visitors, Bounce
Attribution Window, Lookback Window
Funnel definition

they work. Reporting, Dashboarding and Visualization Capabilities:

How many and what pivots do you need?
Which pivots and how many are supported in dashboard?
Can your engineers tactfully get more out of the tool?
Design, Spec & Execute the Implementation
Commandment #6 (contd.)

Attribution
A change in attribution model may result in dramatically
different RoI splits between traffic sources.
Know the tools youve What schema(s) of attribution is supported?

chosen inside out. Do Degree of configurability

not assume or leave

any doubt about how
Multi-Platform Behaviour
Same tool may have different definitions of common terms
and attribution across platforms

they work. Design and spec needs to account for data points and user
profiles that will be shared across each of your properties
How you implement can make or break such use cases
Design, Spec & Execute the Implementation
Commandment #7

Some pointers to approach this activity:

Spreadsheet it out
Design and spec your The tabular format lends a natural structure

requirements like a No compromise on detailed specing

Track engg. status against each scenario/data point
Yoda. Do not leave
even minute details.

Being brief or assuming here doesnt
count, but being crystal clear does.
Design, Spec & Execute the Implementation
Commandment #7 (contd.)
Design and spec your
requirements like a
Yoda. Do not leave
even minute details.
Single living document across platforms
Something I picked up from my stint at Microsoft
Living spec also a handy source of future reference
Scalability for instrumentation
Extensive instrumentation of your codebase is needed
Does it make sense to introduce a standard middleware?
Decide basis the following:
Scalability - instrument once, use everywhere
Ease of Troubleshooting
Ease of Correction correct at only a single place
Extensibility add new logic quickly & at low risk
Design, Spec & Execute the Implementation
Commandment #7 (contd.)

Translate all known use-cases into data points

Which events will you trigger?
How many & which dimensions to instrument for?
Design and spec your What information is required for every individual event?

requirements like a Which events will be interaction vs non-interaction?

What will be the micro-funnel steps be?
Yoda. Do not leave
even minute details.

Which data points will you capture per user profile?
In which context should information be captured in?

Decide how you'd ingest external data

Which data points sitting in your own systems do you wish
to ingest into the analytics tool youve chosen?
How do you upload these data points to your tool and in
which format?
How frequently will you need to do this?
Do you need tools API integration with internal systems?
Design, Spec & Execute the Implementation
Commandment #8

Your stakeholders are your internal customers

Essential to share your design - even boring, technical nuances

Do not start your Call out limitations (if any) of the tool

implementation

Capabilities of reporting dashboards
Get stakeholders to sign-off before engineering picks up your
without reviewing it
with stakeholders and
spec for implementation

getting sign-offs.

PMs need to behave as business manager

and change managers in their organization.
Design, Spec & Execute the Implementation
Commandment #9

Become a Technical PM at this stage

Doesnt matter how much overview and briefing you provide

Engineering Your developers will almost always have lesser context than you

implementation of
your analytics spec is
high-availability time

for you as a PM.
Typical queries you may expect:
Source of data to be sent to the analytics tool
Always be around for your developers to
Exact trigger of data event
clarify ambiguity.
Format required for instrumenting a data-point
Finalizing and Deploying Your Implementation
Commandment #10

The undisputed onus lies on the PM. The buck stops at you!

While QA engineers are generally more aware of user scenarios

Testing & validating than developers, they will still not have as much context as a PM.

your analytics
implementation is the
PM's job too!

Is the analytics implementation ready to
power reports, visualizations & campaigns?
Finalizing and Deploying Your Implementation
Commandment #11

Dependency alignment with stakeholders is an often neglected

but a critical step. Its time to ensure data inputs are clean!

Be obsessed about
not polluting your
brand new analytics
implementation.

Align stakeholders to ensure correct data is
passed on to your analytics infrastructure.
Finalizing and Deploying Your Implementation
Commandment #12

Empower all
consumers of
analytics data and
reports through
thorough training.

A new implementation will never be 100% at

par with what your stakeholders were used to.
Handhold first and then let self-serve follow!
Finalizing and Deploying Your Implementation
Commandment #13

A successful rollout represents the beginning of a culture change.

Its unlikely you will get it 100% right in the first go

Remember, enriching
and fine-tuning Improving your analytics is a continuous process.

analytics is a
continuous process.
With time and new features getting added to your product, with
new business requirements and use cases, youll discover:

Plenty of missing scenarios over time (eg: filters)

Edge cases not taken care of
Data corruptions you will find only at scale
Logic errors
Reporting/visualization needs that will demand design
changes in the way youve implemented the analytics
New requirements
Time to celebrate your first implementation
Heres your brownie!
Want to read more about the 13 commandments?
Heres an elaborate version of what we covered in 2 hours!

Part 1
5 Things I Learnt While Planning Out My Product
Analytics Implementation

Part 2
Rolling Up Your Sleeves to Design, Spec &
Execute Your First Product Analytics

Part 3
This is how I made my 1st product analytics
implementation truly effective
Thank You! Now get started. Vishrut Shukla
in.linkedin.com/in/vishrutshukla
Feedback, comments and suggestions welcome.