Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
pandas
Methods to read data are all named read_* to_*
pd.read_* where * is the le type. Series
and DataFrames can be saved to disk using
their to_* method.
DataFrame
Usage Patterns h5 X Y Z h5
a
Use pd.read_clipboard() for one-o data b
extractions. c
+ +
Reading Text Files into a DataFrame
Colors highlight how dierent arguments map from the data le to a DataFrame.
# Historical_data.csv
Date Cs Rd
Date, Cs, Rd >>> read_table(
2005-01-03, 64.78, - 'historical_data.csv',
sep=',',
2005-01-04, 63.79, 201.4
header=1,
2005-01-05, 64.46, 193.45
skiprows=1,
... skipfooter=2,
Data from Lab Z. index_col=0,
Recorded by Agent E parse_dates=True,
na_values=['-'])
, ,
X Y X Y X Y
a a a
>>> df_list = read_html(url) b b b
c c c
Take y our Pand a s skills to the next le ve l! Reg ister at w w w .enth ou g h t.com/pan d as-master-cl ass
2 0 1 6 En t h o u gh t , I n c. , lice n s e d u n de r t h e Cre a t iv e Co mmo n s At t ribu t io n -No n Co mme rcial-NoDeriv at iv e s 4.0 I n t e rn at io n al Lice n s e.
T o v ie w a co py o f t h is lice ns e , v is it h t t p:/ / cre at iv e co mmo n s .o rg/ lice n s e s / by -n c-n d/ 4.0/
Split / Apply / Combine with DataFrames
pandas
1. Split the data based on some criteria.
2. Apply a function to each group to aggregate, transform, or
lter. Split/Apply/Combine
3. Combine the results.
The apply and combine steps are typically done together in X Y
Pandas. a 1 1.5
X Y a 2
a 1 X Y
Split: Group By b
c
3
2
X Y
b 3 2
a
b
1.5
2
Group by a single column: b 1 b 1 c 2
> g = df.groupby(col_name) c 2
a 2 X Y
Grouping with list of column names creates DataFrame with MultiIndex. c 2 2
(see Reshaping DataFrames and Pivot Tables cheatsheet): c 2
> g = df.groupby(list_col_names)
Pass a function to group based on the index:
Split Apply Combine
> g = df.groupby(function)
Groupby Apply
Window Functions Group-specic transformations
X Y Z
0 a Aggregation
X Y Z 2 a Group-specic Filtering
0 a
df.groupby('X')
1 b X Y Z
2
3
a
b
1 b
3 b
Split: Whats a GroupBy Object?
4 c
X Y Z
It keeps track of which rows are part of which group.
4 c
> g.groups Dictionary, where keys are group
names, and values are indices of rows in a given group.
Apply/Combine: General Tool: apply It is iterable:
> for group, sub_df in g:
More general than agg, transform, and filter. Can
...
aggregate, transform or lter. The resulting dimensions
can change, for example:
> g.apply(lambda x: x.describe())
Apply/Combine: Aggregation
Perform computations on each group. The shape changes;
Apply/Combine: Transformation the categories in the grouping columns become the index.
Can use built-in aggregation methods: mean, sum, size,
The shape and the index do not change.
count, std, var, sem, describe, first, last, nth,
> g.transform(df_to_df)
min, max, for example:
Example, normalization:
> g.mean()
> def normalize(grp):
or aggregate using custom function:
. return (grp - grp.mean()) / grp.var()
> g.agg(series_to_value)
> g.transform(normalize)
or aggregate with multiple functions at once:
X Y Z
Other Groupby-Like Operations: Window Functions
0 a 1 1
X Y Z resample, rolling, and ewm (exponential weighted
2 a 1 1
0 a 1 1 0
X Y Z
g.filter() function) methods behave like GroupBy objects. They keep
1 b 1 1 1
1 b 1 1 track of which row is in which group. Results must be
2 a 1 1 2
3 b 1 1 aggregated with sum, mean, count, etc. (see Aggregation).
3 b 1 1 resample is often used before rolling, expanding, and 3
X Y Z
4 c 0 0 ewm when using a DateTime index. 4
Take y our Pand a s skills to the next le ve l! Reg ister at w w w .enth ou g h t.com/pan d as-master-cl ass
2 0 1 6 E n t h o u gh t , I n c. , lice n s e d u n de r t h e Cre a t iv e Co mmo n s At t ribu t io n -No n Co mme rcial-No De riv at iv e s 4.0 I n t e rn at io n al Lice n s e.
T o v ie w a copy o f t h is lice n s e , v is it h t t p:/ / cre at iv e co mmo n s .o rg/ lice n s e s / by -n c-n d/ 4.0/
Manipulating Dates and Times
pandas
Use a Datetime index for easy time-based indexing and slicing,
as well as for powerful resampling and data alignment.
Timestamps vs Periods
Pandas makes a distinction between timestamps, called
Timestamps
Datetime objects, and time spans, called Period objects.
> pd.date_range(start=None, end=None,
>>> pd.to_datetime('12/01/2000') # 1st December
periods=None, freq=offset,
Timestamp('2000-12-01 00:00:00')
tz='Europe/London')
>>> pd.to_datetime('13/01/2000') # 13th January!
Specify either a start or end date, or both. Set number of
Timestamp('2000-01-13 00:00:00')
"steps" with periods. Set "step size" with freq; see "Frequen-
>>> pd.to_datetime('2000-01-13') # 13th January
cy osets" for acceptable values. Specify time zones with tz.
Timestamp('2000-01-13 00:00:00')
Frequency Offsets
Used by date_range, period_range and resample:
Creating Ranges or Periods
B: Business day A: Year end > pd.period_range(start=None, end=None,
D: Calendar day AS: Year start periods=None, freq=offset)
W: Weekly H: Hourly
M: Month end T, min: Minutely
MS: Month start S: Secondly
Resampling
BM: Business month end L, ms: Milliseconds
> s_df.resample(freq_offset).mean()
Q: Quarter end U, us: Microseconds
For more: N: Nanoseconds resample returns a groupby-like object that must be
Lookup "Pandas Oset Aliases" or check out pandas.tseries.offsets, aggregated with mean, sum, std, apply, etc. (See also the
and pandas.tseries.holiday modules. Split-Apply-Combine cheat sheet.)
Take y our Pand a s skills to the next le ve l! Reg ister at w w w .enth ou g h t.com/pan d as-master-cl ass
2 0 1 6 En t h o u gh t , I n c. , lice n s e d u n de r t h e Cre a t iv e Co mmo n s At t ribu t io n -No n Co mme rcial-NoDeriv at iv e s 4.0 I n t e rn at io n al Lice n s e.
T o v ie w a co py o f t h is lice ns e , v is it h t t p:/ / cre at iv e co mmo n s .o rg/ lice n s e s / by -n c-n d/ 4.0/
Pandas Data Structures: Series and DataFrames
pandas
A Series, s, maps an index to values. It is:
Like an ordered dictionary
A Numpy array with row labels and a name
A DataFrame, df, maps index and column labels to values. It is:
Indexing and Slicing
Like a dictionary of Series (columns) sharing the same index
A 2D Numpy array with row and column labels Use these attributes on Series and DataFrames for indexing,
s_df applies to both Series and DataFrames. slicing, and assignments:
Assume that manipulations of Pandas object return copies.
s_df.loc[] Refers only to the index labels
s_df.iloc[] Refers only to the integer location,
similar to lists or Numpy arrays
Creating Series and DataFrames
s_df.xs(key, level) Select rows with label key in level
Series Series level of an object with MultiIndex.
s.dtype, df.dtypes Type of Series, of each column
len(s_df) Number of rows > df[mask]['X'] = 1
SettingWithCopyWarning
s_df.head() and s_df.tail() First/last rows
s.unique() Series of unique values > df.loc[mask , 'X'] = 1
s_df.describe() Summary stats
df.info() Memory usage
Take y our Pand a s skills to the next le ve l! Reg ister at w w w .enth ou g h t.com/pan d as-master-cl ass
2 0 1 6 En t h o u gh t , I n c. , lice n s e d u n de r t h e Cre a t iv e Co mmo n s At t ribu t io n -No n Co mme rcial-NoDeriv at iv e s 4.0 I n t e rn at io n al Lice n s e.
T o v ie w a co py o f t h is lice ns e , v is it h t t p:/ / cre at iv e co mmo n s .o rg/ lice n s e s / by -n c-n d/ 4.0/
Combining DataFrames
pandas
Tools for combining Series and DataFrames
together, with SQL-type joins and concatenation. Concatenating DataFrames
Use join if merging on indices, otherwise use
> pd.concat(df_list)
merge. Stacks DataFrames on top of each other.
Set ignore_index=True, to replace index with RangeIndex.
Note: Faster than repeated df.append(other_df).
Merge on Column Values
> pd.merge(left, right, how='inner', on='id')
Ignores index, unless on=None. See value of how below.
Join on Index
Use on if merging on same column in both DataFrames, otherwise
> df.join(other)
use left_on, right_on.
Merge DataFrames on index. Set on=keys to join on index of df and
on keys of other. Join uses pd.merge under the covers.
Merge Types: The how Keyword
Take y our Pand a s skills to the next le ve l! Reg ister at w w w .enth ou g h t.com/pan d as-master-cl ass
2 0 1 6 En t h o u gh t , I n c. , lice n s e d u n de r t h e Cre a t iv e Co mmo n s At t ribu t io n -No n Co mme rcial-NoDeriv at iv e s 4.0 I n t e rn at io n al Lice n s e.
T o v ie w a co py o f t h is lice ns e , v is it h t t p:/ / cre at iv e co mmo n s .o rg/ lice n s e s / by -n c-n d/ 4.0/
Reshaping Dataframes and Pivot Tables
pandas
Tools for reshaping DataFrames from the wide to the long format and back.
The long format can be tidy, which means that "each variable is a column,
each observation is a row"1. Tidy data is easier to filter, aggregate,
transform, sort, and pivot. Reshaping operations often produce multi-level Long to Wide Format and Back
indices or columns, which can be sliced and indexed. with stack() and unstack()
1 Hadley Wickham (2014) "Tidy Data", http://dx.doi.org/10.18637/jss.v059.i10
Omitting index, columns, or values will use all remaining columns of df.
You can "pivot" a table manually using groupby, stack and unstack. pd.melt(team, id_vars=['Color'],
value_vars=['A', 'B', 'C'],
Index var_name='Team', value_name='Score')
Columns
Number of Continent Continent
0 Recently updated stations code AN EU
code Color Team Score
1 FALSE 1 EU Recently
updated
Team 0 Red A 1
2 FALSE 1 EU Color A B C
FALSE 1 3 Melt 1 Blue A 2
3 FALSE 1 EU
0 Red 1 3 4 2 Red B 3
TRUE 2 1
1 Blue 2 - 6 3 Blue B -
4 TRUE 1 EU
pd.pivot_table(df, 4 Red C 4
5 FALSE 1 AN index="Recently updated",
5 Blue C 5
columns="continent code",
6 TRUE 1 AN
values="Number of Stations",
7 TRUE 1 AN
aggfunc=np.sum)
df.pivot() vs pd.pivot_table
Red Panda
df.pivot() Does not deal with repeated values in Ailurus fulgens
index. It's a declarative form of stack
and unstack.
pd.pivot_table() Use if you have repeated values in index
(specify aggfunc argument).
Take y our P and a s skills to the ne xt le ve l! Reg ister at w w w .enthou g h t.com/p an d as-master-class
2 0 1 6 E n t h o u gh t , I n c. , lice n s e d u n de r t h e Cre a t iv e Co mmo n s At t ribu t io n -No n Co mme rcial-No Deriv at iv e s 4.0 I n t e rn at io n al Lice n s e.
T o v ie w a co py o f t h is lice n s e , v is it h t t p:/ / cre at iv e co mmo n s .o rg/ lice n s e s / by -n c-n d/ 4.0/