Preprocessing module

The chemml.preprocessing module includes (please click on links adjacent to function names for more information):
chemml.preprocessing.ConstantColumns(df)

Removes single-value columns. NOTE: Deprecated in v1.3.4, now backend uses RemoveInvFeatures. Will be removed in v1.4.

Parameters

df: pandas dataframe

input dataframe

Returns

df: pandas dataframe

chemml.preprocessing.MissingValues(df, strategy='ignore_row', string_as_null=True, inf_as_null=True, missing_values=None)

find missing values and interpolate/replace or remove them.

Parameters

df : pandas dataframe

strategy: string, optional (default=”ignore_row”)

list of strategies: - interpolate: interpolate based on sorted target values - zero: set to the zero - ignore_row: remove the entire row in data and target - ignore_column: remove the entire column in data and target

string_as_null: boolean, optional (default=True)

If True non numeric elements are considered to be null in computations.

missing_values: list, optional (default=None)

where you define specific formats of missing values. It is a list of string, float or integer values.

inf_as_null: boolean, optional (default=True)

If True inf and -inf elements are considered to be null in computations.

Returns

dataframe

Notes

mask is a binary vector whose length is the number of rows/indices in the df. The index of each bit shows if the row/column in the same position has been removed or not. The goal is keeping track of removed rows/columns to change the target data frame or other input data frames based on that. The mask can later be used in the transform method to change other data frames in the same way.

chemml.preprocessing.Outliers(df, m=2.0, strategy='median')

remove all rows where the values of a certain column are within an specified standard deviation from mean/median.

Parameters

df: pandas dataframe

input dataframe

m: float, optional (default=3.0)

the outlier threshold with respect to the standard deviation

strategy: string, optional (default=’median’)

available options: ‘mean’ and ‘median’ Values of each column will be compared to the ‘mean’ or ‘median’ of that column.

Returns

dataframe

Notes

We highly recommend you to remove constant columns first and then remove outliers.

chemml.preprocessing.RemoveCorrFeatures(df, correlation_threshold=0.9)

remove highly correlated feature columns.

Parameters

df: pandas dataframe

input dataframe

correlation_threshold: float, optional (default=0.9)

absolute correlation threshold. If any pair of columns has absolute correlation larger than this threshold, the later column (based on column order) is removed.

Returns

df: pandas dataframe

dataframe with highly correlated columns removed

chemml.preprocessing.RemoveInvFeatures(df, sanitize_threshold=0.9, filter_single_value=True, sanitize_binary=True, sanitize_nonbinary=True, use_variance_filtering=True, variance_threshold=0.01, keep_filtered_columns=False)

remove invariant or near-invariant feature columns.

Parameters

df: pandas dataframe

input dataframe

sanitize_threshold: float, optional (default=0.9)

dominant-frequency threshold used for invariant filtering. Columns with dominant value frequency greater than or equal to this fraction of rows are removed.

sanitize_nonbinary: boolean, optional (default=True)

if True, remove nonbinary columns (nunique > 2) where one value is dominant in more than 90 percent of rows.

use_variance_filtering: boolean, optional (default=True)

if True, remove numeric columns with variance less than variance_threshold.

variance_threshold: float, optional (default=0.01)

minimum variance required to keep a numeric column.

keep_filtered_columns: boolean, optional (default=False)

if True, returns a tuple (clean_df, removed_df) where removed_df contains all removed columns from the original input dataframe.

Returns

df: pandas dataframe

cleaned dataframe when keep_filtered_columns=False

tuple

(clean_df, removed_df) when keep_filtered_columns=True