Preprocessing module

The chemml.preprocessing module includes (please click on links adjacent to function names for more information):
chemml.preprocessing.ConstantColumns(df)

Removes single-value columns. NOTE: Deprecated in v1.3.4, now backend uses RemoveInvFeatures. Will be removed in v1.4.

Parameters

df: pandas dataframe

input dataframe

Returns

df: pandas dataframe

chemml.preprocessing.GAFSel(df=None, target=None, target_features_count=None, evaluator=None, n_generations=50, pop_size=100, test_size=0.7, crossover_ratio=0.6, random_state=42)

Genetic Algorithm-based Feature Selection wrapper for standardized feature selection workflows.

Selects the most modeling-useful features from a dataset using a genetic algorithm that directly optimizes for a given estimator’s predictive performance (Mean Absolute Error).

Parameters

dfpandas.DataFrame

Input dataframe containing both feature columns and the target column. A ‘SMILES’ column, if present, is automatically excluded from the feature set.

targetstr

The name of the column containing the target variable to predict.

target_features_countint, optional (default=50)

Target number of features to select. The genetic algorithm will bias selection toward solutions with approximately this many active features. Default behaviour is to select around half the number of input features.

evaluatorsklearn-style estimator, optional (default=None)

A scikit-learn compatible regression estimator with fit() and predict() methods. If None, defaults to LinearRegression().

n_generationsint, optional (default=50)

Number of generations to evolve the GA population for.

pop_sizeint, optional (default=100)

Size of the GA population in each generation.

crossover_ratiofloat, optional (default=0.6)

Fraction of the population to use for crossover in each generation. Must be in (0, 1).

test_sizefloat, optional (default=0.7)

Fraction of data to use for testing in each fitness evaluation (train/test split). Must be in the interval (0, 1).

random_stateint, optional (default=42)

Random seed for reproducibility in train/test splits.

Returns

pandas.DataFrame

Dataframe containing only the selected features plus the target column. The number of selected features will typically be close to target_features_count, depending on GA convergence and the data.

Raises

ValueError

If df or target is None, target is not in df.columns, target_features_count is not a positive integer, or test_size is not in (0, 1).

Notes

The genetic algorithm uses: - Crossover type: Uniform - Algorithm variant: 3 (best members from init/crossover/mutation pools) - Fitness: minimization (MAE) - Crossover size: 60% of population - Mutation size: 40% of population

Each individual in the GA population is a binary string where each bit represents whether a feature is included (1) or excluded (0). The objective function trains the estimator on a subset of features and returns the MAE on held-out test data.

Note that due to its inherently random nature, GA won’t hit the exact target_features_count, but will start with a bias toward that number of features. The final selection may vary slightly depending on the data and random seed.

Examples

>>> import pandas as pd
>>> from chemml.preprocessing import GAFSel
>>> df = pd.DataFrame({'feature1': [1, 2, 3], 'feature2': [4, 5, 6], 
...                     'target': [10, 20, 30]})
>>> selected = GAFSel(df=df, target='target', target_features_count=1, 
...                    n_generations=5, pop_size=20)
>>> print(selected.shape)  # (3, 2) — target + 1 selected feature
chemml.preprocessing.MissingValues(df, strategy='ignore_row', string_as_null=True, inf_as_null=True, missing_values=None)

find missing values and interpolate/replace or remove them.

Parameters

df : pandas dataframe

strategy: string, optional (default=”ignore_row”)

list of strategies: - interpolate: interpolate based on sorted target values - zero: set to the zero - ignore_row: remove the entire row in data and target - ignore_column: remove the entire column in data and target

string_as_null: boolean, optional (default=True)

If True non numeric elements are considered to be null in computations.

missing_values: list, optional (default=None)

where you define specific formats of missing values. It is a list of string, float or integer values.

inf_as_null: boolean, optional (default=True)

If True inf and -inf elements are considered to be null in computations.

Returns

dataframe

Notes

mask is a binary vector whose length is the number of rows/indices in the df. The index of each bit shows if the row/column in the same position has been removed or not. The goal is keeping track of removed rows/columns to change the target data frame or other input data frames based on that. The mask can later be used in the transform method to change other data frames in the same way.

chemml.preprocessing.Outliers(df, m=2.0, strategy='median')

remove all rows where the values of a certain column are within an specified standard deviation from mean/median.

Parameters

df: pandas dataframe

input dataframe

m: float, optional (default=3.0)

the outlier threshold with respect to the standard deviation

strategy: string, optional (default=’median’)

available options: ‘mean’ and ‘median’ Values of each column will be compared to the ‘mean’ or ‘median’ of that column.

Returns

dataframe

Notes

We highly recommend you to remove constant columns first and then remove outliers.

chemml.preprocessing.RemoveCorrFeatures(df, correlation_threshold=0.9)

remove highly correlated feature columns.

Parameters

df: pandas dataframe

input dataframe

correlation_threshold: float, optional (default=0.9)

absolute correlation threshold. If any pair of columns has absolute correlation larger than this threshold, the later column (based on column order) is removed.

Returns

df: pandas dataframe

dataframe with highly correlated columns removed

chemml.preprocessing.RemoveInvFeatures(df, sanitize_threshold=0.9, filter_single_value=True, sanitize_binary=True, sanitize_nonbinary=True, use_variance_filtering=True, variance_threshold=0.01, keep_filtered_columns=False)

remove invariant or near-invariant feature columns.

Parameters

df: pandas dataframe

input dataframe

sanitize_threshold: float, optional (default=0.9)

dominant-frequency threshold used for invariant filtering. Columns with dominant value frequency greater than or equal to this fraction of rows are removed.

sanitize_nonbinary: boolean, optional (default=True)

if True, remove nonbinary columns (nunique > 2) where one value is dominant in more than 90 percent of rows.

use_variance_filtering: boolean, optional (default=True)

if True, remove numeric columns with variance less than variance_threshold.

variance_threshold: float, optional (default=0.01)

minimum variance required to keep a numeric column.

keep_filtered_columns: boolean, optional (default=False)

if True, returns a tuple (clean_df, removed_df) where removed_df contains all removed columns from the original input dataframe.

Returns

df: pandas dataframe

cleaned dataframe when keep_filtered_columns=False

tuple

(clean_df, removed_df) when keep_filtered_columns=True

chemml.preprocessing.ZScoreFSel(features: DataFrame, target: Series, threshold=2.0)

Select binary features based on Z-score of the target variable.

Parameters

featurespandas.DataFrame

Feature matrix.

targetpandas.Series

Target variable.

thresholdfloat, optional

Z-score threshold for feature selection. Default is 2.0.

Returns

pandas.DataFrame

DataFrame containing selected features and the target column.

chemml.preprocessing.remove_complex_columns(df)

Remove columns with complex data types from the dataframe.

Parameters

df: pandas dataframe

input dataframe

Returns

df: pandas dataframe

dataframe with complex columns removed