Feature Selection using chemml.optimization.GeneticAlgorithm
We use a sample dataset from ChemML library which has the SMILES codes and Dragon molecular descriptors for 500 small organic molecules with their densities in \(kg/m^3\). For simplicity, we perform feature selection using Genetic Algorithm on a subset of 50 molecular descriptors.
For more information on Genetic Algorithm, please refer to our paper
Note: Due to the inherent randomness of the genetic algorithm, it is recommended to perform multiple runs and look for features that persistently appear.
[1]:
from chemml.datasets import load_organic_density
_,density,features = load_organic_density()
num_cols = 50
col_names = features.iloc[:, :num_cols].columns
features = features.iloc[:,:num_cols]
print(density.shape, features.shape)
density, features = density.values, features.values
(500, 1) (500, 50)
Defining hyperparameter space
For this, each individual feature is encoded as a binary bit of the chromosome.
0 indicates feature is discarded.
1 indicates feature is selected.
[2]:
space = tuple([{i: {'choice': [0,1]}} for i in range(features.shape[1])])
Defining objective function
The objective function is defined as a function that receives one ‘individual’ of the genetic algorithm’s population that is an ordered list of the hyperparameters defined in the space variable. Within the objective function, the user does all the required calculations and returns the metric (as a tuple) that is supposed to be optimized. If multiple metrics are returned, all the metrics are optimized according to the fitness defined in the initialization of the Genetic Algorithm class.
Here, we use a simple linear regression model to fit the data.
[3]:
from sklearn.metrics import mean_absolute_error
import pandas as pd
from sklearn.linear_model import LinearRegression
def obj(individual, features=features):
df = pd.DataFrame(features)
new_cols = list(map(bool, individual))
df = df[df.columns[new_cols]]
features = df.values
ridge = LinearRegression(n_jobs=1)
ridge.fit(features[:400], density[:400])
pred = ridge.predict(features[400:])
return mean_absolute_error(density[400:], pred)
Optimize the feature space
[4]:
from chemml.optimization import GeneticAlgorithm
ga = GeneticAlgorithm(evaluate=obj, space=space, fitness=("min", ), crossover_type="Uniform",
pop_size = 10, crossover_size=6, mutation_size=4, algorithm=3)
fitness_df, final_best_features = ga.search(n_generations=20)
The search converged with convergence criteria = 10
ga.search returns:
a dataframe with the best individuals of each generation along with their fitness values and the time taken to evaluate the model
a dictionary containing the best individual (in this case the top features)
[5]:
fitness_df
[5]:
| Best_individual | Fitness_values | Time (hours) | |
|---|---|---|---|
| 0 | (1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.294035 | 0.000011 |
| 1 | (1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.294035 | 0.000010 |
| 2 | (0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, ... | 10.283015 | 0.000011 |
| 3 | (0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, ... | 10.283015 | 0.000011 |
| 4 | (0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, ... | 10.283015 | 0.000011 |
| 5 | (0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, ... | 10.283015 | 0.000011 |
| 6 | (0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, ... | 10.283015 | 0.000011 |
| 7 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000011 |
| 8 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000010 |
| 9 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000011 |
| 10 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000011 |
| 11 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000012 |
| 12 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000011 |
| 13 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000012 |
| 14 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000010 |
| 15 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000014 |
| 16 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000011 |
| 17 | (1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, ... | 10.134219 | 0.000011 |
[6]:
print(final_best_features)
{0: 1, 1: 1, 2: 0, 3: 0, 4: 0, 5: 0, 6: 1, 7: 0, 8: 0, 9: 0, 10: 1, 11: 0, 12: 0, 13: 1, 14: 1, 15: 0, 16: 0, 17: 0, 18: 0, 19: 1, 20: 1, 21: 1, 22: 1, 23: 0, 24: 1, 25: 0, 26: 1, 27: 1, 28: 1, 29: 1, 30: 1, 31: 1, 32: 0, 33: 1, 34: 1, 35: 1, 36: 1, 37: 0, 38: 0, 39: 0, 40: 0, 41: 0, 42: 1, 43: 1, 44: 1, 45: 0, 46: 0, 47: 0, 48: 1, 49: 0}
[7]:
# Printing out which columns were selected
df = pd.DataFrame(features, columns=col_names)
new_cols = list(map(bool, tuple([v for k,v in final_best_features.items()])))
print(df[df.columns[new_cols]])
MW AMW Mv GD nTA nBT RBF nAB nH nC ... \
0 285.54 7.932 0.620 0.140 0.0 38.0 0.079 5.0 19.0 13.0 ...
1 240.24 9.240 0.714 0.131 1.0 28.0 0.071 17.0 8.0 12.0 ...
2 313.42 8.471 0.666 0.108 0.0 40.0 0.075 16.0 15.0 15.0 ...
3 218.32 9.492 0.659 0.179 2.0 24.0 0.042 5.0 10.0 8.0 ...
4 319.45 9.396 0.685 0.114 0.0 37.0 0.081 15.0 13.0 13.0 ...
.. ... ... ... ... ... ... ... ... ... ... ...
495 296.36 7.799 0.649 0.108 0.0 41.0 0.073 16.0 16.0 16.0 ...
496 328.50 8.645 0.674 0.108 1.0 41.0 0.049 16.0 16.0 16.0 ...
497 373.52 8.120 0.674 0.088 0.0 50.0 0.060 22.0 19.0 21.0 ...
498 323.50 8.513 0.636 0.114 0.0 41.0 0.073 10.0 17.0 12.0 ...
499 348.42 8.103 0.708 0.088 0.0 47.0 0.064 28.0 16.0 24.0 ...
N% O% nCsp2 nCIC nCIR TRS NRS NNRS nR05 ARR
0 2.8 0.0 3.0 3.0 3.0 16.0 3.0 1.00 2.0 0.263
1 15.4 7.7 12.0 3.0 3.0 17.0 3.0 1.00 1.0 0.850
2 13.5 2.7 12.0 4.0 4.0 22.0 4.0 1.00 2.0 0.640
3 0.0 13.0 4.0 2.0 2.0 11.0 2.0 1.00 1.0 0.357
4 14.7 2.9 10.0 4.0 4.0 21.0 4.0 1.00 3.0 0.625
.. ... ... ... ... ... ... ... ... ... ...
495 10.5 5.3 13.0 4.0 4.0 22.0 4.0 1.00 2.0 0.640
496 10.5 0.0 13.0 4.0 5.0 23.0 3.0 0.75 1.0 0.640
497 10.9 0.0 18.0 5.0 6.0 29.0 4.0 0.80 1.0 0.710
498 18.4 0.0 4.0 4.0 4.0 21.0 4.0 1.00 3.0 0.417
499 4.7 2.3 24.0 5.0 6.0 29.0 4.0 0.80 1.0 0.903
[500 rows x 25 columns]
New in v1.3.4: Tunable target feature lengths
You can now specify a target number of features (or fraction of starting features) to bias the GA. The optimization will start from the target number of features on average, and only add more features if it is optimal.
Previous implementations would always filter out about half of input features and not go much lower than that. This is much more useful when trying to filter out a core subset of features (e.g top 25 out of 200) and makes further model interpretation easier.
[8]:
ga_tuned = GeneticAlgorithm(evaluate=obj, space=space, fitness=("min", ), crossover_type="Uniform",
pop_size = 10, crossover_size=6, mutation_size=4, algorithm=3,
target_features_count=10)
fitness_df_tuned, final_best_features_tuned = ga_tuned.search(n_generations=20)
[9]:
# Printing out which columns were selected
df = pd.DataFrame(features, columns=col_names)
new_cols_tuned = list(map(bool, tuple([v for k,v in final_best_features_tuned.items()])))
print(df[df.columns[new_cols_tuned]])
MW AMW GD nBT nBM RBN RBF nAB nH nC nO \
0 285.54 7.932 0.140 38.0 5.0 3.0 0.079 5.0 19.0 13.0 0.0
1 240.24 9.240 0.131 28.0 17.0 2.0 0.071 17.0 8.0 12.0 2.0
2 313.42 8.471 0.108 40.0 16.0 3.0 0.075 16.0 15.0 15.0 1.0
3 218.32 9.492 0.179 24.0 5.0 1.0 0.042 5.0 10.0 8.0 3.0
4 319.45 9.396 0.114 37.0 15.0 3.0 0.081 15.0 13.0 13.0 1.0
.. ... ... ... ... ... ... ... ... ... ... ...
495 296.36 7.799 0.108 41.0 16.0 3.0 0.073 16.0 16.0 16.0 2.0
496 328.50 8.645 0.108 41.0 16.0 2.0 0.049 16.0 16.0 16.0 0.0
497 373.52 8.120 0.088 50.0 22.0 3.0 0.060 22.0 19.0 21.0 0.0
498 323.50 8.513 0.114 41.0 10.0 3.0 0.073 10.0 17.0 12.0 0.0
499 348.42 8.103 0.088 47.0 28.0 3.0 0.064 28.0 16.0 24.0 1.0
H% N% O% nCsp2 nCIR RFD NNRS ARR
0 52.8 2.8 0.0 3.0 3.0 0.000 1.00 0.263
1 30.8 15.4 7.7 12.0 3.0 0.000 1.00 0.850
2 40.5 13.5 2.7 12.0 4.0 0.000 1.00 0.640
3 43.5 0.0 13.0 4.0 2.0 0.000 1.00 0.357
4 38.2 14.7 2.9 10.0 4.0 0.000 1.00 0.625
.. ... ... ... ... ... ... ... ...
495 42.1 10.5 5.3 13.0 4.0 0.000 1.00 0.640
496 42.1 10.5 0.0 13.0 5.0 0.095 0.75 0.640
497 41.3 10.9 0.0 18.0 6.0 0.074 0.80 0.710
498 44.7 18.4 0.0 4.0 4.0 0.000 1.00 0.417
499 37.2 4.7 2.3 24.0 6.0 0.074 0.80 0.903
[500 rows x 19 columns]