11.5 Machine learning in drug discovery

Machine learning is reshaping drug discovery by speeding up the identification of new drug candidates and optimizing existing compounds. By working with large chemical and biological datasets, ML algorithms can streamline stages from virtual screening to lead optimization and ADMET prediction, cutting both cost and development time.

This topic covers ML fundamentals, key algorithms used in medicinal chemistry, data preprocessing techniques, model evaluation, and how ML integrates with other computational and experimental approaches.

Machine learning fundamentals

Machine learning (ML) is a subset of artificial intelligence where computers learn patterns from data rather than following explicitly programmed rules. ML algorithms build mathematical models from sample data (called training data) and use those models to make predictions on new, unseen data.

In drug discovery, this means you can train a model on thousands of known active and inactive compounds, and it will learn to predict whether a new molecule is likely to be active against your target.

Artificial intelligence vs machine learning

These terms get used interchangeably, but they're distinct:

• Artificial intelligence (AI) is the broad field of building machines that perform tasks requiring human-like intelligence. This includes rule-based expert systems, logic programming, and more.
• Machine learning is a subset of AI that specifically uses statistical techniques to learn from data. Instead of hard-coding rules, you let the algorithm discover patterns on its own.

The key distinction: a rule-based system follows instructions a human wrote ("if molecular weight > 500, flag it"). An ML system learns its own rules from examples in the data.

Supervised vs unsupervised learning

Supervised learning trains on labeled data, meaning each data point has a known input-output pair. The algorithm learns to map inputs to correct outputs.

• Classification example: predicting whether a compound is active or inactive against a target
• Regression example: predicting a compound's binding affinity (a continuous value)

Unsupervised learning works with unlabeled data and tries to discover hidden structure without predefined answers.

• Clustering: grouping similar compounds together based on structural features
• Dimensionality reduction: compressing high-dimensional molecular descriptors into fewer variables for visualization or downstream modeling

Deep learning and neural networks

Deep learning uses artificial neural networks with multiple layers to learn hierarchical representations of data. Each successive layer captures increasingly abstract features.

• Neural networks consist of interconnected nodes (neurons) organized in layers: input, hidden, and output
• Convolutional Neural Networks (CNNs) excel at grid-like data such as molecular images or 2D/3D chemical structures
• Recurrent Neural Networks (RNNs) handle sequential data, making them useful for SMILES string-based molecular generation

Deep learning has driven major advances in image recognition, natural language processing, and, increasingly, molecular property prediction and de novo drug design.

Reinforcement learning principles

Reinforcement learning (RL) involves an agent that learns by interacting with an environment and receiving feedback in the form of rewards or penalties.

The key components:

• Agent: the decision-maker (e.g., a molecule-generating algorithm)
• Environment: the context the agent operates in (e.g., chemical space with property constraints)
• State: the current situation (e.g., a partially built molecule)
• Action: a choice the agent makes (e.g., adding a functional group)
• Reward: feedback on how good the action was (e.g., improved predicted binding affinity)

The agent's goal is to learn a policy that maximizes cumulative reward over time. In drug discovery, RL has been applied to de novo molecular design, where the agent iteratively builds molecules optimized for desired properties.

Machine learning applications in drug discovery

ML techniques can be applied across the drug discovery pipeline. The sections below cover the most impactful applications.

Virtual screening of drug candidates

Virtual screening computationally identifies potential drug candidates from large compound libraries, narrowing down millions of molecules to a manageable set for experimental testing.

ML algorithms trained on known active and inactive compounds can predict the bioactivity of new molecules. Two main approaches exist:

• Ligand-based approaches: use information about known active compounds. These include similarity searching, pharmacophore modeling, and QSAR modeling.
• Structure-based approaches: use 3D information about the target protein. ML can improve docking scoring functions or directly predict binding affinity from protein-ligand complex features.

Prediction of drug-target interactions

Understanding which proteins a drug interacts with is essential for predicting both efficacy and off-target side effects. ML models predict drug-target interactions using features derived from chemical structure, protein sequence, or both.

Common techniques include:

• Matrix factorization methods: treat the drug-target interaction problem like a recommendation system, filling in missing interactions from known ones (similar to collaborative filtering)
• Network-based approaches: use graph convolutional networks or network embedding to capture relationships in drug-target interaction networks

QSAR modeling for lead optimization

Quantitative Structure-Activity Relationship (QSAR) modeling relates a compound's chemical structure to its biological activity using mathematical equations. You encode molecular structures as numerical descriptors (e.g., molecular weight, logP, topological indices), then train a model to predict activity.

Commonly used algorithms for QSAR include:

• Multiple linear regression (for simpler, linear relationships)
• Support Vector Machines (SVM)
• Random Forest

QSAR models guide lead optimization by predicting how structural modifications will affect activity. They can also incorporate physicochemical and ADMET parameters, letting you optimize multiple properties simultaneously.

De novo drug design strategies

De novo design generates entirely new molecular structures with desired properties, rather than screening existing libraries.

ML approaches for de novo design include:

• Generative models: Variational Autoencoders (VAEs) learn a continuous "latent space" of molecular structures, allowing you to sample new molecules by navigating that space. Generative Adversarial Networks (GANs) use a generator-discriminator pair to produce realistic molecules.
• Reinforcement learning methods: RNNs can generate SMILES strings character by character, with RL rewards guiding the generation toward molecules with optimized properties. Graph Convolutional Networks (GCNs) can build molecules atom-by-atom on a molecular graph.

These methods can produce chemically valid, synthetically accessible, and diverse molecules.

Data preprocessing techniques

Raw chemical and biological data is rarely ready for ML out of the box. Preprocessing ensures data quality and compatibility with your chosen algorithm. Poor preprocessing is one of the most common reasons ML models underperform.

Feature selection and dimensionality reduction

Molecular descriptors can number in the hundreds or thousands. Many will be irrelevant or redundant, and including them can hurt model performance (the "curse of dimensionality").

Feature selection removes unhelpful features:

• Filter methods: rank features by statistical measures like correlation or information gain, independent of the ML algorithm
• Wrapper methods: evaluate subsets of features by training the model repeatedly (e.g., recursive feature elimination, forward/backward selection). More accurate but computationally expensive.

Dimensionality reduction transforms features into a lower-dimensional space:

• Principal Component Analysis (PCA) creates new uncorrelated variables (principal components) that capture the most variance in the data
• t-SNE is used mainly for visualization, projecting high-dimensional molecular data into 2D or 3D plots to reveal clusters

Data normalization and standardization

Different molecular descriptors can have vastly different scales (e.g., molecular weight in hundreds vs. logP values around 1-5). Without scaling, features with larger numerical ranges will dominate the model.

• Normalization (min-max scaling): rescales each feature to a fixed range, typically [0, 1]. Formula: $X_{norm} = \frac{X - X_{min}}{X_{max} - X_{min}}$
• Standardization (z-score): transforms each feature to have a mean of 0 and standard deviation of 1. Formula: $X_{std} = \frac{X - \mu}{\sigma}$

Use standardization when your data has outliers (since min-max scaling is sensitive to extreme values). Use normalization when you need bounded values.

Handling imbalanced datasets

In drug discovery, active compounds are typically far outnumbered by inactive ones. If 95% of your data is inactive, a model that always predicts "inactive" achieves 95% accuracy but is useless for finding hits.

Strategies to address this:

• Oversampling: increase minority class examples. SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic examples by interpolating between existing minority class instances.
• Undersampling: reduce majority class examples by random removal. Simpler but risks losing useful information.
• Class weights: assign higher weights to the minority class during training so misclassifying an active compound incurs a larger penalty.
• Ensemble methods: train multiple models on different balanced subsets and combine their predictions.

Cross-validation methods

Cross-validation estimates how well your model will perform on unseen data, helping you detect overfitting.

1. K-fold cross-validation: split the data into K equal folds. Train on K-1 folds, test on the remaining fold. Repeat K times so every fold serves as the test set once. Average the results.
2. Stratified K-fold: same as K-fold but maintains the class distribution in each fold. This is especially important for imbalanced datasets.
3. Leave-one-out (LOO): K equals the total number of data points. Each instance is used as the test set once. Very thorough but computationally expensive for large datasets.

Cross-validation gives a more reliable performance estimate than a single train-test split.

Machine learning algorithms for drug discovery

The choice of algorithm depends on your data type, dataset size, and whether you need an interpretable model or maximum predictive power.

Support Vector Machines (SVM)

SVM is a supervised algorithm for classification and regression. It finds the hyperplane that maximally separates classes in feature space.

Key concepts:

• Margin: the distance between the hyperplane and the nearest data points from each class. SVM maximizes this margin.
• Support vectors: the data points closest to the hyperplane that define the margin. Only these points determine the decision boundary.
• Kernel trick: when data isn't linearly separable, a kernel function (e.g., radial basis function, polynomial) maps it into a higher-dimensional space where a linear separator exists.

SVMs work well with high-dimensional molecular descriptor data and relatively small datasets, making them popular for virtual screening and QSAR modeling.

Random Forest and decision trees

A decision tree makes predictions through a series of if-then splits on individual features. They're intuitive but prone to overfitting.

Random Forest addresses this by combining many decision trees:

1. Create multiple bootstrap samples (random subsets with replacement) from the training data
2. Build a decision tree on each sample, randomly selecting a subset of features at each split
3. Aggregate predictions across all trees: majority vote for classification, averaging for regression

This "bagging" approach reduces variance and overfitting. Random Forest also provides feature importance scores, which help identify which molecular descriptors drive predictions. This is valuable for guiding medicinal chemistry decisions.

Artificial Neural Networks (ANN)

ANNs learn complex, non-linear relationships between inputs and outputs through layers of interconnected neurons.

Architecture:

• Input layer: receives molecular descriptors or fingerprints
• Hidden layer(s): each neuron computes a weighted sum of its inputs, applies an activation function, and passes the result forward
• Output layer: produces the prediction (e.g., activity class or binding affinity value)

Training uses backpropagation: the algorithm calculates the error at the output, then propagates it backward through the network, adjusting weights to minimize prediction error. This process repeats over many iterations (epochs).

ANNs are flexible and powerful but require larger datasets than SVMs or Random Forests, and they function as "black boxes" with limited interpretability.

Convolutional Neural Networks (CNN)

CNNs are deep learning architectures designed for grid-like data. In drug discovery, they process molecular graphs, 2D fingerprints, or 3D voxelized representations of protein-ligand complexes.

Key components:

• Convolutional layers: apply learnable filters that slide across the input to detect local patterns (e.g., specific substructural motifs)
• Activation functions: non-linear functions like ReLU ($f(x) = \max(0, x)$) applied after convolution
• Pooling layers: downsample the feature maps, reducing dimensionality and providing some translation invariance

CNNs have been applied to predicting drug-target interactions from molecular images, virtual screening, and de novo drug design. Graph convolutional networks (a variant) operate directly on molecular graphs, treating atoms as nodes and bonds as edges.

Evaluating machine learning models

Building a model is only half the work. Rigorous evaluation determines whether your model will actually be useful in practice.

Performance metrics for classification tasks

Classification tasks (e.g., active vs. inactive) use these metrics:

• Accuracy: proportion of correct predictions overall. Misleading with imbalanced datasets.
• Precision: of all compounds predicted active, how many truly are? $\text{Precision} = \frac{TP}{TP + FP}$
• Recall (sensitivity): of all truly active compounds, how many did the model catch? $\text{Recall} = \frac{TP}{TP + FN}$
• F1 score: harmonic mean of precision and recall. Balances both metrics. $F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$
• AUROC: area under the ROC curve, which plots true positive rate vs. false positive rate at varying thresholds. An AUROC of 0.5 means random guessing; 1.0 means perfect classification.

For drug discovery, recall is often prioritized because missing a true active (false negative) is usually worse than testing a false positive in the lab.

Regression evaluation metrics

Regression tasks (e.g., predicting $IC_{50}$ or binding affinity) use:

• Mean Squared Error (MSE): $MSE = \frac{1}{n}\sum(y_{pred} - y_{actual})^2$. Penalizes large errors heavily.
• Root Mean Squared Error (RMSE): $\sqrt{MSE}$. Same units as the target variable, making it more interpretable.
• Mean Absolute Error (MAE): $\frac{1}{n}\sum|y_{pred} - y_{actual}|$. Less sensitive to outliers than MSE.
• R-squared ($R^2$): proportion of variance in the target explained by the model. Ranges from 0 to 1 (higher is better).

Overfitting and underfitting challenges

Overfitting: the model memorizes training data noise instead of learning generalizable patterns. High training performance, poor test performance.

Remedies:

• Regularization (L1/L2 penalties constrain weight magnitudes)
• Dropout (randomly deactivates neurons during training in neural networks)
• Early stopping (halt training when validation performance stops improving)
• More training data

Underfitting: the model is too simple to capture the underlying patterns. Poor performance on both training and test data.

Remedies:

• Increase model complexity (more layers, more neurons, more trees)
• Use more expressive algorithms
• Engineer better features

Cross-validation helps you detect both problems by comparing training and validation performance across multiple data splits.

Model interpretability and explainability

In drug discovery, you need to understand why a model makes a prediction, not just what it predicts. A model that says "this compound is active" is more useful if it can point to the structural features driving that prediction.

• Inherently interpretable models: decision trees and linear regression have transparent decision logic
• Feature importance: Random Forest provides Gini importance scores showing which descriptors matter most
• Partial dependence plots: show how changing one feature affects predictions while holding others constant
• LIME (Local Interpretable Model-agnostic Explanations): approximates any complex model locally with a simpler, interpretable model to explain individual predictions

Interpretability builds trust in model predictions and helps medicinal chemists validate whether the model's reasoning aligns with known SAR (structure-activity relationships).

Integration of machine learning with other approaches

ML is most powerful when combined with other computational and experimental methods rather than used in isolation.

Combining machine learning with molecular docking

Molecular docking predicts how a ligand binds to a target protein, but traditional scoring functions often have limited accuracy. ML can enhance docking in several ways:

• ML-based scoring functions: train on experimentally determined binding affinities to develop more accurate scoring of docked poses
• Post-docking filtering: use ML classifiers to re-rank docking results and prioritize the most promising candidates
• Binding site prediction: ML models can predict favorable binding sites on the protein surface, guiding where to dock

This combination yields more reliable virtual screening results than either method alone.

Machine learning-guided ADMET prediction

ADMET properties determine whether a drug candidate will succeed in clinical trials. ML models predict these properties early, allowing you to filter out problematic compounds before expensive synthesis and testing.

Key ADMET properties predicted by ML:

• Solubility: aqueous solubility affects oral bioavailability
• Permeability: ability to cross biological membranes (e.g., Caco-2 cell permeability as a model for intestinal absorption)
• Metabolic stability: susceptibility to CYP450-mediated degradation
• Toxicity: hERG channel inhibition (cardiac risk), hepatotoxicity, mutagenicity

By incorporating ADMET predictions into the optimization cycle, you can prioritize compounds with favorable pharmacokinetic and safety profiles early in the pipeline.

Synergy with high-throughput screening

High-throughput screening (HTS) experimentally tests large compound libraries against a target, but it's expensive and generates noisy data. ML complements HTS by:

• Pre-screening: virtually screening compounds to select a focused subset for HTS, reducing costs
• Hit expansion: after HTS identifies initial hits, ML models predict activity for untested compounds, finding additional actives without more screening
• Data analysis: identifying patterns in HTS results that connect chemical structure to activity, guiding follow-up chemistry

Integration with systems biology and omics data

Systems biology uses computational modeling to understand complex biological networks. Combining ML with omics data (genomics, transcriptomics, proteomics, metabolomics) enables:

• Identification of novel drug targets through analysis of disease-associated gene expression patterns
• Prediction of drug response based on patient-specific omics profiles (supporting precision medicine)
• Understanding of drug mechanisms of action through network-level analysis of how compounds perturb biological pathways

This integration moves drug discovery beyond single-target approaches toward a more holistic understanding of how drugs interact with biological systems.