By: Albi Celaj, Director, Machine Learning/AI Lab
At Deep Genomics, we have developed a model called DeepRNAi for predicting siRNA activity, both on- and off-target effects, across the transcriptome. We’ve shown that it is accurate to the point of rivalling experimental screening methods, which is in sharp contrast to all existing bioinformatic and machine learning methods to date.
What I want to talk about here is the crucial role that biological foundation models (BioFMs) play in DeepRNAi. Building these models takes a lot of people, expertise, and compute, so it’s important to evaluate whether they make a difference in downstream applications, in this case siRNA activity and off-target prediction.
A major problem we had to solve when developing DeepRNAi is how to combine two opposite views that the model needs to have. These views operate at different scales - the scale of “local” siRNA-transcript hybridization sites, and of “global” context covering the genomic region of the transcript as a whole.
On one hand, we need a sequence-specific model that focuses on the siRNA sequence and its alignments within on- and off-target transcripts. We know that siRNA knockdown is affected by the specific hybridization of the siRNA itself, as well as properties of the binding site (e.g. overall secondary structure and accessibility). Given the therapeutic importance of siRNAs, and the importance of endogenous micro-RNAs for regulation, a lot of impressive work has been done on this problem, like with the TargetScan models and REPRESS. We discovered early on with DeepRNAi development that we must make sure that the model is given local target-context information. The view is generally that the siRNA-loaded RISC complex must interact favourably with the target sequence, which is opened up and degraded.
Generally, we found that models that only take local target context as input work in some cases, but largely do not provide the level of accuracy that is useful for drug discovery purposes. Why? Because they are entirely missing the global part of the equation. Factors like transcript stability, expression levels, localization, site position, and cell state also matter. They can make all the difference in a drug discovery context.
We can try to enumerate all of the relevant factors based on prior knowledge, and that’s exactly what my colleagues and I did at Deep Genomics when we created a baseline Random Forest model. We then built a much more powerful deep learning model that uses an innovative architecture to take into account multi-scale context. To do this, we took advantage of the extensive efforts we’ve made, as have others, to build biological foundation models (BioFMs) of genome biology. Our flagship models are BigRNA, Enigma, and REPRESS. Unlike the local view of siRNA binding, these models take into account up to a million base-pairs of sequence context to predict how a DNA or RNA sequence behaves in a real cell, across thousands of outputs, and provide rich high-dimensional per-base-pair embedding in doing so. While they are of minimal use on their own when predicting siRNA activity, when we generated fit-for-purpose data profiling siRNAs to build upon their foundation, we obtained impressive performance on both statistical/ML metrics and real drug discovery tasks.
So, what happens when we use the local and global scales to have the best of both worlds? The simplest way to quantify the gain from BioFMs is to look at overall performance metrics and contrast them with versions of the model where the contribution of BioFMs is ablated. Let’s take a look at the performance in predicting seed-mediated knockdown again, and compare the full model against various baselines:
For the DeepRNAi model without BioFMs, we performed about as well in aggregate as the Random Forest approach, which contains manual transcript context features. (Note that the random forest approach performs even more poorly when tested on unseen transcripts, as discussed in my last post.) We nearly double our performance by incorporating BioFMs, and this makes us able to provide high-confidence predictions that are validated in follow-up experiments.
There are many ways to dig into this, but what actually helped drug developers understand how DeepRNAi benefits from BioFMs was the ways in which its predictions “surprised” us. While I cannot disclose the targets we are working on, I can provide two examples relevant to drug discovery.
First, when going through the patent literature for Alnylam’s PLG program, we found that DeepRNAi predicted that many of Alnylam’s molecules would have very little on-target effect. In fact, the predictions were near zero! We initially thought these could be errors and we should revisit model training, but upon closer inspection, we found that the predictions made biological sense. These siRNAs were targeting the distal 3’ UTR region of the annotated PLG transcript, which, as it turns out, is downstream of the primary poly(A) site and not well-expressed in liver cell lines. What’s impressive is that the BioFM automatically provided this context to DeepRNAi, so that it could incorporate this biology into its predictions. In contrast, we found that the version of DeepRNAi that does not use BioFMs made very different predictions for the siRNAs in the UTR region (see below).
While we can only speculate that these molecules were screened at Alnylam due to the selection of an isoform that is not fully expressed in the liver, their experimental data shows (see below), that the DeepRNAi predictions were correct: the siRNAs targeting this distal region were indeed inactive.
A second example came up for us when designing siRNAs that avoid a lack of activity caused by common human variation. A standard bioinformatics pipeline will avoid tiling over regions with common variants, but cannot account for variant effects that alter siRNA efficacy (e.g. through alternative splicing or polyadenylation). When we looked at the IRF5 gene (a potential siRNA target for reduced inflammatory signaling and improved infarct healing), we discovered a variant-induced version of this same phenomenon that standard RNA-seq would miss. Here, a common variant (allele frequency = 58% globally) creates a proximal polyadenylation signal leading to an isoform lacking ~600 bp of the distal 3′ UTR. DeepRNAi with BioFMs immediately flagged that siRNAs tiling this region are unlikely to work for a large portion of the human population, where as a model without BioFMs just tells you to avoid siRNAs that bind to a very precise locus that contains the variant:
Because we are interested in applying DeepRNAi for drug discovery, the above examples illustrate cases in real deployment where the BioFMs made a meaningful difference in our decision making. With a one-million base-pair view, and each base-pair receiving a high-dimensional embedding, this is the “tip of the iceberg” for the meaningful context that they provide for our model. We are looking forward to diving in further to the more nuanced biology that they provide.
Across this series, following on Brendan’s post on why genetic medicine has an AI-friendly technology stack, we’ve announcedDeepRNAi, the experimental data needed to build it, why learning causal signal matters, and how its predictions rival experimental reproducibility. The BioFM context described above was critical in making DeepRNAi useful in drug discovery. It has been exciting using DeepRNAi in our own programs to see the difference it makes in designing cleaner and more effective molecules. I’m very proud of what our team has built, and excited to announce that we will soon be releasing the DeepRNAi pre-print, which will give a much more detailed view of the methods and results behind this work. Stay tuned!
To read the previous pieces in this series, visit the Deep Genomics blog here.