EVALUATION OF SOFTWARE FOR MUTATIONAL SIGNATURE DISCOVERY

Abstract

Mutational signature analysis has broad clinical and epidemiological applications, which include quantifying exposures to chemical mutagens, screening for defective DNA repair mechanisms and finding anti-tumor treatments with better clinical outcomes. All these applications require software to accurately extract mutational signatures from the mutation spectra of tumor genomes. However, there have been few benchmarking studies to evaluate the accuracy of signature discovery by these programs. In response to this gap, I carried out two benchmarking studies that use synthetic cancer data sets to assess the accuracy of signature discovery. Both studies showed that two programs (mSigHdp and SigProfilerExtractor) outcompeted others. On the other hand, the two studies also showed that some software had unstable performance across random seeds and synthetic datasets.

The two studies highlighted the importance of selecting the correct number of signatures and of selecting program-specific hyperparameters. The studies also indicated that users should do multiple runs with different random seeds to determine stability and use available diagnostics provided by the software to adjust the numbers of signatures, random seeds, and program-specific parameters.