AffyDEComp: Towards a benchmark for comparing differential expression detection methods on Affymetrix data

Method comparison table

AffyDEComp, like Affycomp, is a tool for benchmarking methods for analysing Affymetrix microarray data. Whereas Affycomp concentrates on expression summarisation methods, the focus of AffyDEComp is on differential expression (DE) detection methods, or, more precisely, on the combination of summarisation and DE detection methods. AffyDEComp is currently based on the Golden Spike data set described in the paper Preferred analysis methods for Affymetrix GeneChips revealed by a wholly defined control dataset of Choe et al. 2005. We believe that, at present, this is the best publicly-available data set for comparison of Affymetrix DE detection methods. However, we also recognise that the data set used might not be representative of data sets more generally. In particular, just because a method does well here, doesn't necessarily mean the method will do well on your data sets. In the future we plan to extend AffyDEComp to other data sets. Further details are included in the paper "A comprehensive re-analysis of the Golden Spike data: towards a benchmark for differential expression methods" by Richard D. Pearson, which has recently been submitted.

The table below gives key performance indicators (KPIs) for different combinations of summarisation and differential expression (DE) methods. Higher values indicate stronger performance for each KPI (with 1 being a perfect classifier), with the exception of KPI 13, where lower values indicate stronger performance (with 0 being perfection).

To order methods by a particular KPI, click on the number at the top of the relevant column. The currently ordered KPI is highlighted in green. To view ROC charts related to a particular KPI, click on the "c" link at the top of the relevant column. To see details of each KPI, click on the question mark (?) at the top of the relevant column. Brief descriptions of each KPI are also included below the table. Click on the name of a method to go to the paper describing that method. fc stands for fold change, and ttest for a standard T-test.

1 c ? 2 c ? 3 c ? 4 c ? 5 c ? 6 c ? 7 c ? 8 c ? 9 c ? 10 c ? 11 c ? 12 c ? 13 c ?
mmgMOS / pplr 0.9507 0.9218 0.8887 0.9906 0.9956 0.9045 0.9369 0.8791 0.7511 0.6540 0.2134 0.9684 0.0004
mmgMOS / cybert 0.9487 0.9187 0.8842 0.9908 0.9949 0.9002 0.9320 0.8812 0.8617 0.6741 0.4050 0.9684 0.0008
CP / cybert 0.9475 0.9190 0.8856 0.9869 0.9916 0.9196 0.9317 0.8839 0.9123 0.6965 0.4771 0.9684 0.0004
GCRMA / fc 0.9455 0.9021 0.8837 0.9849 0.9891 0.9461 0.9503 0.9000 0.9264 0.6617 0.4050 0.9699 0.0008
GCRMA / cybert 0.9444 0.9091 0.8741 0.9902 0.9934 0.9026 0.9395 0.8894 0.9187 0.6753 0.4140 0.9707 0.0004
PLIER / cybert 0.9413 0.9110 0.8676 0.9893 0.9936 0.8657 0.9167 0.8843 0.8876 0.6020 0.4576 0.9639 0.0012
mmgMOS / limma 0.9400 0.9025 0.8639 0.9898 0.9944 0.8451 0.9099 0.8824 0.8669 0.4805 0.1127 0.9684 0.0043
CP / limma 0.9400 0.9046 0.8683 0.9879 0.9892 0.8898 0.9209 0.8852 0.9042 0.5648 0.3358 0.9699 0.0020
mmgMOS / ttest 0.9382 0.9017 0.8655 0.9850 0.9907 0.8820 0.9238 0.8778 0.8014 0.4843 0.0661 0.9684 0.0036
FARMS / fc 0.9376 0.8909 0.8672 0.9825 0.9869 0.9135 0.8115 0.9114 0.8977 0.6877 0.5672 0.9572 0.0000
CP / ttest 0.9353 0.8977 0.8643 0.9821 0.9846 0.8980 0.9253 0.8790 0.8985 0.4941 0.0173 0.9707 0.0047
PLIER / limma 0.9345 0.8982 0.8549 0.9855 0.9916 0.8347 0.9056 0.8838 0.8807 0.5216 0.2314 0.9639 0.0028
MBEI / cybert 0.9341 0.8967 0.8505 0.9884 0.9924 0.8589 0.9066 0.8767 0.8850 0.6241 0.4628 0.9564 0.0004
MAS5 / cybert 0.9338 0.9049 0.8545 0.9841 0.9902 0.9054 0.9236 0.8681 0.8750 0.6342 0.3989 0.9512 0.0012
CP / pplr 0.9324 0.8890 0.8570 0.9805 0.9859 0.8808 0.9194 0.8786 0.8964 0.4258 0.0391 0.9699 0.0063
RMA / fc 0.9320 0.8846 0.8400 0.9921 0.9956 0.8244 0.9022 0.8826 0.9112 0.6094 0.5041 0.9594 0.0000
RMA / cybert 0.9317 0.8857 0.8419 0.9898 0.9944 0.8298 0.9029 0.8803 0.9081 0.5974 0.4568 0.9579 0.0012
PLIER / ttest 0.9296 0.8889 0.8517 0.9789 0.9859 0.8392 0.9067 0.8796 0.8844 0.3847 0.1007 0.9624 0.0083
RMA / limma 0.9292 0.8815 0.8370 0.9887 0.9937 0.8200 0.9015 0.8815 0.9114 0.5815 0.4591 0.9594 0.0008
PLIER / fc 0.9292 0.8888 0.8436 0.9817 0.9924 0.7693 0.8739 0.8791 0.8800 0.3655 0.0098 0.9639 0.0103
MBEI / limma 0.9285 0.8847 0.8383 0.9863 0.9917 0.8423 0.9030 0.8775 0.8923 0.5657 0.4343 0.9564 0.0008
CP / fc 0.9281 0.8725 0.8485 0.9760 0.9864 0.7599 0.8346 0.8790 0.8836 0.1775 0.0023 0.9699 0.0174
MBEI / fc 0.9278 0.8840 0.8355 0.9874 0.9925 0.8368 0.9012 0.8776 0.8948 0.5229 0.1856 0.9564 0.0024
GCRMA / limma 0.9264 0.8905 0.8473 0.9751 0.9843 0.8683 0.9321 0.8723 0.9069 0.3337 0.0826 0.9707 0.0126
MAS5 / limma 0.9244 0.8844 0.8331 0.9815 0.9900 0.8627 0.9040 0.8714 0.8819 0.5224 0.2494 0.9519 0.0043
MAS5 / ttest 0.9213 0.8788 0.8353 0.9734 0.9844 0.8875 0.9158 0.8651 0.8689 0.4150 0.0579 0.9527 0.0067
GCRMA / ttest 0.9205 0.8831 0.8410 0.9687 0.9795 0.8400 0.9269 0.8670 0.9028 0.2548 0.0526 0.9707 0.0178
MBEI / ttest 0.9201 0.8699 0.8311 0.9737 0.9850 0.8322 0.9000 0.8697 0.8770 0.3473 0.0526 0.9572 0.0142
mmgMOS / fc 0.9201 0.8609 0.8387 0.9661 0.9833 0.6436 0.8214 0.8703 0.8995 0.0575 0.0008 0.9684 0.0327
DFW / fc 0.9179 0.8149 0.8261 0.9749 0.9845 0.9020 0.9174 0.9036 0.8908 0.6894 0.5695 0.9309 0.0000
RMA / pplr 0.9166 0.8600 0.8225 0.9718 0.9866 0.8006 0.8974 0.8715 0.8903 0.3530 0.0443 0.9587 0.0107
MBEI / pplr 0.9152 0.8554 0.8110 0.9785 0.9912 0.7638 0.8861 0.8713 0.8993 0.3658 0.0819 0.9564 0.0150
RMA / ttest 0.9137 0.8580 0.8219 0.9694 0.9810 0.7881 0.8955 0.8701 0.8913 0.3312 0.1443 0.9587 0.0146
DFW / pplr 0.9121 0.8059 0.8140 0.9726 0.9839 0.8963 0.9077 0.8982 0.8877 0.6266 0.0000 0.6273 0.3535
MAS5 / fc 0.9084 0.8482 0.8059 0.9673 0.9859 0.7154 0.8172 0.8631 0.8693 0.1545 0.0008 0.9512 0.0229
FARMS / cybert 0.9079 0.8443 0.7961 0.9759 0.9888 0.7542 0.7965 0.8635 0.9068 0.4880 0.3944 0.9572 0.0047
FARMS / pplr 0.8932 0.7721 0.7922 0.9417 0.9778 0.9025 0.8111 0.8958 0.9121 0.4426 0.0526 0.9557 0.0051
FARMS / limma 0.8827 0.8416 0.7905 0.9289 0.9563 0.6803 0.8005 0.8624 0.8966 0.3144 0.1427 0.9564 0.0564
FARMS / ttest 0.8472 0.8053 0.7697 0.8877 0.9078 0.5298 0.8058 0.8246 0.8926 0.0731 0.0586 0.9564 0.1373
DFW / cybert 0.8297 0.7030 0.6860 0.8684 0.9694 0.4665 0.8968 0.8116 0.9052 0.2696 0.2487 0.9294 0.1657
DFW / limma 0.8167 0.7641 0.7793 0.8456 0.8380 0.6197 0.8902 0.8347 0.8952 0.0561 0.0060 0.9234 0.1314
DFW / ttest 0.7943 0.7324 0.7729 0.8158 0.8022 0.5655 0.8846 0.8050 0.8938 0.0463 0.0000 0.9234 0.1775

KPIs:

  1. AUC where equal-valued spike-ins are used as true negatives, spike-ins with FC > 1 are used as true positives, a post-summarization loess normalization based on the equal-valued spike-ins is used, and a 1-sided test of up-regulation is the DE metric.
  2. as 1. but using a 2-sided test of DE.
  3. as 1. but with low FC spike-ins used as true positives.
  4. as 1. but with medium FC spike-ins used as true positives.
  5. as 1. but with high FC spike-ins used as true positives.
  6. as 1. but with all unchanging probesets used as true negatives.
  7. as 1. but with all unchanging probesets used as true negatives, and a post-summarization loess normalization based on the unchanging probesets.
  8. as 1. but with a post-summarization loess normalization based on all spike-in probesets.
  9. as 1. but with a no post-summarization normalization.
  10. as 1. but giving the AUC for FPRs up to 0.01
  11. the proportion of true positives without any false positives (i.e. the TPR for a FPR of 0), using the same conditions as 1.
  12. the TPR for a FPR of 0.5, using the same conditions as 1.
  13. the FPR for a TPR of 0.5, using the same conditions as 1.
AffyDEComp was created and is maintained by Richard Pearson. If you would like a method added to AffyDEComp, this has to be available in a Bioconductor package. Contact Richard Pearson for further details.

To cite AffyDEComp, please cite the paper "A comprehensive re-analysis of the Golden Spike data: towards a benchmark for differential expression methods" by Richard D. Pearson which can be found here.

AffyDEComp version 1.1 - This page last modified 27 March 2008
Other versions of AffyDEComp are available here