To answer the above research questions, we collected 20 open-source Android apps from GitHub that satisfy the following three constraints: (1) contain at least one commit after October 2017 (i.e., actively-maintained), (2) do not overlap with any projects selected for our empirical study, and (3) use at least one of the callback APIs in the seven PI-Graphs encoded in CIDER. We used the latest version of these 20 Android apps as our evaluation subjects to examine if CIDER can detect new callback compatibility issues in them.
To replicate the evaluation results of CIDER, please download the apk files of our evaluation subjects here
The following table shows the evaluation result of CIDER.
| No. | Project Name | Stars | Downloads | KLOC | True Positives | False Positives | Issue Report |
|---|---|---|---|---|---|---|---|
| 1 | AFWall+ | 1,100 | 500K+ | 21.8 | 1 | 0 | 786 |
| 2 | Calendula | 76 | 1K+ | 26.3 | 0 | 0 | - |
| 3 | cccTV | 20 | 100+ | 7.8 | 2 | 0 | 8 |
| 4 | DuckDuckGo-Kotlin | 425 | 1M+ | 10.4 | 1 | 0 | 79 |
| 5 | FOSS Browser | 101 | - | 18.0 | 0 | 0 | - |
| 6 | Kolab Notes | 42 | 1K+ | 73.4 | 0 | 1 | - |
| 7 | MaterialFBook | 90 | - | 68.0 | 0 | 0 | - |
| 8 | Network-monitor | 54 | 50K+ | 20.8 | 0 | 0 | - |
| 9 | NyaaPantsu | 22 | - | 14.1 | 0 | 0 | - |
| 10 | OONI Probe | 41 | 100K+ | 4.9 | 1 | 0 | 146 |
| 11 | OpenKeyChain | 1,001 | 100K+ | 848.5 | 0 | 0 | - |
| 12 | OsmAnd | 1,410 | 5M+ | 662.5 | 1 | 0 | 4868 |
| 13 | Padland | 22 | 100+ | 58.9 | 1 | 0 | 47 |
| 14 | PassAndroid | 362 | 1M+ | 85.0 | 0 | 0 | - |
| 15 | Ring | 82 | 1M+ | 243.5 | 1 | 0 | 1831 |
| 16 | sg for SteamGifts | 40 | 1K+ | 21.5 | 0 | 0 | - |
| 17 | Simple-Solitaire | 49 | 10K+ | 294.4 | 1 | 0 | 108 |
| 18 | SuntimesWidget | 24 | - | 63.1 | 0 | 0 | - |
| 19 | SurvivalManual | 326 | 1M+ | 49.4 | 0 | 0 | - |
| 20 | Uber-ride | 209 | - | 12.7 | 4 | 0 | 105 |