Smartphone-based technology is developing at high speed, and many apps offer potential new ways of monitoring and treating a range of psychiatric disorders and symptoms. However, the effects of most available apps have not been scientifically investigated. Within medicine, randomized controlled trials (RCTs) are the standard method for providing the evidence of effects. However, their rigidity and long time frame may contrast with the field of information technology research. Therefore, a systematic review of methodological challenges in designing and conducting RCTs within mobile health is needed.