Systematic review of artificial intelligence competitions in radiology: a focus on design, evaluation, and trends

Artificial intelligence (AI) in healthcare is evolving through human–machine collaboration, with innovation driven by partnerships between academic healthcare institutions and industry. The proper validation of AI algorithms, effective data sharing, and training for radiologists is essential.¹ Fundamental requirements and quality standards applicable to all AI-related organizations have begun to be established.²

A study examining the impact of AI on radiology and medical imaging through web searches revealed a prevailing positive outlook, highlighting the leading role of radiologists in this discourse.³ Radiology department chairs tend to be optimistic, believing that AI will be beneficial in areas such as quality, efficiency, healthcare costs, and interpretation workflow.⁴ Although radiologists support the idea that AI will streamline workflow, medical students and surgeons approach it more cautiously.⁵

Despite potential biases and pitfalls in the use of AI technologies in medical imaging, their development and advancement are achievable through grand challenges. The expected benefits include creating code and trained datasets, openly sharing them, generating new work areas, and directly involving AI in patient care.⁶

With the widespread use of AI in the medical field, this systematic review aims to investigate the effectiveness of recently organized and popular radiological imaging competitions worldwide.

Ethical committee approval and patient consent are not required for this type of article. A search was conducted on the PubMed database using the terms “competition” or “contest” added to the phrase “AI.” The focus was on articles containing result reports of imaging-related competitions between 2019 and 2023. Completed competitions were identified using the “completed” filter on the grand-challenge website. Versions of identified competitions held in previous or subsequent years were also considered. A total of 26 competitions that provided sufficient information and had a substantial impact were included in the review (Figure 1).

Information recorded for each competition included the competition’s name, year held, imaging modality, target region, search field, dataset source, dataset sample size, dataset accessibility, diversity of contributing institutions, derived academic publications (as of January 2024), citation count according to the Web of Science criteria (as of January 2024), competition location, evaluation criteria, and the number of participating individuals or teams.

This review presents the characteristics of 26 AI and medical imaging-related competitions and datasets between 2017 and 2023 (Tables 1 and 2). These competitions were hosted by organizations such as the Annual Aviation, Space, and Technology Festival (TEKNOFEST), the Radiological Society of North America (RSNA), and the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), either individually or collectively. Final competitions or winner announcements were held either onsite or online.

Various imaging modalities, including magnetic resonance imaging, computed tomography, ultrasound, mammography, and digital breast tomosynthesis, were utilized. Competition themes covered different body regions, ranging from the head to the lower limb, with a focus on segmentation, cancer detection, and disease diagnosis. Most competitions used datasets from universities, but some also incorporated data from national teleradiology systems or screening programs. Although the majority of competition datasets were openly accessible, some required approval for access. One competition was conducted exclusively online, whereas others took place both online and onsite.

In the TEKNOFEST competitions, high school students competed in a separate category, distinguishing them from other competitions. Studies derived from these competition datasets were predominantly published in high-impact journals.

The current review aims to evaluate AI applications in medical imaging competitions, which are rapidly increasing in today’s medical imaging landscape. High-participation competitions are organized online or onsite in different parts of the world. Collaboration in dataset preparation involves radiologists, clinicians, engineers, and data scientists from different countries and institutions. Studies produced after competitions are published in high-quality journals, and their citation potential is relatively high. Competitions play an effective role in increasing the positive impact and benefits of AI in medical imaging and in generating greater interest in this field.

Organizations such as RSNA, MICCAI, and TEKNOFEST, or online platforms such as the grand-challenge website, host these competitions.^{7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35} Dataset organization teams have sometimes come together as multinational teams and are generally multi-institutional. AI competitions in medical imaging lead to the establishment of collaborations not only between interdisciplinary teams but also between institutions and countries, both for competition teams and data preparation teams. The robust infrastructure of national teleradiology systems and the strict preservation of imaging data enable the preparation of competition datasets and the generation of results closest to real-world data.

A study examining 2,517 clinical trials related to AI-associated medical devices revealed that research is generally conducted in specific countries at the national level, with studied populations limited to certain regions. In the last few decades, the development of AI technologies in the medical field has turned into a global competition led by China and the United States.³⁶ Allowing free participation from around the world in AI competitions in the health sector is increasing the momentum of innovation. The expansion of competitions to low-income countries will diversify the data population and facilitate the availability of developed software for the benefit of these countries.

In 2023, a competition format involving young radiologists and radiology trainees was first organized at the European Society of Medical Imaging Informatics Annual Meeting in Pisa, Italy; this marked a milestone in radiologists’ orientation toward AI.³⁷ Participating in such competitions during the radiology training period can contribute to radiology education in the current era of strong momentum in AI and radiology collaboration.

In a survey conducted among medical students in Canada, it was observed that although radiology specialization was among the top choices, there were widespread concerns about the negative effects of AI on radiologists. Information meetings are suggested to address these concerns.³⁸ The negative effects of AI on radiology career development have also been noted among US medical students.³⁹ Public competitions involving medical students will contribute to a more realistic understanding of the relationship between AI and radiology expertise. Encouraging high school students to participate in some competitions strategically promotes AI development and raises social awareness among young individuals who have not yet made career choices. Technology teachers at the high school and even middle school levels can take the lead in encouraging participation in such activities during their training.

Additionally, such competitions can lead to the generation of new study topics on emerging issues and the establishment of new networks, facilitating the creation of start-ups. AI summer schools in medicine for high school students have begun to be established at universities.⁴⁰ Ethical dilemmas such as bias risk and data security, along with AI’s potential to assist medical professionals, cannot be overlooked in the realm of AI in healthcare.⁴¹ AI training programs should comprehensively address all these aspects.

The impact of AI-related medical imaging competitions on scientific publication conversion, citation potential, and integration into the literature was investigated. However, another crucial aspect—their clinical application and commercial utilization—lacks sufficient and effective information based on available datasets and publications. To bridge the gap between scientific innovation and clinical practice, it may be beneficial to increase awareness of these competitions among healthcare institutions, AI-related public organizations, and commercial entities while also expanding networking opportunities for competition participants.

Efforts have been made to standardize and enhance transparency in the evaluation of medical imaging competitions, from defining the competition’s mission to dataset preparation methodologies and participant ranking metrics and criteria. However, substantial variations have been observed across these stages.⁴² Proper competition design and interpretation can facilitate the validation of AI algorithms and promote their translation into clinical applications.⁴³ Several factors influence the outcomes of AI-related medical imaging competitions, including the dataset used, the reference annotations determined by annotators, and the scoring system applied for ranking.⁴⁴ Quality control at all stages of a competition enhances the validity and reliability of its results. In our review, a comprehensive framework has been established, detailing the design, execution, and outcomes of current AI-related medical imaging competitions.

This review has some limitations. Not all the databases where competitions could be included were examined for all dates. However, by focusing on recent competitions in the most well-known databases and platforms, an attempt was made to minimize selection bias. There are only a few studies in the literature examining competitions related to AI and radiology.^{6, 45} However, our review is the first to address the dataset, organization teams, and competition features.

In conclusion, as AI continues to play an increasing role in radiology, competitions related to AI and medical imaging contribute to quality dataset sharing, collaboration among experts, and increased awareness in this field.