ABSTRACT

The rapid evolution of artificial intelligence (AI), particularly in deep learning, has significantly impacted radiology, introducing an array of AI solutions for interpretative tasks. This paper provides radiology departments with a practical guide for selecting and integrating AI solutions, focusing on interpretative tasks that require the active involvement of radiologists. Our approach is not to list available applications or review scientific evidence, as this information is readily available in previous studies; instead, we concentrate on the essential factors radiology departments must consider when choosing AI solutions. These factors include clinical relevance, performance and validation, implementation and integration, clinical usability, costs and return on investment, and regulations, security, and privacy. We illustrate each factor with hypothetical scenarios to provide a clearer understanding and practical relevance. Through our experience and literature review, we provide insights and a practical roadmap for radiologists to navigate the complex landscape of AI in radiology. We aim to assist in making informed decisions that enhance diagnostic precision, improve patient outcomes, and streamline workflows, thus contributing to the advancement of radiological practices and patient care.

Keywords: Radiology, artificial intelligence, clinical decision-making, computer-assisted healthcare economics and organizations, data security in healthcare, regulatory compliance in medicine

Main points

• The paper provides guidance on choosing artificial intelligence (AI) solutions that align with clinical goals and enhance diagnostic accuracy in radiology

• It emphasizes the importance of thorough performance evaluation and external validation of AI models for reliable clinical application.

• The paper highlights the necessity for AI solutions to integrate smoothly into existing workflows with user-friendly interfaces.

• The paper discusses the financial aspects of AI solutions, focusing on cost-effectiveness and the potential for a positive return on investment.

• The paper stresses the importance of adhering to regulatory standards and ensuring data security and privacy in AI integration in radiology.

In the initial phases of DL’s emergence in the healthcare sector, its application was largely confined to academic and research settings.³ These environments provided the perfect proving ground for DL algorithms, allowing researchers to refine their models against vast repositories of medical images and data. However, as the technology matured and demonstrated its efficacy, there was a noticeable shift from academic research to real-world clinical applications.^4-7

This transition was marked by a significant uptick in the development and deployment of DL-based tools and systems in clinical settings. As of 2023, the market has seen a proliferation of DL-based applications, with approximately 692 United States Food and Drug Administration (FDA)-approved and 220 “Conformité Européenne” (CE)-marked products available from a diverse range of vendors.^8,9 This surge reflects not only the growing confidence in DL technologies but also an increasing demand for advanced AI tools in medical diagnostics. Each year, the commercial market is enriched with new and innovative DL products designed for medical image interpretation, signaling a dynamic and rapidly evolving landscape in radiological practices. Figure 1 illustrates the commercially available CE and FDA-marked DL applications per subspecialty, while Figure 2 depicts the number of commercially available FDA-approved applications each year since 2010.⁸

Figure 1

Figure 2

In radiology, DL is broadly utilized for two types of tasks: interpretative and non-interpretative. Interpretative tasks include quantification, segmentation, and diagnosis, which traditionally required manual labor from radiologists or were not available at all.^10-12 Non-interpretative tasks, on the other hand, encompass image creation using DL reconstruction, patient scheduling, and other administrative processes.^13,14 This distinction is crucial in understanding the comprehensive impact of DL in this field.

Although non-interpretative tasks are critical for the efficiency of radiology departments, they predominantly address administrative aspects that typically concern a specific subset of radiologists, such as department heads. These tasks are often managed through built-in features provided by image manufacturers, hospital information systems, or picture archive and communication systems, which may minimize the need for active radiologist involvement in decision-making.

In contrast, interpretative tasks, which form the core of radiological practice, are significantly impacted by DL technologies. These tasks necessitate the expertise and active participation of radiologists, placing them at the forefront of decision-making processes and directly benefiting from DL advancements.

Scope of the paper

Considering the substantial role and direct involvement of radiologists in interpretative tasks, this paper focuses on the selection and implementation of AI solutions for these specific functions. The emphasis on interpretative tasks aligns with the overarching goal of the paper: to guide radiology departments in choosing AI solutions that not only enhance diagnostic precision but also improve patient outcomes and streamline workflows.

This review-opinion paper does not aim to provide an exhaustive list of commercially available applications or to meticulously review the scientific evidence behind them, as these topics have already been explored in earlier studies.^15-18 Instead, the focus is on outlining the key factors to consider when choosing the right AI solutions for your radiology department. While several earlier review-opinion pieces have addressed key factors relevant to purchasing an AI solution for radiology departments,^18-20 the rapid pace of developments in this field highlights the need for up-to-date and practical guidance.

Furthermore, the authors of this paper, drawing on over 5 years of clinical, industrial, and academic experience, offer unique insights on each topic, thereby contributing to and enriching the existing literature. Throughout this paper, we aim to illustrate each factor with hypothetical scenarios to enhance understanding and relatability for our readers, thereby making the content more accessible and practically applicable. The genesis of these hypothetical scenarios lies in our extensive firsthand experience, which spans clinical usage, participation in radiology hackathons, contributions to academic research, and involvement in the development and assessment of AI technologies-both commercially available and in experimental stages. While these scenarios are presented in a hypothetical format, they are deeply informed by real-world situations and challenges we have encountered in our professional journey. This methodological choice is driven by our commitment to sharing valuable, generalized insights without referencing specific brands, entities, or institutions, thereby avoiding potential bias and preserving the focus on the universal applicability of the guidelines we propose.

These scenarios are crafted to provide a clearer understanding of the fundamental aspects of each factor, enabling readers to make more informed decisions when considering the purchase of an AI solution. Ultimately, this paper aspires to serve as a practical guide for radiologists and department administrators, aiding them in making well-informed decisions about integrating AI solutions into their practice and thus contributing significantly to the advancement of radiological services and patient care.

Key factors to consider

Among the numerous commercially available AI products, the first step for radiologists is to sift through the “hype” surrounding the use of AI and assess its “clinical relevance.” This involves identifying their current or near-term needs and goals and determining which AI products may meet those needs. We have determined that assessing clinical relevance is the most crucial initial step for radiologists. This is because, without ensuring the suitability of a proposed solution for the department’s needs, attention to other outlined aspects may divert focus and complicate the decision-making process. Indeed, in its 2019 white paper, the European Society of Radiology underscored the significance of the clinical relevance of AI products to specific radiology departments, introducing the term “use case.” This paper defines a use case as a specific clinical application of AI in radiology, suggesting that use cases represent precise scenarios within the radiology service chain where automation could deliver significant value and help establish standards.²¹

Following this initial evaluation of a product’s use case and its relevance to a radiology department’s needs (i.e., clinical relevance) and drawing from our experience and previous work, radiologists should consider the following key factors: performance and validation,²² implementation and integration,⁴ clinical usability,^23,24 costs and return on investment (ROI),⁴ regulations,^25,26 security and privacy.^27-29

Figure 3 presents a roadmap for choosing the right AI solutions for radiology departments, addressing these key factors. Additionally, Table 1 provides a checklist presenting items for each factor when purchasing an AI solution for a radiology department.

Figure 3

Table 1

Clinical relevance

In the context of the burgeoning AI technology market, it is crucial for radiology departments to critically review and evaluate the myriad of commercial AI products available. An AI solution can provide a wide spectrum of clinical value, as documented in earlier work.^30,31 However, most current AI solutions in medical imaging are designed to focus on one or, at most, two aspects of radiological tasks, limiting their scope to specific diagnostic or operational challenges. This specialization underscores the narrow focus of AI, which, while beneficial in certain contexts, may not fully address the complexity of radiological diagnostics or operational efficiency. These products, while offering a broad spectrum of potential clinical value, are predominantly examples of “narrow AI.” Narrow AI, also known as weak AI, refers to AI systems that are designed and trained for a particular task. These systems, unlike their “broad AI” or “artificial general intelligence (AGI)” counterparts, do not possess the ability to perform any intellectual task that a human can. An example of AGI, which remains a theoretical concept at this stage, would be an AI such as ChatGPT, which can engage in a wide range of tasks, including conversing, reasoning, and learning across different domains without being explicitly programmed for them.³²

Given this backdrop, the clinical relevance of an AI solution for one department may not necessarily align with the needs of another, emphasizing the importance of departments defining their unique requirements and expectations from AI technologies. This is best illustrated through a hypothetical scenario: imagine an AI solution designed for prostate magnetic resonance imaging (MRI) interpretation, aiming to automate the Prostate Imaging Reporting and Data System process.³³

This hypothetical product does not offer automated volumetric analyses of the gland or prostate density scores using prostate-specific antigen, nor does it assess the probability of lesions harboring clinically significant prostate cancer. Its performance surpasses that of radiology residents and less-experienced readers but does not reach the level of expertise of prostate imaging specialists. Primarily, this product is aimed at institutions with high volumes of prostate MRI examinations and those employing less-experienced radiologists or residents for interpretations.

In an academic center specializing in prostate imaging, where radiology residents frequently pre-read prostate MRI examinations, such a product may yield high user satisfaction and a strong ROI. Conversely, its benefits may be minimal in a center that seldom performs prostate MRI examinations or in a tertiary center where only expert radiologists interpret these examinations.

If this hypothetical product were to include additional features such as volumetric assessment and structured reporting, it could significantly reduce reporting time and improve interdisciplinary communication.^34,35 Thus, the product could also become invaluable in the tertiary center only staffed by expert prostate radiologists. Additionally, it could enhance the performance of less-experienced radiologists in the center who infrequently perform prostate MRI examinations, potentially enabling these centers to expand their patient base into prostate imaging.³⁶

While this example is hypothetical, it mirrors the complexity and diversity of real-world scenarios in radiology departments. It serves as a prompt for radiologists and administrators to consider a wide array of factors in their decision-making process, ensuring that the AI solutions they adopt truly enhance their practice and patient care.

This scenario underscores several critical considerations for radiology departments when selecting an AI solution: scope of the product, intended end-users, product features and outputs, and potential clinical benefits. Further details are presented in Table 1.

Performance and validation

The evaluation of diagnostic performance in AI models is led by established guidelines, such as the Consolidated Standards of Reporting Trials-Artificial Intelligence and the Checklist for Artificial Intelligence in Medical Imaging.^37,38 These standards advocate best practices in assessing AI models, necessitating detailed information about the AI model. This includes model architecture, training strategy, hyperparameters, unique additions to the base model, overfitting avoidance techniques, and explainability mechanisms. In addition, comprehensive data details, such as size, demographics, scanner types, potential biases, preprocessing methods, and augmentation techniques, are crucial.

In research contexts, where models are trained, validated, and tested, this exhaustive information is typically available for review and audit. This transparency facilitates the assessment of AI models against performance and transparency standards.

However, in clinical settings, AI solutions are often provided by commercial firms or startups, which may be hesitant to disclose in-depth information about their technology or data due to proprietary concerns and commercial sensitivities.³⁹

To illustrate these challenges, let us consider a hypothetical scenario: imagine an AI solution designed for detecting large vessel occlusion (LVO) in computed tomography (CT) angiography. This product, boasting a diagnostic accuracy of 88% on an external dataset from two hospitals and 24,355 patients, is marketed for its rapid assessment capability, alerting physicians within 5 minutes of image receipt.

At first glance, this seems promising; however, critical information is often missing. For example:

- The diagnostic accuracy alone offers an incomplete performance picture; additional metrics, such as sensitivity, specificity, and negative and positive predictive values scores, are necessary for a comprehensive understanding.

- The prevalence of LVO in the study population and the patient demographics should be clarified.

- Details on the model’s ability to detect distal or posterior system LVOs and whether these cases were included in the performance metrics are crucial.

- Information on whether the study was retrospective or if the model was tested in real-time clinical settings is vital.

- Independent research using the software, potential conflicts of interest among authors, and the nature of the research are important considerations.

Radiologists may also seek real-life performance evidence from peers who have used similar products or may request the company to demonstrate the product’s efficacy with their own cases.

This scenario underscores several critical considerations regarding the performance and validation for radiology departments when selecting an AI solution: evidence of external validation, comprehensive performance metrics, real-world performance data, and pilot testing. These are summarized in Table 1.

Although the “performance and validation” is crucial, as discussed above, we also acknowledge the difficulty of rigorously evaluating the diagnostic performance of AI models, particularly in high-demand radiological environments. The practical challenges of conducting such evaluations in busy practices cannot be overlooked, given that not every radiology department has the necessary infrastructure or resources for comprehensive, independent assessments of AI solutions.

This recognition extends to well-established and trusted firms in our sector, including picture archiving and communication systems (PACS) and hardware providers, who have already begun offering validated products from third-party vendors (e.g., startups that are relatively new compared with the sector’s stalwarts).^40,41 Such initiatives by trusted firms and collaborations between radiology departments and these entities are crucial. They enable radiology departments to leverage the expertise and resources of established providers, streamlining the process and ensuring that the evaluation of AI solutions meets rigorous and reliable standards. This collaborative approach not only reduces the workload on individual departments but also facilitates a more efficient and effective integration of AI technologies into radiological practices.

Implementation and integration

A critical, yet often overlooked, aspect of AI products is the ease of integration into the existing workflow of a radiology department. It is essential that this integration causes minimal-ideally no-disruption. Determining who will perform the implementation is equally important. Additionally, choosing between cloud-based or on-premises solutions, and considering the necessary computational power, are key elements. These latter factors are often more apparent to potential buyers of radiology AI solutions.

A recent survey by the Dutch Society of Radiology highlights implementation challenges as a major obstacle to broader AI adoption in radiology clinics.⁴ The implementation process, though seemingly straightforward, is in fact complex and multifaceted, requiring collaboration between radiologists, IT specialists, software engineers, and hospital administrators to ensure accurate and safe integration.⁴²

Hospitals vary in their infrastructure, expertise, and PACS capabilities. Many experts emphasize the need to build a suitable infrastructure for the seamless integration of machine learning-based applications in radiology clinics.^43,44 This infrastructure development requires substantial effort and time and extends beyond the sole responsibility of radiologists. However, radiologists should be aware of their current capabilities and seek products that align well with their existing infrastructure.

From our perspective, the burden of ensuring smooth integration should not rest solely on radiologists. AI solution developers for radiology must provide flexible, easily integrable products suitable for a range of infrastructures, from basic to advanced. They should actively participate in the integration process, alleviating the strain on IT resources and facilitating the adoption of their products.⁴⁵

Here, we construct a hypothetical scenario to illustrate the complexity of implementation and the factors radiologists should consider when purchasing an AI product. Consider a small team of radiologists from a mid-sized department who are evaluating an AI solution for chest X-ray analysis, driven by an inability to keep pace with high imaging demand. Having successfully identified their clinical needs and evaluated the diagnostic performance of the solution, they now face several implementation challenges.

First, system compatibility issues arise, as the AI tool is not fully compatible with their existing PACS system, necessitating system modifications or additional modules from the AI vendor. This adds complexity to the implementation process. Additionally, dedicated IT personnel are needed for the setup and integration, but the department’s IT resources are already overstretched and are raising concerns about managing this extra workload. After buying the product, the radiologists realize that the vendor of the software provided implementation via a third party in their country, which does not have the necessary expertise or knowledge regarding the product.

As the implementation process progresses, the radiology department encounters additional challenges related to hardware requirements. The AI tool, being advanced in its capabilities, requires a robust computational setup, including high-performance graphics processing units, to efficiently process the imaging data. The department’s existing hardware infrastructure is not equipped to handle such demanding computational tasks, leading to the necessity of significant hardware upgrades.

This example shows the importance and complexity of implementation and what may go wrong and cause frustrations during the implementation. Unfortunately, such a hypothetical scenario is not uncommon and underscores the need for a thorough evaluation of various factors beyond clinical utility and performance when implementing an AI solution in a radiology department.⁴⁶

Detailed explanations of critical considerations concerning the implementation and integration of AI solutions for radiology departments are provided in Table 1, categorized under the subheadings of vendor compatibility, availability of IT resources, support for implementation, hardware requirements, and workflow integration.

Apart from the key factors discussed above, we acknowledge that the challenges of implementing AI in radiology, notably the communication gap between AI developers and PACS administrators, alongside the necessity for interface customization, are well recognized. In addressing these, the emergence of commercial platforms, including AI application marketplaces, offers a promising solution.^47-50 These platforms enhance the integration process, providing standardized frameworks and addressing customization needs, thereby supporting productivity improvements in radiological practices. Their role in facilitating effective communication and streamlined implementation is crucial for the seamless adoption of AI technologies in radiology.

Clinical usability

The widespread adoption and success of AI in radiology depend on several factors, notably how radiologists interact with AI systems. While no universally established best practices exist for designing effective AI tools in radiology, lessons can be learned from successful technology products in other industries. A common feature of such products is their simplicity, offering benefits in the most straightforward manner possible.

Simplicity and user-centric design are essential. AI tools that are easy to use and feature intuitive interfaces are more likely to be adopted by radiologists, facilitating their integration into daily practice. Our experience and literature review indicate that simplicity in design and operation is crucial for the effectiveness and acceptance of AI tools in clinical settings.^23,24

Consider a hypothetical scenario highlighting the importance of simplicity: radiologists at a tertiary center specializing in musculoskeletal imaging purchase an AI solution for automated fracture evaluation in plain radiographs. This high-performing product integrates well with the department’s infrastructure and PACS. Radiologists interact with the tool through a hyperlink in their patient browser, which opens a graphical user interface for running the AI engine and viewing results.

While this software may be suitable for low-throughput examinations such as prostate or cardiac imaging, its use becomes problematic for high-volume radiographic examinations, particularly for time-pressed radiologists. The repetitive interaction, taking a few minutes per case, adds up significantly over the day. A more efficient solution would be an AI tool that processes steps automatically and presents results directly in PACS, reducing the need for additional interactions.

It is noteworthy that many products in the market are even more difficult to use, as they require radiologists to switch from their PACS to open the AI program or even change computers. Purchasing a product not integrated into PACS and having a complex, intrusive interface can be more burdensome and time-consuming than manually examining the image.

We suggest that a successful AI solution must be fully integrated with PACS, featuring a user-friendly interface that can be used with minimal mouse clicks, or even without any clicks in a fully automated fashion for high-volume examinations. Such a solution should immediately distinguish pathological cases from normal ones with binary classification, ideally identify priority cases with acute findings, and highlight them on the radiologist’s worklist for triage. Such a product would effectively serve as a second eye for the radiologist, exemplifying the vital role of simplicity in the design and functionality of AI tools in radiology.

In this context, radiologists should evaluate the simplicity of a radiology solution as follows: integration with a PACS system, user interface, efficiency of use, automated processing and result presentation, alignment with workflow and case types, and outputs (Table 1).

Costs and return on investment

According to the Dutch Society of Radiology, the primary obstacle to AI adoption in radiology clinics is cost.⁴ Implementing AI solutions in radiology departments involves significant financial investments. In most healthcare systems, AI solutions are not generally covered by government or private insurance policies, with a few exceptions in the United States, as indicated by the expanding coverage of certain AI products under Medicare or Medicaid.⁵¹ This supports the findings of the Dutch Society of Radiology survey, which highlights the crucial role of cost in AI solution adoption.

AI solutions are presently available with various purchasing models, including lifelong licensing, annual licenses, and per-use fees. Radiologists must carefully consider which model best suits their department’s needs. Many AI startups favor subscription-based models over outright capital investments in contrast to traditional medical hardware or software manufacturers.²⁰

Determining the most suitable purchasing model is a critical task that requires careful evaluation. For example, consider a radiology department that performs 10,000 mammography examinations each year and is considering adopting software to perform breast cancer evaluation for mammography screening. Suppose this software is available with an annual licensing fee of $20,000, inclusive of hardware and service costs. Although this may be a familiar purchasing approach for many radiologists, it is not necessarily the most cost-effective option.

Now consider an alternative purchasing model where the software is available with an initial installation fee of $2,000, plus a per-examination fee of $1. In this scenario, the department may realize significant cost savings. With 10,000 examinations a year, the total cost would be $12,000 ($2,000 installation plus $10,000 for examinations), substantially lower than the flat annual license fee. This example underscores the importance of thoroughly examining and comparing different purchasing models to identify the most economically viable option for the department.

When evaluating the costs of AI solutions in radiology, it is equally important to consider the ROI. This analysis assesses whether the AI solution will provide sufficient value and efficiency improvements to justify its expenses. While ROI analysis is more straightforward for algorithms that speed up MRI examinations or reduce CT radiation doses, it becomes complex for AI solutions aimed at medical image interpretation.⁵² This complexity arises from the challenge of quantifying intangible benefits.

For hospital or radiology administrators, financial viability often hinges on ROI, which can be a major obstacle, particularly in the absence of insurance reimbursements for radiology AI software.⁵³ The evaluation of ROI encompasses two aspects: quality and efficiency improvement. Quality improvement, such as enhanced diagnostic accuracy or error reduction, can be difficult to translate into financial ROI due to its complexity and reliance on assumptions. Efficiency improvements, however, can be more directly measured by time savings in study read times using AI software. Saved person-hours provide a tangible way to demonstrate the software’s value, making it particularly appealing in competitive markets where efficiency in repetitive tasks is prized.⁵⁴

Although calculating the exact ROI can be challenging or even impossible for radiology departments or hospital managers, it remains a critical exercise. This challenge is comparable with the difficulty in demonstrating the financial benefits of national breast cancer screening programs, which require extensive time and data across numerous cases.⁵⁵ Consequently, it may not be feasible for a radiology department to precisely determine the ROI of an AI solution for breast cancer detection in mammography.

Despite this, radiologists should still evaluate potential financial benefits and roughly estimate the ROI, while also considering other key factors, such as clinical relevance, performance, implementation, and simplicity. If a department has the necessary budget or means to cover the product’s costs, and there is an expectation of benefits, such as reduced workload or diagnostic errors, potentially supported by scientific evidence, it may still be justifiable to invest in an AI solution without exact ROI calculations.

In this context, radiologists should evaluate the costs and ROI of radiology solutions as follows: financial investment evaluation, insurance reimbursement, purchasing models comparison, ROI analysis, and potential benefits estimation (Table 1).

In addition to the considerations discussed above, it is important to acknowledge the geographical variability in practices related to the cost and reimbursement for AI solutions in radiology. For example, in the United States, Medicare offers reimbursement for certain medical imaging AI solutions, such as those related to stroke imaging, as well as candidate applications, such as pulmonary embolism and subdural hematoma.⁵¹ However, even these policies are subject to change, being valid for defined periods with uncertain futures regarding their continuation. Therefore, it is essential to stress that reimbursement policies for AI solutions are rapidly evolving. While it may be possible to describe the current state of affairs, these conditions are likely to change over time, making it challenging to provide a static overview that remains accurate.

Regulations, security, and privacy

Security and privacy are paramount in healthcare and radiology departments, particularly when considering the sensitive nature of health and imaging data. The rapid integration of AI solutions into radiology necessitates robust security and privacy measures. AI solutions, whether on-premises or cloud-based, involve integration with various data sources, such as health information systems (HIS), research information systems (RIS), and PACS, encompassing both imaging and other data processing tasks. The sensitive nature of medical data, including patient images, diagnostic information, and personally identifiable information, requires stringent data protection mechanisms. Radiology departments, being central hubs of medical data, AI solutions, and caregivers, must adhere to strict security standards and regulations. In 2021, Expert Insights from Health Devices identified AI in radiology as a major healthcare risk.⁵⁶

Medical device regulations vary globally but typically align with major standards, such as the FDA in the United States²⁵ or the European Union’s CE marking.²⁶ Health data protection standards, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States or the General Data Protection Regulation (GDPR) in Europe,^27,28 are also critical. Additionally, national standards, such as Türkiye’s “Kişisel Veri Koruma Kanunu” (Personal Data Protection Act) have similarities with GDPR and HIPAA but also include unique requirements, such as mandating that healthcare providers keep health data within the country’s borders, irrespective of anonymization.²⁹ Furthermore, the FDA and CE markings encompass responsibilities regarding patient data integrity and cybersecurity.^57,58

However, it is important to note that addressing regulations, security, and privacy in the context of AI acquisition is a complex task. It involves not only radiologists but also hospital managers, IT specialists, device providers, and legal experts specializing in medical devices. Consider a scenario in which a leading radiology department in Europe plans to implement an AI system for enhanced brain MRI analysis. The department is known for its innovative approach and diverse patient base and must navigate complex regulatory and security challenges. The AI system, developed by a global technology company, claims compliance with stringent regulations, including the FDA standards in the United States, the European Union’s CE marking, and the HIPAA and GDPR frameworks.

The integration of the device involves connecting with existing HIS, RIS, and PACS systems. It is declared compliant with international standards and incorporates data security measures, encryption protocols, and access controls. The radiologists conduct evaluations with the manufacturer, focusing on the “CIA triad”: confidentiality, integrity, and availability,⁵⁹ which includes robust authentication protocols, regular security audits, and keeping up with cybersecurity best practices.⁶⁰ A decision is made to initiate a demonstration integration of the product.

However, initial document checks by the information technology (IT) department reveal that the device complies with HIPAA but not GDPR requirements and runs on the cloud, not accommodating the local data storage rule of their country. Discussions with the manufacturer reveal that they are close to finalizing GDPR registration, as evidenced by their documents. When the hospital IT team and the manufacturer assess the possibility of running the device on cloud servers within the country’s borders, they find no dedicated hardware available for running this advanced algorithm locally. Moreover, the legal team, upon reviewing the contract proposed by the manufacturer, discovers a clause allowing the company to use the hospital’s data to improve their product, which is firmly opposed by the hospital administrators.

Despite initial intentions, the process brings many challenges and ends in disappointment, highlighting the complexities and importance of considering regulations, security, and privacy. This scenario underscores the need for involvement from legal experts and IT personnel in the early stages and the meticulous assessment of these key areas.

Table 1 offers additional elaboration with subsections including compliance with international standards, adherence to data protection laws, data security measures, local data storage capabilities, vendor data usage policies, and trial integration and IT assessment.

These items provide a comprehensive checklist to guide radiologists and hospital administrators in ensuring that the integration of AI solutions into radiology practices complies with necessary regulations and maintains the highest standards of security and privacy.

Final thoughts and conclusions

In this review-opinion article, we first provided an overview of the current AI solutions for radiology and discussed key factors to consider when choosing appropriate AI solutions for radiology departments. We mainly focused on AI solutions aiming at carrying out interpretative tasks, which routinely necessitate the expertise and active participation of radiologists.

Although we did not and could not cover every aspect of choosing the right solution for a radiology department, and key factors and their content may rapidly change due to the fast pace of developments of AI technologies, throughout the paper, we attempted to simplify the concepts with hypothetical examples, drawing on our 5 years of clinical, industrial, and academic experience and the existing literature. Furthermore, we provided a checklist consisting of a set of questions and/or items for each criterion, which radiologists may quickly check before starting discussions with AI providers. This will help them make well-informed decisions about integrating AI solutions into their practice, thus contributing significantly to the advancement of radiological services and patient care.

In addition to the considerations outlined above, we recommend the establishment of a departmental review (or assessment) board specifically for the procurement of AI solutions. This board should ideally comprise a multidisciplinary team, including institutional IT officials, legal counsel, end-users, and administrative officials. The creation of such a board facilitates a structured and comprehensive evaluation process, ensuring that the selected AI solutions align with the strategic goals of the department, adhere to legal and ethical standards, and meet the practical needs of end-users. The establishment of these boards, as observed in our experiences and those of our international colleagues, may represent a proactive step towards embracing the complexities and opportunities presented by AI in radiology.

Conflict of interest disclosure

Deniz Alis is the CEO and co-founder of Hevi AI Health Tech. The authors declared no conflicts of interest.

Funding

This paper has been produced benefiting from the 1001 Science and Technology Grant Program National Program of TUBITAK (project no: 122E022). However, the entire responsibility of the publication/paper belongs to the owner of the paper. The financial support received from TUBITAK does not mean that the content of the publication is approved in a scientific sense by TUBITAK.

References

Miller DD, Brown EW. Artificial intelligence in medical practice: the question to the answer? Am J Med. 2018;131(2):129-133.

Chartrand G, Cheng PM, Vorontsov E, et al. Deep learning: a primer for radiologists. Radiographics. 2017;37(7):2113-2131.

Rodríguez-Ruiz A, Krupinski E, Mordang JJ, et al. Detection of breast cancer with mammography: effect of an artificial intelligence support system. Radiology. 2019;290(2):305-314.

van Leeuwen KG, de Rooij M, Schalekamp S, van Ginneken B, Rutten MJCM. Clinical use of artificial intelligence products for radiology in the Netherlands between 2020 and 2022. Eur Radiol. 2024;34(1):348-354.

Allen B, Agarwal S, Coombs L, Wald C, Dreyer K. 2020 ACR data science institute artificial intelligence survey. J Am Coll Radiol. 2021;18(8):1153-1159.

Coppola F, Faggioni L, Regge D, et al. Artificial intelligence: radiologists’ expectations and opinions gleaned from a nationwide online survey. Radiol Med. 2021;126(1):63-71.

Agrawal A, Khatri GD, Khurana B, Sodickson AD, Liang Y, Dreizin D. A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations. Emerg Radiol. 2023;30(3):267-277.

AI for Radiology. Accessed November 19, 2023.

Health C for D and R. Artificial intelligence and machine learning (AI/ML)-enabled medical devices. FDA. Published online October 20, 2023. Accessed November 19, 2023.

Heuvelmans MA, van Ooijen PMA, Ather S, et al. Lung cancer prediction by deep learning to identify benign lung nodules. Lung Cancer. 2021;154:1-4.

Karagoz A, Alis D, Seker ME, et al. Anatomically guided self-adapting deep neural network for clinically significant prostate cancer detection on bi-parametric MRI: a multi-center study. Insights into Imaging. 2023;14(1):110.

Park HJ, Park B, Lee SS. Radiomics and deep learning: hepatic applications. Korean J Radiol. 2020;21(4):387-401.

Sandino CM, Cole EK, Alkan C, et al. Upstream machine learning in radiology. Radiol Clin North Am. 2021;59(6):967-985.

Bizzo BC, Almeida RR, Michalski MH, Alkasab TK. Artificial Intelligence and clinical decision support for radiologists and referring providers. J Am Coll Radiol. 2019;16(9 Pt B):1351-1356.

van Leeuwen KG, de Rooij M, Schalekamp S, van Ginneken B, Rutten MJCM. How does artificial intelligence in radiology improve efficiency and health outcomes? Pediatr Radiol. 2022;52(11):2087-2093.

Tariq A, Purkayastha S, Padmanaban GP, et al. Current clinical applications of artificial intelligence in radiology and their best supporting evidence. J Am Coll Radiol. 2020;17(11):1371-1381.

Tanguay W, Acar P, Fine B, et al. Assessment of radiology artificial intelligence software: a validation and evaluation framework. Can Assoc Radiol J. 2023;74(2):326-333.

Tadavarthi Y, Vey B, Krupinski E, et al. The state of radiology AI: considerations for purchase decisions and current market offerings. Radiol Artif Intell. 2020;2(6):e200004.

Filice RW, Mongan J, Kohli MD. Evaluating artificial intelligence systems to guide purchasing decisions. J Am Coll Radiol. 2020;17(11):1405-1409.

Omoumi P, Ducarouge A, Tournier A, et al. To buy or not to buy-evaluating commercial AI solutions in radiology (the ECLAIR guidelines). Eur Radiol. 2021;31(6):3786-3796.

European Society of Radiology (ESR). What the radiologist should know about artificial intelligence - an ESR white paper. Insights Imaging. 2019;10(1):44.

Wamba-Taguimdje SL, Fosso Wamba S, Kala Kamdjoug JR, Tchatchouang Wanko CE. Influence of artificial intelligence (AI) on firm performance: the business value of AI-based transformation projects. Business Process Management Journal. 2020;26(7):1893-1924.

Lam Shin Cheung J, Ali A, Abdalla M, Fine B. U”AI” testing: user interface and usability testing of a chest X-ray AI tool in a simulated real-world workflow. Can Assoc Radiol J. 2023;74(2):314-325.

Combi C, Amico B, Bellazzi R, et al. A manifesto on explainability for artificial intelligence in medicine. Artif Intell Med. 2022;133:102423.

Commissioner O of the. U.S. Food and Drug Administration. FDA. Published December 4, 2023. Accessed December 9, 2023.

CE marking. Accessed December 9, 2023.

Health | European Data Protection Supervisor. Accessed December 9, 2023.

Health Insurance Portability and Accountability Act of 1996 (HIPAA) | CDC. Published June 28, 2022. Accessed December 9, 2023.

Kişisel Verileri Koruma Kurumu | KVKK | Kişisel Verileri Koruma Kurumu Başkanlığı. Accessed December 9, 2023.

Martín Noguerol T, Paulano-Godino F, Martín-Valdivia MT, Menias CO, Luna A. Strengths, weaknesses, opportunities, and threats analysis of artificial intelligence and machine learning applications in radiology. J Am Coll Radiol. 2019;16(9 Pt B):1239-1247.

Hwang EJ, Park CM. Clinical Implementation of deep learning in thoracic radiology: potential applications and challenges. Korean J Radiol. 2020;21(5):511-525.

Zhang T, Tan T, Samperna R, et al. Radiomics and artificial intelligence in breast imaging: a survey. Artif Intell Rev. 2023;56(Suppl 1):857-892.

Turkbey B, Rosenkrantz AB, Haider MA, et al. Prostate imaging reporting and data system version 2.1: 2019 update of prostate imaging reporting and data system version 2. Eur Urol. 2019;76(3):340-351.

Soerensen SJC, Fan RE, Seetharaman A, et al. Deep learning improves speed and accuracy of prostate gland segmentations on magnetic resonance imaging for targeted biopsy. J Urol. 2021;206(3):604-612.

Wetterauer C, Winkel DJ, Federer-Gsponer JR, et al. Structured reporting of prostate magnetic resonance imaging has the potential to improve interdisciplinary communication. PLoS One. 2019;14(2):e0212444.

Labus S, Altmann MM, Huisman H, et al. A concurrent, deep learning-based computer-aided detection system for prostate multiparametric MRI: a performance study involving experienced and less-experienced radiologists. Eur Radiol. 2023;33(1):64-76.

Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK; SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit Health. 2020;2(10):537-e548.

Mongan J, Moy L, Kahn CE Jr. Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell. 2020;2(2):e200029.

Clark P, Kim J, Aphinyanaphongs Y. Marketing and US Food and Drug Administration Clearance of Artificial Intelligence and Machine Learning Enabled Software in and as Medical Devices: a systematic review. JAMA Network Open. 2023;6(7):e2321792.

Lunit Expands Collaboration with GE Healthcare to Advance AI Adoption across Healthcare Industry. Accessed March 12, 2024.

Lunit AI Solutions to Power Samsung’s X-ray Devices for Advanced Chest Screening. Accessed March 12, 2024.

Vandewinckele L, Claessens M, Dinkla A, et al. Overview of artificial intelligence-based applications in radiotherapy: Recommendations for implementation and quality assurance. Radiother Oncol. 2020;153:55-66.

Bassett M. The reality of deep learning/artificial intelligence in radiology: they will redefine the specialty. Accessed November 25, 2023.

Dikici E, Bigelow M, Prevedello LM, White RD, Erdal BS. Integrating AI into radiology workflow: levels of research, production, and feedback maturity. J Med Imaging (Bellingham). 2020;7(1):016502.

Gleeson F, Revel MP, Biederer J, et al. Implementation of artificial intelligence in thoracic imaging-a what, how, and why guide from the European Society of Thoracic Imaging (ESTI). Eur Radiol. 2023;33(7):5077-5086.

Strohm L, Hehakaya C, Ranschaert ER, Boon WPC, Moors EHM. Implementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factors. Eur Radiol. 2020;30(10):5525-5532.

Sectra Amplifier Marketplace - Easily navigate the ecosystem of AI applications. Sectra Amplifier Marketplace. Accessed March 12, 2024.

Edison Artificial Intelligence & Analytics. Accessed March 12, 2024.

Blackford Analysis - Your strategic AI platform partner. Accessed March 12, 2024.

deepcOS - The Radiology AI Operating System. Accessed March 12, 2024.

Parikh RB, Helmchen LA. Paying for artificial intelligence in medicine. NPJ Digit Med. 2022;5(1):63.

Lee MH, Schemmel AJ, Pooler BD, et al. Radiology workflow dynamics: how workflow patterns impact radiologist perceptions of workplace satisfaction. Acad Radiol. 2017;24(4):483-487.

Reimbursement for AI in Radiology: Current Practices and Future Considerations. Accessed November 25, 2023.

Liew C. The future of radiology augmented with artificial intelligence: a strategy for success. Eur J Radiol. 2018;102:152-156.

Rim SH, Allaire BT, Ekwueme DU, et al. Cost-effectiveness of breast cancer screening in the National Breast and Cervical Cancer Early Detection Program. Cancer Causes Control. 2019;30(8):819-826.

Top 10 Health Technology Hazards for 2023 Executive Brief. ECRI. Accessed December 9, 2023.

Harrington SG, Johnson MK. The FDA and artificial intelligence in radiology: defining new boundaries. J Am Coll Radiol. 2019;16(5):743-744.

Harvey HB, Gowda V. How the FDA Regulates AI. Acad Radiol. 2020;27(1):58-61.

Jennett P, Watanabe M, Igras E, Premkumar K, Hall W. Telemedicine and security. Confidentiality, integrity, and availability: a Canadian perspective. Stud Health Technol Inform. 1996;29:286-298.

Shah C, Nachand D, Wald C, Chen PH. Keeping patient data secure in the age of radiology artificial intelligence: cybersecurity considerations and future directions. J Am Coll Radiol. 2023;20(9):828-835.

Choosing the right artificial intelligence solutions for your radiology department: key factors to consider