Can better design of deep learning studies lead to the faster transformation of medical practices? According to the authors of “Designing deep learning studies in cancer diagnostics,” published in Nature Reviews Cancer‘s latest issue, the answer is yes.

“We propose several protocol items that should be defined before evaluating the external cohort” says first author Andreas Kleppe at the Institute for Cancer Diagnostics and Informatics at Oslo University Hospital.”

“In this way, the evaluation becomes rigorous and more reliable. Such evaluations would make it much clearer which systems are likely to work well in clinical practice, and these systems should be further assessed in phase III randomized clinical trials.”

Slow implementation is partly a natural consequence of the time needed to evaluate and adapt systems affecting patient treatment. However, many studies assessing well-functioning systems are at high risk of bias.

According to Kleppe, even among the seemingly best studies that evaluate external cohorts, few predefine the primary analysis. Adaptations of the deep learning system, patient selection or analysis methodology can make the results presented over-optimistic.

The frequent lack of stringent evaluation of external data is of particular concern. Some systems are developed or evaluated on too narrow or inappropriate data for the intended medical setting. The lack of a well-established sequence of evaluation steps for converting promising prototypes into properly evaluated medical systems limits deep learning systems’ medical utilization.

Millions of adjustable parameters

Deep learning facilitates utilization of large data sets through direct learning of correlations between raw input data and target output, providing systems that may use intricate structures in high-dimensional input data to model the association with the target output accurately. Whereas supervised machine learning techniques traditionally utilized carefully selected representations of the input data to predict the target output, modern deep learning techniques use highly flexible artificial neural networks to correlate input data directly to the target outputs.

The relations learnt by such direct correlation will often be true but may sometimes be spurious phenomena exclusive to the data utilized for learning. The millions of adjustable parameters make deep neural networks capable of performing correctly in training sets even when the target outputs are randomly generated and, therefore, utterly meaningless.

Design and evaluation challenges

The high capacity of neural networks induces severe challenges for designing and developing deep learning systems and validating their performance in the intended medical setting. An adequate clinical performance will only be possible if the system has good generalisability to subjects not included in the training data.

The design challenges involve selecting appropriate training data, such as representativeness of the target population. It also includes modeling questions such as how the variation of training data may be artificially increased without jeopardizing the relationship between input data and target outputs in the training data.

The validation challenge includes verifying that the system generalizes well. For example, does it perform satisfactorily when evaluated on relevant patient populations at new locations and when input data are obtained using differing laboratory procedures or alternative equipment? Moreover, deep learning systems are typically developed iteratively, with repeated testing and various selection processes that may bias results. Similar selection issues have been recognized as a general concern for the medical literature for many years.

Thus, when selecting design and validation processes for diagnostic deep learning systems, one should focus on the generalization challenges and prevent more classical pitfalls in data analysis.

“To achieve good performance for new patients, it is crucial to use various training data. Natural variation is always essential, but so is introducing artificial variation. These types of variation complement each other and facilitate good generalisability,” says Kleppe.

The post Designing and evaluating medical deep learning systems appeared first on Artificial Intelligence.

Artificial Intelligence and Deep Learning are being used to solve some of the world’s biggest problems and is finding application in autonomous driving, marketing and advertising, health and medicine, manufacturing, multimedia and entertainment, financial services, and so much more.  This is made possible by incredible advances in a wide range of technologies, from computation to interconnect to storage, and innovations in software libraries, frameworks, and resource management tools.  While there are many critical challenges, an open technology approach provides significant advantages.

The Scaling Challenge

The full deep learning story, though, must be an end-to-end technology discussion and encompass production at scale.  As we scale out deep learning workloads to the massive compute clusters required to tackle these big issues, we begin to run into the same challenges that hamper scaling of traditional high-performance computing (HPC) workloads.

Ensuring optimal use of compute resources can be challenging, particularly in heterogeneous architectures that may include multiple central processing unit (CPU) architectures, such as x86, ARM64, and Power, as well as accelerators, such as graphical processing units (GPUs), field programmable gate arrays (FPGAs), tensor processing units (TPUs), etc. Architecting an optimal deep learning solution for training or inferencing, with potentially varied data types, can result in the application of one or more of these architectures and technologies. The flexibility of open technologies allows one to deploy the optimal platform at server, rack, and data center scales.

One of the most important uses of deep learning is in gaining value from large data sets. The need to effectively manage large amounts of data, which may have varying ingest, processing, and persistent storage and data warehouse needs, is at the center of a modern deep learning solution. The performance requirements throughout the data workflow and processing stages can vary greatly, and, at production-scale, it can simultaneously involve data collection, training, and inference.  The balance of cost effectiveness and high performance is key to providing a properly-scaled deployment. The flexibility of open technologies, allows one to take a software-designed data center approach to the deep learning environment.

Workload orchestration is another familiar challenge in the HPC realm.  A variety of tools and libraries have been developed over the years, including resource managers and job schedulers, parallel programming libraries, and other software frameworks.  As software applications have grown in complexity, with rapidly evolving dependencies, a new approach has been needed.  One such approach is containerization.  Containers allow applications to be bundled with their dependencies and deployed on a variety of compute hosts.  However, challenges have remained for providing access to compute, storage, and other resources.  Moreover, managing the deployment, monitoring, and clean-up of containerized applications presents its own set of challenges.

The Open Technology Approach

Penguin Computing applies its decades of expertise in high-performance and scale-out computing to deliver deep learning solutions that support customer workload requirements, whether at development or production scales.  Penguin Computing solutions feature open technologies, enabling design choices that focus on meeting the customer’s needs.

In the Penguin Computing AI/DL whitepaper, you will learn more about our approach to:

• Open Architectures for Artificial Intelligence and Deep Learning, combining flexible compute architectures, rack scale platforms, and software-defined networking and storage, to provide a scalable software-defined AI/DL environment.
• Discuss AI/DL strategies, providing insight into everything from specialty compute for training vs. inference to Data Lakes and high performance storage for data workflows to orchestration and workflow management tools.
• Deploying the AI/DL environments from development to production scale and from on-premise to hybrid to public cloud.

The post Designing the Future of Deep Learning appeared first on Artificial Intelligence.