Main content

AI-enhanced instrumentation - the fusion of deep learning and medical sensors creates dramatic improvements

Kurt Marko Profile picture for user kmarko April 17, 2018
AI-enhanced instrumentation - an exciting area of research that illustrates the promise of AI augmentation.

medical scan
Part of what fuels societal angst about the use of increasingly sophisticated, quasi-autonomous algorithms collectively known as AI is the fear that machines will supplant people in an ever-growing number of jobs.

While this is undoubtedly true and has been for every prior generation of new technology from the power loom to the backhoe, the more significant question is whether AI replaces or merely displaces people?  Namely, does it lead to the unemployed or to the differently employed.

Medicine is one of the most promising areas of AI development and as I discussed last week, in hospitals and clinics there's a good argument that deep learning (DL) will augment human expertise, not supersede it.

Ground zero of the DL disruption in medicine is imaging, however as my column pointed out, experts are optimistic that AI will make for better radiologists, not fewer. Indeed, in developing countries with a severe shortage of specialists, algorithms will enable overworked radiologists to lower their stress levels and error rates.

An exciting area of research that illustrates the promise of AI augmentation is what I call AI-enhanced instrumentation. The concept is similar to techniques like high-dynamic range (HDR) photography, digital remastering of recordings or even film colorization in that one or more original sources of data are post-processed and enhanced to bring out added detail, remove noise or improve aesthetics. When applied to radiology and pathology, the concept entails taking either real-time or recorded images and processing them using DL algorithms and possibly CGI (like ray tracing, etc.) to produce an enhanced image that highlights features of interest like cancer cells or creates a more photorealistic, lifelike result.

Nvidia Project Clara

Nvidia unveiled Project Clara at its recent GTC conference (see my overview here), showing early results using DL post-processing to dramatically enhance existing, often grainy and indistinct echocardiograms (sonograms of the heart).

Although Nvidia is still short on details, the project appears to be a cloud-based “medical imaging supercomputer” that can be simultaneously used by multiple researchers to develop enhancement algorithms for any medical instrument including CT, MRI, sonogram or conventional X-ray. The early results are quite amazing as the company showed a 3D ultrasound of the heart (echocardiogram) that had been enhanced to clearly show the left ventricle.

Clara should accelerate research being done on several fronts that exploits explosive growth in DL computational capability to perform analysis that was previously impossible or far too costly. One technique is called 3D volumetric segmentation that can accurately measure the size of organs, tumors or fluids, such as the volume of blood flowing through arteries and the heart. NVIDIA claims that a recent implementation, an algorithm called V-Net, “would’ve needed a computer that cost $10 million and consumed 500 kW of power.

Today, it can run on a few Tesla V100 GPUs.” For reference, the DGX-2 systems NVIDIA unveiled at GTC consist of 16 tightly-coupled V100s, consuming 10 kW and costs about $400,000. Presumably, Clara will aggregate dozens of such systems into a service that can accommodate scores of researchers developing similarly complex algorithms.

Spanning medical specialities

AI enhancement of echocardiograms was the feature application at GTC, however there are many other research avenues that illustrate the promise of AI image enhancement. Given the extensive use of CT and MRI scans, one of the most broadly applicable developments is the AUTOMAP algorithm from Massachusetts General Hospital, a new system using GPU-powered neural networks for processing and enhancing a variety of electromagnetic imaging systems. The approach not only improves image quality, but reduces the time required to process MRI images along with the dosage necessary for useful X-Ray, CT and PET scans. According to the senior researcher on the project (emphasis added):

Since AUTOMAP is implemented as a feedforward neural network, the speed of image reconstruction is almost instantaneous – just tens of milliseconds. Some types of scans currently require time-consuming computational processing to reconstruct the images. In those cases, immediate feedback is not available during initial imaging, and a repeat study may be required to better identify a suspected abnormality. AUTOMAP would provide instant image reconstruction to inform the decision-making process during scanning and could prevent the need for additional visits.

Diagnosis of bone and joint problems is another area ripe for AI enhancement. For example, detecting the extent and severity of osteoarthritis is typically done via a combination of two-dimensional X-rays and an interpretation of the patient’s symptoms and medical history. A new technique from ImageBiopsy Lab uses a combination of computer vision and DL to build three-dimensional representations of 2D images. NVIDIA seeded the startup as part of its Inception program and notes that IB Lab trained is algorithms on over 150,000 radiographs, so doctors can receive accurate measurements of the areas around the bones in the knees. The results can indicate the severity of the patient’s osteoarthritis without any further processing needed.

According to a paper IB Lab is presenting at the World Congress on Osteoporosis, its predictive algorithm "In combination with the joint space width and area, the prediction worked significantly better than using joint space measures and clinical features alone.

Google researchers have combined DL with augmented reality (AR) to enhance images from microscopes in real time to help pathologists better interpret and diagnose tissue samples. They described a prototype AR microscope in a paper presented at the recent meeting of the American Association for Cancer Research (AACR) that "consists of a modified light microscope that enables real-time image analysis and presentation of the results of machine learning algorithms directly into the field of view." Significantly, the AR platform can be attached to existing light microscopes without expensive modifications, nor does it need whole slide digital versions of the slide sample. According to the researchers:

A machine learning algorithm projects its output back into the optical path in real-time. This digital projection is visually superimposed on the original (analog) image of the specimen to assist the viewer in localizing or quantifying features of interest. Importantly, the computation and visual feedback updates quickly — our present implementation runs at approximately 10 frames per second, so the model output updates seamlessly as the user scans the tissue by moving the slide and/or changing magnification.

CGI meets medical images

Other image enhancement techniques don't rely on DL, but techniques from Hollywood-style computer graphics to visually reconstruct lifelike virtual 3D models from a series of images. A team including Dr. Eliot Fishman at Johns Hopkins and researchers at Siemens have developed a technique called cinematic rendering that combines 3D CT or MRI scans with volumetric visualization and other CGI enhancements to produce photorealistic images. According to Klaus Engel at Siemens:

Cinematic VRT basically operates as a virtual camera. The program makes it possible to hide soft tissue, muscles, and blood vessels, giving a clear view of the bone structure. … Our images can provide a completely new view of tissue structures. This allows a doctor, for instance, to gain a very precise overview of a fracture right before an operation. The doctor can also visualize for the patient in a very striking way that is easy to understand what this fracture looks like.

Indeed, the technique can reconstruct both bone and tissue structures with Siemens showing examples of vertebrae and ribs, carotid and coronary arteries and blood vessels in the brain. The software also allows physicians to hide or add tissue layers from a virtual image and view the bone and tissue structure from any angle. As Fishman's abstract points out, cinematic rendering "should prove useful in diagnosis, treatment planning, surgical navigation, trainee education, and patient engagement."

My take (and enterprise implications)

The benefits inherent in dramatically improved medical imaging are profound, however the example such research provides for similar developments in other industries is equally exciting. The fact that each of these techniques works with existing imaging systems and doesn’t require an entirely new technology nor apparatus to gather data is hugely significant since it allows adding capabilities to millions of existing devices and even processing archived images to reveal new insights. Indeed, it’s similar to how Tesla can add features to its cars via a software update instead of a trip to the mechanic.

When the concept of AI post-processing is applied to other industries it means that organizations are sitting on a goldmine of data waiting to be algorithmically refined. Likewise, the idea of augmenting existing imaging and other sensors with DL and AR has myriad applications, for example in physical security (real-time face recognition), predictive maintenance (failure detection), retail (virtual modeling of out-of-stock clothing/sizes), traffic and traffic control (signal, escalator, etc. adjustments based on real-time conditions) and QA (physical defect detection) among others. If these seem to overlap with examples we’ve cited for using AI at the edge, it’s because such instruments are an edge device, performing independently of a cloud controller.

We’re on the cusp of another generation of software-enhanced hardware in which AI and other computationally-intensive techniques dramatically improve the capabilities of existing devices. It’s time to dream big about how giving your sensors a brain transplant can enhance your business.

A grey colored placeholder image