Authors: Jerome Groopman
One film of the sixty was of a patient who was missing his left clavicle. Presenting such a chest x-ray was meant to assess performance in noticing what was
not
on the film rather than merely searching for a positive finding—an exercise that points out our natural preference for focusing on positive data and ignoring the negative, as James Lock emphasized. Remarkably, 60 percent of the radiologists failed to identify the missing clavicle. When clinical data were added to the exercise, informing the radiologist that the sixty chest x-rays were obtained as part of an "annual physical examination," which primary care doctors perform in order to screen for serious diseases like lung cancer, 58 percent of the radiologists still missed it and scored the film as normal. However, when they were told that the chest x-rays were obtained as part of a series of studies to find a cancer, then 83 percent of the radiologists identified the missing bone. This highlighted that a specific clinical cue can substantially improve performance, because the radiologist is systematically searching with attention to a particular condition, rather than relying on a flash impression.
One of the most interesting outcomes of Potchen's study using the sixty films was to compare the top twenty radiologists, who had a diagnostic accuracy of nearly 95 percent, with the bottom twenty, who had a diagnostic accuracy of 75 percent. Most worrisome was the level of confidence each group had in its analysis. The radiologists who performed poorly were not only inaccurate; they were also very confident that they were right when they were in fact wrong. "Observers' lack of ability to discriminate normal from abnormal films does not necessarily diminish their confidence," Potchen wrote. His study also measured the time it took to read a set of films as an indication of the observer's decisiveness. "All observers have characteristic ways in which they manage the threshold of uncertainty in making decisions. Some people are risk takers, and they are likely to have more false-positive errors." This means that they "overread" the images, calling a normal finding abnormal—a false positive. "Others are risk averse, and they are more likely to have high false-negative rates." This means that their excess caution causes them to classify as normal what is actually diseased—a false negative. "Still others cannot make up their minds, and they will have high ambiguity numbers and more frequently require additional films before reaching conclusions."
Ironically, Potchen pointed out, based on his studies of radiologists, "if you look at a film too long, you increase the risk of hurting the patient." After about thirty-eight seconds, he found, many radiologists begin to "see things that are not there." In essence, they generate false positives and begin to designate normal structures as abnormal. Potchen believes that this reflects their level of insecurity about what they are observing. As we have seen in the studies of Roter and Hall, and the writings of Croskerry, temperament can have a significant impact on diagnostic accuracy, even among doctors like radiologists, who are not in direct contact with the patient.
There is ample precedent for both significant intraobserver and interobserver variability beyond the diagnosis of lung cancer. For example, interpretation of chest x-rays used for screening for tuberculosis showed interobserver variability of about 33 percent and intraobserver variability of about 20 percent. In screening mammography, a sample of 110 radiologists who interpreted the mammograms of 148 women, the fraction of patients actually having cancer who were correctly diagnosed varied from 59 to 100 percent, and the fraction of patients without disease who were correctly diagnosed as normal ranged from 35 to 98 percent. Overall, the accuracy rate varied from 73 to 97 percent.
Ehsan Samei of the Advanced Imaging Laboratories at Duke University Medical Center recently summarized results from a variety of radiological procedures: "Currently, the average diagnostic error in interpreting medical images is in the twenty percent to thirty percent range. These errors, being either of the false-negative or false-positive type, have significant impact on patient care." The question then is, how can radiologists improve their performance?
It is not only in radiology that observation and analysis can vary widely among doctors. David Eddy, the health policy professor at Duke, writes about the physical examination, specifically for cyanosis, the bluing of the face and fingers, that indicates a low level of oxygen in the blood: "A study of 22 doctors was performed to assess their ability to diagnose cyanosis in 20 patients, with the true diagnosis confirmed by direct measurement of oxygen levels. Only 53 percent of the physicians were definite in diagnosing cyanosis in subjects with extremely low blood oxygen, and 26 percent of the physicians said cyanosis existed in subjects who had normal blood oxygen."
Similarly, EKGs can be variously interpreted by physicians. One group of experts compiled 100 EKG readings, 50 of which showed myocardial infarction (heart attack), 25 of which were normal, and 25 of which showed some other abnormality. These EKGs were then given to ten other cardiologists to test their diagnostic skills. The proportion of EKGs judged by the ten fellow cardiologists to show a myocardial infarct varied by a factor of two. If you had an infarct and went to Physician A, there would be a 20 percent chance he would miss it. If you did not have an infarct and went to Physician B, there would be a 26 percent chance that he would say you had one. Even among specialists examining a routine test, like an EKG, there can be widely divergent conclusions.
Medical instruments do not necessarily yield definitive answers. Using a microscope, thirteen pathologists read 1,001 specimens obtained from biopsies of the cervix, and repeated the readings later. On average, each pathologist agreed with himself only 89 percent of the time, and with a panel of senior pathologists only 87 percent of the time. With the patients who actually had an abnormal cervix, the doctors who reconsidered their own earlier conclusions agreed with their first readings only 68 percent of the time; the senior pathologists concurred with their juniors in only 51 percent of the cases. While the pathologists generally did well on distinguishing clearly cancerous tissue from clearly normal tissue, they did less well in identifying precancerous lesions.
Orwig has sought ways to avoid making errors by slowing his perception and analysis. He uses his dictated report as a mechanism to be systematic. The format of his dictation follows a highly structured checklist. For example, in reading a chest x-ray, he will explicitly comment not only on the lungs and heart, but also on the bones, the soft tissues of the chest, and the mediastinum (the central structure of the thorax), as well as the pleura, the lining of the lungs; only when he comes to the summary will he home in on the explicit clinical question that was posed by the internist or surgeon who requested the x-ray. "Once I got a call from a clinician who said: 'I ordered this x-ray to see if this man had pneumonia. Why did you put all the stuff in about his ribs?'" Indeed, there was a large patch of white within the black image of the man's right lung, indicating a pneumonia. But Orwig had made special note in his dictated report of several healed rib fractures. "Some radiologists would not take the breath to report these old fractures because they do not seem to be an active issue or relevant to the primary diagnosis, which is an infection," Orwig told me. Part of his rationale is just to be complete, but part is because any observation could prove to have clinical import. For example, old fractures might suggest that the patient had fallen in the past, because he is an alcoholic, or might have passed out due to a seizure disorder that was not recognized. People who are drunk or have a seizure sometimes aspirate their mucus, setting up a fertile field for bacteria to enter the lungs and trigger pneumonia. As it happened, the clinician went back and interviewed the man, who confessed that he was a binge drinker.
The morning I spoke with Orwig, he had just returned home after a night on call. He had been asked to read a CT scan of a middle-aged man in the ICU. The patient was an alcoholic with liver disease and had been admitted to Marin General Hospital confused and delirious. It turned out that he was bleeding internally, and like many patients with cirrhosis, was tipped into his delirious state because his liver was unable to detoxify the products of digested blood from his gut. The CT scan was ordered because after initially improving, the patient had become delirious again. The ICU physician assumed that the man was once more bleeding internally.
Orwig looked at the CT scan and then went through his methodical checklist. He traced every loop of bowel on an abdominal CT scan; his colleagues often joke when he lingers over the study, "There goes Dennis again, tracing every loop from the stomach to the anus." As he followed the turns of the intestines, he noticed what appeared to be small air bubbles in the abdomen. These bubbles did not look like the kind of gas we all have in our bowels. "I finally decided that the air bubbles couldn't be in the bowel," Orwig told me, "so they had to be in the superior mesenteric vein." Somehow, gas had accumulated in the vessels that drain blood from the intestines. "Then I noticed that the loops of bowel near the gas bubbles were thickened." Orwig reasoned that the blood supply to the intestines must somehow be impaired, so there had been a breakdown in tissue, with gas from inside the bowel moving into the surrounding blood vessels. This condition is called ischemic bowel, meaning the bowel is starved of its nourishing blood supply and begins to decompose.
When Orwig spoke with the clinician in the ICU, he was met with skepticism. "You guys aren't very good in diagnosing ischemic bowel," the physician said. Orwig agreed that it was a difficult diagnosis based on a CT scan, but explained that he had thoroughly traced every loop of intestine in the man's abdomen, and the gas didn't belong where he found it. It was imperative that a surgeon be called to assess the patient; if Orwig's presumption was correct, there was an urgency to operate and restore the blood supply to the bowel. Orwig was right, and the patient's life was saved.
"Sometimes, going with your gut just doesn't work in my field," Orwig quipped. "There is so much gas in the abdomen that just seeing a few little bubbles doesn't mean anything in the picture as a whole. It only has significance when you segregate out each structure, and then you can see that the gas doesn't belong there."
As he systematically reviews every aspect of the film, Orwig explained, "My brain is forced to work in a similar stepwise way. It is easier—certainly quicker—to simply look at the pneumonia in the right lower lobe of the lung," he said, "and not take the time to detail all of the other information. But this protects me." Orwig is "protected" from the most common mistake that radiologists make, the error of search satisfaction. As we saw earlier, it is a natural cognitive tendency to stop searching, and therefore stop thinking, when one makes a major finding. This is all the more true in radiology, where a busy internist informs the radiologist that the patient has typical findings of fever, cough, and yellow sputum, and so directs the radiologist's attention on the lungs in his search for the expected pneumonia; but if he focuses solely on the lungs, Orwig said, and snaps to the correct diagnosis of pneumonia, he risks missing a dense area in an upper rib that suggests there may be an underlying cancer, or a widening in the mediastinum that could be an aneurysm of the aorta.
Orwig is part of a large private radiology practice that consists of eleven doctors. They are conscious of the risk of making errors if overloaded, so recently they added two new members to the practice in order to limit the number of x-rays each one is required to read on a shift. Like primary care physicians, they are seeking new ways to secure enough time to think about each case. They also instituted a quality assurance program. Every day, each radiologist in Orwig's group reads four or five x-rays that are independently read by a colleague. Then the two readings are compared for discrepancies. Sometimes a discrepancy is insignificant; other times it may be of major import. The results of this daily exercise are entered into a database for the entire group, so that there is ongoing monitoring of each radiologist as well as the whole team. "This way, we learn from each other's mistakes as well as our own," Orwig said.
Orwig was chastened some time ago when one of his fellow radiologists came into the dark reading room with an MRI scan of a knee. "What do you think of this case?" his colleague asked. Orwig looked at the scan and said, "Torn ACL"—anterior cruciate ligament, a common sports injury. The colleague put Orwig's report down in front of him. It read: "normal anterior cruciate ligament." "I was mystified," Orwig told me. "It's incredible that at one time I could look at a film and only later see what I had missed." The only explanation that came to Orwig's mind was that he had relied too much on "gestalt" and not methodically traced every anatomic component in the knee.
"This is also a problem of high volume," Orwig said. "A hematologist like you, Jerry, cares for patients over the course of months to years. You see them in follow-up visits, so when something goes wrong, you can deconstruct the steps in the diagnosis and treatment and figure out where you made mistakes. I am literally reading hundreds of x-rays, day after day after day, on different patients. Most of the x-rays are not follow-ups. So it is difficult for me to go back and figure out why I erred in my observation. I have no ability to put myself back in the seat when that knee case was in front of me." Because of this, Orwig said, "I have to keep reminding myself to be systematic. The more experience you have, the more seasoned you are, the greater the temptation to rely on gestalt."
E. James Potchen of Michigan State commented on Orwig's strategy of sticking to a methodical checklist. He agreed that Orwig would have some "marginal gain" by forcing himself to look at each of the anatomic structures on the film. But the "real added value," Potchen said, was what Orwig did in drawing the clinician's attention to the rib fractures or gas bubbles. Potchen has studied decision-making under uncertainty, not only in medicine but also in law and in business. He recalled how Dr. Merrill Sosman, who was chief of radiology at the Brigham Hospital in Boston, would be given a chest x-ray and then declare to the residents in training, "This patient has kidney failure." This was a deduction worthy of Arthur Conan Doyle, and the residents would wonder: How can you diagnose a problem with the kidneys by looking at the chest? Sosman explained that he saw a thickening of the ribs, which led to the insight that there had been remodeling of the bones because the kidneys had failed, changing the metabolism of calcium and phosphate. "That's how you add value as a radiologist," Potchen said. "You discover what is not known about the patient at the time that the x-ray was taken. And that's how you develop your cachet. You add something beyond what other people do." This, of course, is why primary care physicians are sending so many patients for radiological studies.