Quantifying Impact
Earlier this month, a group of researchers from Albert-Laszlo Barabasi’s circle of network scientists published an important paper in the journal Science. Its nondescript title, “Quantifying the evolution of individual scientific impact,” obfuscates its exciting content: a massive big-data study that dissects the publication careers of over 2800 physicists to determine the combination of factors that best predicts their probability of publishing high impact papers.
As you might expect, this endeavor caught my attention.
A high-level summary of the researchers’ results highlights two major findings:
- The first is that “productivity” matters. The more results a physicist produced during a given period the more likely he or she was to stumble onto a high impact result. A common assumption in highly technical fields like physics is that only young people can make breakthroughs. While this research indicated that younger researchers are more likely to produce high impact papers, this was only because they tended to be in stages of their career where they can produce more total results. If you control for productivity, the role of age disappears. In other words, the force stopping a middle-aged theoretician from producing breakthroughs is less deteriorating neurons than it is strengthening university service demands.
- The second major finding is that “skill” also matters. The researchers identified a hard to pin down quantity, that they identified with the variable Q, as also playing a key role. If your Q level is sufficient and you are productive, you are likely to produce a high impact paper. If your Q level is high and you are not productive, or if you have a low Q level but still maintain high productivity, you are less likely to generate impact. The key element of Q is that is seems to remain constant throughout a scientist’s career. It’s possible, therefore, that Q captures some fixed natural intelligence that is hard to budge. It is also possible that it captures a lot of the positive impacts of elite level training early in your career.
There are a lot of interesting implications from this work.
For one thing, it hints that individuals in creative fields should bias toward completing and finishing as many skill-based projects as possible. Put simply: productivity yields impact. (One of my first widely read guest posts as a blogger made this exact point that top performers seem to obsess about finishing things.)
Another interesting implication is that the often criticized publish or perish culture in R1 academia might actually have a strong base in evidence: the more academics publish, the more likely they are to produce something impactful.
Of course, this is just a single study so we shouldn’t extrapolate with too much confidence. Its underlying theme, however, is one that seems to come up often (c.f., here and here and here): if you want to produce things that matter, aggressively hone your skill, then apply it to generate as much output as possible.
(Hat tip: Suzyn and the NYT)
Hi, Cal. This was something I needed to ready precisely at this moment. I support the general education curriculum at a medium regional comprehensive university. I am not a decision maker, but I do provide advice to decision makers. The majority of my work is shallow, and I am looking for ways to engage in more deep work. I also try to pay attention to the important versus the urgent. I just completed a small (micro) project and was considering writing a report on it, but then I thought, does this even matter to anyone else? Your post has given me a new perspective on that question.
One of my favorite quotes from a PI was “no pub-y, no grant-y.” Simple, yet to the point; if you’re not getting things done, why should you be funded over someone that is?
Publish or perish certainly can have some issues, but at the end of the day, publication record is going to track fairly closely with the overall quality of work. People doing interesting stuff are going to be able to write about it, and the more interesting stuff you do, the more you get to write.
I wonder whether this study has relevance to rest of the scientific research fields as well ?
Is Q value so crucial ? Could that be possible to improve Q!!! by compensating elite level training later in the career ?
Isn’t this essentially what Ira Glass used to say (without the compelling quantitative evidence, of course)?
https://zenpencils.com/comic/90-ira-glass-advice-for-beginners/
How much of the “Q” factor is predicted by intelligence (IQ) and the various subdimensions of the openness-to-new-experience personality trait?
Two studies suggest a close link: IQ and a work sample test together are the strongest predictor for professional job performance (Hunter & Schmidt 1998). Top-of-their-field scientists generally have IQs of around 160 (can’t find that study right now).
Finally, what other factors influence this magical Q? Food for career advice.
Yes. And the above is true across nearly all fields, not just academia. On a related note, thinking big picture – We know that Deep Work is important to produce the aforementioned results. How do we motivate the masses to turn the TV off, put the smart phones on airplane mode, and get back to work? Are we fighting a losing battle? Are we all Deep Work curmudgeons? How do we break through?
Cal, are you yourself a fan of the Least Publishable Unit mentality? I am currently a graduate student in Computer Science at Stanford and the professors I know would rather have a paper that’s twice as useful than two that are average. By work output, do you mean metrics like publishing or do you mean an amorphous quantity of “work”?
Perhaps skill is the art of critical thinking analyzing material, thinking it through and assessing for depth, breadth, relevancy, completeness, etc. The productivity is the outward component and critical thinking is the introspective part. This crossed my mind because I’m currently reading through Critical Thinking: Tools for Taking Charge of Your Learning & Your Life by Richard Paul and Linda Elde
Linus Pauling trumps them all: “The best way to have a good idea is to have a lot of ideas.”
I would like to see more of this magical sounding Q variable. Seems to conflict with the research of Carol Dweck.
Does anyone know if he is planning to write new books?
Hes lates books is incredible mind blowingly good
Sorry for going off topic just didn’t know where else to ask
As a fellow academic, I always appreciate when you tie Deep Work concepts more concretely into the type of work we do. One thing I didn’t get a good sense of, however, is how to balance learning new concepts and skills with actually producing things. To give an example, as a PhD student in computational biology, I’m unsure how to balance the needs to cross-train between several fields (e.g. I have a lot of statistics background to “fill-in”) and to write papers.
Under fixed-schedule productivity constraints, I find myself spending almost all of my time developing and publishing middling results, where I may be better served by taking a *lot* of time to read papers and textbooks in order to be able to tackle much harder, more interesting problems.
What’s the best way for a PhD student like me to balance “catching up” and publishing, under the type of system outlined in Deep Work?
Hey Sean,
Since you’re still doing your PhD, it seems to me that as long as you’re completing these projects using your newly developed skills than your fine. You won’t actually know what’s middling or what’s high-impact after the fact anyways. Who knows, perhaps one of your papers from now will lead to an insight down the road that leads to a publication of a high-impact paper.
So just keep following your schedule.
Hi Sean,
As a practicing biochemist/biophysicist working in the biotech industry for the past 3+ years (Boston area) who did a ~3.5 year postdoc following my interdisciplinary (molecular biophysics) Ph.D., I recognize and still grapple with the issues you’ve raised. Working at disciplinary intersections is challenging because there are fewer ready-made curricula, possibly no previously laid-out logical progressions of textbooks/review articles, landmark papers, and other resources, and the paths across disciplines are less well-trodden, etc. Often the various interdisciplinary skill sets are not complimentary in the early learning phases, and you have to adopt a “cobble something together and make it work, then refine as you go” mindset. You’ll need to find multiple people to ask questions and seek advice across your fields of interest. But I will attest that the rewards for putting in the time, effort, discipline, and willingness to learn from mistakes along the way is totally worth it, as you’ll find and perhaps create awesome new opportunities once you’ve established your track record, found mentors in multiple fields, and assembled a diverse professional network.
You’ve hit on a key issue though: balancing your time/effort between honing your skill set to increase your capabilities vs. diving into existing/new projects and executing new research to answer questions – including identifying pieces that you can “finish/complete” and publish in a reasonable time frame. It really is a balancing act and you need to plan and get into the “deep work” habit to execute on both fronts (building your skills vs. building your track record). You might be able to make strong progress on both daily, or maybe at least a few days a week. But I would strongly advise you this: don’t let even 1 week go by that you don’t make progress on executing the research (which will also result in skill-building). Building your skillset and tempering it with accumulated wisdom, knowledge, and experience is a life-long process. But you can’t afford to wait months, even weeks, but especially not years to begin building or continue adding on to your track record, experiences where you can clearly state what you executed and why it mattered, and reputation/stable of colleagues, supervisors, and mentors who can attest to your capabilities. That’s far more powerful than citing the books or articles that you’ve read and telling a potential future employer that you should be able to do something because you’re confident that you’ve learned it. You need to be able to demonstrate this with your track record and recommendations.
I have a hypothesis for what Q may be. On top of your “deep work,” which I call “determined mind wandering,” one must also be exposed to many inputs of many different kinds, such as from unrelated experiences and trying to reach possibly unrelated goals, some of which will serendipitously lead to insight.
If one is disciplined about this, she can produce insights consistently. For example, Feynman had many adventures. So did Gary Kildall.
I’m sure you’ve seen this already, but see Richard Hamming’s “You and Your Research.”
https://www.cs.virginia.edu/~robins/YouAndYourResearch.html
I bring this up, because I think too much “deep work” without enough external inputs is a recipe for doing a lot of unimportant work.
innovation is oftentimes found at the interface of two or more disciplines. In academia, with the hypercompetitive nature of federal funding these days, it is very important to have innovative ideas that you can propose and develop. I would argue that exposure to a broad set of topics, via attendance at multidisciplinary conferences or skimming a variety of journals, is essential…but so is the deep work!
Seems like some distribution of effort between deep work and your “determined mind wandering” is in order. Could be 100/0 (or 90/10) many weeks but it probably is good to have a few 0/100 weeks as well, maybe once per quarter.
I think the focus employed during deep work is mediated by the advice Dr. Newport also gives in his recent book to give your brain a break. Pursue a hobby, work with your hands, listen to the radio, read unrelated material. A) We are all limited in the amount of high octane cognition we can perform in a day. B) These ‘extracurricular’ pursuits can be just as important in generating insights as focused, deep activities. My favorite example of this is Stanis?aw Ulam’s idea which is now known as Monte Carlo methods. A powerful statistical framework, he got the idea while playing solitaire recovering from surgery. If he hadn’t been playing solitaire, how long would the idea’s formulation have been delayed? To draw a parallel with physical ability (e.g., bodybuilding), the strengthening occurs after the training.
I absolutely love the Ulam example. Thanks for sharing it.
There is this story about quality increasing on repeated execution:
https://www.lifeclever.com/what-50-pounds-of-clay-can-teach-you-about-design/
Probably just a myth, but talks about the same point
impactful does not equal good/important/useful/world-changing etc.
In other words, citations smitations.
Yet “unread” or particularly “unpublished” (generally listed as “in preparation” on CVs – including mine in the past) might as well have never been done, at least to some degree. The researcher(s) in question might have gotten a lot out of the work in terms of their own personal growth, but translating that into future tangible impact requires “completion” and generally publication, at least for academic research. By “tangible impact” I mean:
1.) New opportunities (in many forms)
2.) New collaborators
3.) New research inspired in other researchers
4.) New research performed by the original researcher(s) – because without finishing and publishing the current work, it will continue to “hang around” uncompleted, eating a certain amount of mental bandwidth that could be applied on new problems.
5.) New jobs for the original researcher(s), who have gone through the entire process of articulating what was done and why it was important, thus preparing them to communicate this to potential new employers (whether academic search committees or industry recruiters, hiring managers, and hiring teams).
6.) New grants or budget allocations for the original researcher(s), which can further positively impact all of the above.
So yes, it’s not just about citations. But it IS about establishing a clear track record of accomplishment, gaining recognition of colleagues, supervisors, & mentors, and being able to keep moving forward on other new projects.
This makes me think of an effect that my colleagues and I have noticed where folks who seem to be superstars at one stage of academic training fail to transition successfully to the next stage. While not everyone succeeds in their bid for tenure, we’ve noted multiple cases where folks seem to crash from the top of their cohort to the bottom on a major transition — e.g. the graduate student who publishes 7 papers during their PhD (a lot in my field) but then fails to publish a single paper in their post-doc or a superstar post-doc who publishes multiple very high impact articles, is hired as a faculty and then fails to ever get an R01 or other major supporting grant. I’d think they have the “Q” you talk about, past success would suggest they also know how to be productive and likely they have the same time constraints as others at their stage — it makes me think there must be some additional factor that leads to their failure to transition? Have you noticed this in your studies of deep work and remarkable careers? Thank you!
Would it be safe to assume, then, that if one wished to increase his or her odds of producing a high impact paper, the best way to do so is to obsess over productivity (i.e., publishing)? If Q doesn’t change that much, it seem fruitless to dedicate extensive time towards its increase. We have no control of luck, so the pursuit of productivity seems to be what matters most (thinking ‘Law’ of the Vital Few).
h-index is nearly as good a predictor as Q, fwiw. Check out Figure 6 – the difference in AUC between the two methods is only 0.04.
The authors even admit that they don’t know what Q really signifies so I think it’s a little premature to say that it measures talent. First of all Q might actually be somewhat malleable; they’re kind of underpowered to detect small changes in Q, if indeed there were any (check out the error bars in Fig 6). And second of all, for all we know it could signify something more like scientific pedigree; getting mentored by the right person has lasting impacts on your career, since being well-connected still impacts where you publish, what awards you’re nominated for, how prestigious of a job you can get, etc (see e.g. https://science.sciencemag.org/content/355/6329/1022.full). Or it could be a complicated mixture of the two, or something else entirely.