Study Hacks Blog Decoding Patterns of Success

The Impact Formula: New Evidence on the Factors that Lead to Breakthroughs

November 16th, 2016 · 19 comments

impact-625px

Quantifying Impact

Earlier this month, a group of researchers from Albert-Laszlo Barabasi’s circle of network scientists published an important paper in the journal Science. Its nondescript title,  “Quantifying the evolution of individual scientific impact,” obfuscates its exciting content: a massive big-data study that dissects the publication careers of over 2800 physicists to determine the combination of factors that best predicts their probability of publishing high impact papers.

As you might expect, this endeavor caught my attention.

A high-level summary of the researchers’ results highlights two major findings:

  • The first is that “productivity” matters. The more results a physicist produced during a given period the more likely he or she was to stumble onto a high impact result. A common assumption in highly technical fields like physics is that only young people can make breakthroughs. While this research indicated that younger researchers are more likely to produce high impact papers, this was only because they tended to be in stages of their career where they can produce more total results. If you control for productivity, the role of age disappears. In other words, the force stopping a middle-aged theoretician from producing breakthroughs is less deteriorating neurons than it is strengthening university service demands.
  • The second major finding is that “skill” also matters. The researchers identified a hard to pin down quantity, that they identified with the variable Q, as also playing a key role. If your Q level is sufficient and you are productive, you are likely to produce a high impact paper. If your Q level is high and you are not productive, or if you have a low Q level but still maintain high productivity, you are less likely to generate impact. The key element of Q is that is seems to remain constant throughout a scientist’s career. It’s possible, therefore, that Q captures some fixed natural intelligence that is hard to budge. It is also possible that it captures a lot of the positive impacts of elite level training early in your career.

There are a lot of interesting implications from this work.

For one thing, it hints that individuals in creative fields should bias toward completing and finishing as many skill-based projects as possible. Put simply: productivity yields impact. (One of my first widely read guest posts as a blogger made this exact point that top performers seem to obsess about finishing things.)

Another interesting implication is that the often criticized publish or perish culture in R1 academia might actually have a strong base in evidence: the more academics publish, the more likely they are to produce something impactful.

Of course, this is just a single study so we shouldn’t extrapolate with too much confidence. Its underlying theme, however, is one that seems to come up often (c.f., here and here and here): if you want to produce things that matter, aggressively hone your skill, then apply it to generate as much output as possible.

(Hat tip: Suzyn and the NYT)

19 thoughts on “The Impact Formula: New Evidence on the Factors that Lead to Breakthroughs

  1. Denise says:

    Hi, Cal. This was something I needed to ready precisely at this moment. I support the general education curriculum at a medium regional comprehensive university. I am not a decision maker, but I do provide advice to decision makers. The majority of my work is shallow, and I am looking for ways to engage in more deep work. I also try to pay attention to the important versus the urgent. I just completed a small (micro) project and was considering writing a report on it, but then I thought, does this even matter to anyone else? Your post has given me a new perspective on that question.

  2. Aaron says:

    One of my favorite quotes from a PI was “no pub-y, no grant-y.” Simple, yet to the point; if you’re not getting things done, why should you be funded over someone that is?

    Publish or perish certainly can have some issues, but at the end of the day, publication record is going to track fairly closely with the overall quality of work. People doing interesting stuff are going to be able to write about it, and the more interesting stuff you do, the more you get to write.

  3. aniket says:

    I wonder whether this study has relevance to rest of the scientific research fields as well ?
    Is Q value so crucial ? Could that be possible to improve Q!!! by compensating elite level training later in the career ?

  4. Avi says:

    Isn’t this essentially what Ira Glass used to say (without the compelling quantitative evidence, of course)?
    http://zenpencils.com/comic/90-ira-glass-advice-for-beginners/

  5. Tilman says:

    How much of the “Q” factor is predicted by intelligence (IQ) and the various subdimensions of the openness-to-new-experience personality trait?

    Two studies suggest a close link: IQ and a work sample test together are the strongest predictor for professional job performance (Hunter & Schmidt 1998). Top-of-their-field scientists generally have IQs of around 160 (can’t find that study right now).

    Finally, what other factors influence this magical Q? Food for career advice.

  6. Yes. And the above is true across nearly all fields, not just academia. On a related note, thinking big picture – We know that Deep Work is important to produce the aforementioned results. How do we motivate the masses to turn the TV off, put the smart phones on airplane mode, and get back to work? Are we fighting a losing battle? Are we all Deep Work curmudgeons? How do we break through?

  7. Mark Miller says:

    Cal, are you yourself a fan of the Least Publishable Unit mentality? I am currently a graduate student in Computer Science at Stanford and the professors I know would rather have a paper that’s twice as useful than two that are average. By work output, do you mean metrics like publishing or do you mean an amorphous quantity of “work”?

  8. Vince says:

    Perhaps skill is the art of critical thinking analyzing material, thinking it through and assessing for depth, breadth, relevancy, completeness, etc. The productivity is the outward component and critical thinking is the introspective part. This crossed my mind because I’m currently reading through Critical Thinking: Tools for Taking Charge of Your Learning & Your Life by Richard Paul and Linda Elde

  9. ZenBen says:

    Linus Pauling trumps them all: “The best way to have a good idea is to have a lot of ideas.”

  10. Roneil Sivanandan says:

    I would like to see more of this magical sounding Q variable. Seems to conflict with the research of Carol Dweck.

  11. Dan says:

    Does anyone know if he is planning to write new books?
    Hes lates books is incredible mind blowingly good
    Sorry for going off topic just didn’t know where else to ask

  12. Sean says:

    As a fellow academic, I always appreciate when you tie Deep Work concepts more concretely into the type of work we do. One thing I didn’t get a good sense of, however, is how to balance learning new concepts and skills with actually producing things. To give an example, as a PhD student in computational biology, I’m unsure how to balance the needs to cross-train between several fields (e.g. I have a lot of statistics background to “fill-in”) and to write papers.

    Under fixed-schedule productivity constraints, I find myself spending almost all of my time developing and publishing middling results, where I may be better served by taking a *lot* of time to read papers and textbooks in order to be able to tackle much harder, more interesting problems.

    What’s the best way for a PhD student like me to balance “catching up” and publishing, under the type of system outlined in Deep Work?

    1. Hey Sean,

      Since you’re still doing your PhD, it seems to me that as long as you’re completing these projects using your newly developed skills than your fine. You won’t actually know what’s middling or what’s high-impact after the fact anyways. Who knows, perhaps one of your papers from now will lead to an insight down the road that leads to a publication of a high-impact paper.

      So just keep following your schedule.

    2. Hi Sean,
      As a practicing biochemist/biophysicist working in the biotech industry for the past 3+ years (Boston area) who did a ~3.5 year postdoc following my interdisciplinary (molecular biophysics) Ph.D., I recognize and still grapple with the issues you’ve raised. Working at disciplinary intersections is challenging because there are fewer ready-made curricula, possibly no previously laid-out logical progressions of textbooks/review articles, landmark papers, and other resources, and the paths across disciplines are less well-trodden, etc. Often the various interdisciplinary skill sets are not complimentary in the early learning phases, and you have to adopt a “cobble something together and make it work, then refine as you go” mindset. You’ll need to find multiple people to ask questions and seek advice across your fields of interest. But I will attest that the rewards for putting in the time, effort, discipline, and willingness to learn from mistakes along the way is totally worth it, as you’ll find and perhaps create awesome new opportunities once you’ve established your track record, found mentors in multiple fields, and assembled a diverse professional network.

      You’ve hit on a key issue though: balancing your time/effort between honing your skill set to increase your capabilities vs. diving into existing/new projects and executing new research to answer questions – including identifying pieces that you can “finish/complete” and publish in a reasonable time frame. It really is a balancing act and you need to plan and get into the “deep work” habit to execute on both fronts (building your skills vs. building your track record). You might be able to make strong progress on both daily, or maybe at least a few days a week. But I would strongly advise you this: don’t let even 1 week go by that you don’t make progress on executing the research (which will also result in skill-building). Building your skillset and tempering it with accumulated wisdom, knowledge, and experience is a life-long process. But you can’t afford to wait months, even weeks, but especially not years to begin building or continue adding on to your track record, experiences where you can clearly state what you executed and why it mattered, and reputation/stable of colleagues, supervisors, and mentors who can attest to your capabilities. That’s far more powerful than citing the books or articles that you’ve read and telling a potential future employer that you should be able to do something because you’re confident that you’ve learned it. You need to be able to demonstrate this with your track record and recommendations.

  13. Darwin Lo says:

    I have a hypothesis for what Q may be. On top of your “deep work,” which I call “determined mind wandering,” one must also be exposed to many inputs of many different kinds, such as from unrelated experiences and trying to reach possibly unrelated goals, some of which will serendipitously lead to insight.

    If one is disciplined about this, she can produce insights consistently. For example, Feynman had many adventures. So did Gary Kildall.

    I’m sure you’ve seen this already, but see Richard Hamming’s “You and Your Research.”

    https://www.cs.virginia.edu/~robins/YouAndYourResearch.html

    I bring this up, because I think too much “deep work” without enough external inputs is a recipe for doing a lot of unimportant work.

    1. Bernard Choi says:

      innovation is oftentimes found at the interface of two or more disciplines. In academia, with the hypercompetitive nature of federal funding these days, it is very important to have innovative ideas that you can propose and develop. I would argue that exposure to a broad set of topics, via attendance at multidisciplinary conferences or skimming a variety of journals, is essential…but so is the deep work!

      Seems like some distribution of effort between deep work and your “determined mind wandering” is in order. Could be 100/0 (or 90/10) many weeks but it probably is good to have a few 0/100 weeks as well, maybe once per quarter.

  14. Felipe B says:

    There is this story about quality increasing on repeated execution:
    http://www.lifeclever.com/what-50-pounds-of-clay-can-teach-you-about-design/
    Probably just a myth, but talks about the same point

  15. Patrick Johnstone says:

    impactful does not equal good/important/useful/world-changing etc.

    In other words, citations smitations.

    1. Yet “unread” or particularly “unpublished” (generally listed as “in preparation” on CVs – including mine in the past) might as well have never been done, at least to some degree. The researcher(s) in question might have gotten a lot out of the work in terms of their own personal growth, but translating that into future tangible impact requires “completion” and generally publication, at least for academic research. By “tangible impact” I mean:
      1.) New opportunities (in many forms)
      2.) New collaborators
      3.) New research inspired in other researchers
      4.) New research performed by the original researcher(s) – because without finishing and publishing the current work, it will continue to “hang around” uncompleted, eating a certain amount of mental bandwidth that could be applied on new problems.
      5.) New jobs for the original researcher(s), who have gone through the entire process of articulating what was done and why it was important, thus preparing them to communicate this to potential new employers (whether academic search committees or industry recruiters, hiring managers, and hiring teams).
      6.) New grants or budget allocations for the original researcher(s), which can further positively impact all of the above.

      So yes, it’s not just about citations. But it IS about establishing a clear track record of accomplishment, gaining recognition of colleagues, supervisors, & mentors, and being able to keep moving forward on other new projects.

Leave a Reply

Your email address will not be published. Required fields are marked *