Deconstructing Theory
As a self-observant theoretician, I’ve learned that my research success depends on two intertwined factors: (1) my ability to digest and understand diverse results in my field; and (2) my ability to persistently attack good problems once identified.
Through practice over the past few years, I’ve become adept at the second factor. My deep work hours per week are quite high and have recently led to a correspondingly high rate of producing publishable results.
A nagging concern of mine, however, is that I’m not as good with the first factor. Indeed, I’m often frustrated with how long it takes me to digest interesting new results (and how often I end up aborting the process).
This concerns me because in my field voracious reading is required to keep the pipeline of good problems full.
What’s going wrong?
I’ve identified three issues that slow me down when reading existing work:
- The results I read tend to be heavily mathematical but also appear in venues with space constraints. As a result, many details are missing. In a standard theory paper in my field it’s common to encounter a lot of (semi) complex inequalities that are just plopped down in the paper with minimal derivation or explanation, and then combined to achieve the proof. This style of mathematics by fiat makes it difficult to understand the argument and tends to overwhelm me.
- Results are often expressed as a combination of text and math that can lead to ambiguity. (This morning, for example, I spent 90 minutes trying to understand the explanatory text that accompanied a frustratingly simple modular equivalence.) Often these ambiguities can be resolved in the larger context of the paper, but I have a tendency to remain where I get stuck, obsessing over the ambiguity until I get too tired to make any more progress.
- I tend to lose my concentration faster when reading than when thinking. This occurs because understanding someone else’s result requires that you keep organized and accessible in your mind an ever growing collection of definitions and intermediate claims — many of which can seem arbitrary at first. This is tiring and I often flag too quickly to make productive headway.
To be clear, I’m not terrible at reading papers. As a professor, of course, I already do this activity at a high level as measured by most objective scales. But given my interest in continuing to push my abilities, I know that I could be even better.
Here’s my project: I want to overcome these issues and push my ability to quickly digest relevant academic papers to an extreme. This experiment is useful to me for obvious professional reasons, but I hope that it might also provide a nice case study in how such deliberate improvement proceeds in a knowledge work setting.
To get started on this project, however, I need some help. Please share in the comments if you have specific strategies you think I should consider…
Hi Cal,
I am incompetent at math but I do have an excellent understanding of reading and writing. Here are my suggestions:
1) Recognize that many academic papers are badly written. Don’t assume the fault is yours. It may also belong to the author.
2) Don’t think while sitting. If you want to think, require yourself to move around. There’s growing research that creativity is linked to MOVEMENT. https://buff.ly/1sjEK8U When you sit, read. When you think, walk. If you develop this habit you’ll go a long way to helping yourself.
3) Don’t believe that reading slowly = reading with greater understanding. I learned this when I took a speed-reading course many years ago. We often read slowly when we want to understand. But this is almost always wrong-headed because our we tend to sub-vocalize when we read. (If you want to break this habit, try humming when you read. This will force you to stop sub-vocalizing.) If we are sub-vocalizing that means we are reading as slowly as the human voice can move which is usually no more than 200 wpm. To improve your reading speed it’s important to skim first, to avoid sub-vocalizing and to have a specific goal about what you want to get out of a particular piece of reading.
By the way, it’s probably harder for you to read academic papers than many other academic writers because you write really well.
I agree with the first two points, but the third point is probably not all that useful for reading proofs. I happen to be a PhD student in a proof-intensive area and have taken a speed-reading course. The techniques of speed-reading are helpful when reading articles mainly composed of only words, but they aren’t all that helpful in reading proofs, since most proofs are loaded with math symbols which themselves represent a thousand words. The reason why people read proofs slowly is not because they vocalize and hence can only read up to the speed of speaking, but that it takes time for their mind to absorb the information behind the heavy math expressions.
It does not really matter if an important paper is badly written or not, scholars still need to understand the material. Cal can blame the author, but he still has to sit down and understand the badly written paper.
I do not really buy into the argument that Cal’s writing ability makes it harder for him to understand badly written papers. Badly written papers are hard to understand for all of us. Bad prose describing highly technical content in the form of a riddle compounds the difficulty since the material is inherently difficult and the bad writing adds more distractions.
I don’t have a short term solution, but …
It seems ridiculous to me that scientific and mathematical papers are still presented as they are in journals.
It would make so much more sense to have them on some form of web format, such as a wiki that is editable by the readers.
The readers could contribute content to help other readers, point out ambiguities that the authors could see and correct. Even just a blog with a comment section would help.
If it became popular, structure could be added to the papers that are submitted, for example maybe the mathematics would have to be entered in some sort of structured format that has to be adhered to by all papers. Something like Mathematica could provide a structure as well as a way to computate. This common structure could minimize problem 1.
This common structure could also make writing papers easier and quicker, since there could be a way to “quote” and pull in work from other papers without having to rewrite and rehash.
I don’t know what sort of copyright is applicable to the papers in your field, but I am sure that if you have this problem, many people do. If you (or somebody) were to start a platform like this, I would think it would be an obvious winner.
I don’t think the REAL solution is to work your ass off alone. The real solution is to create an online platform that solves the problem.
This is the goal of academia.edu.
I don’t know the answer to your situation, but consider this. The paradigm of axiom->proof->result->proof->result->etc. is the essence of mathematics, but generally lousy pedagogy. Otherwise we would start first graders in set theory, second graders with the group axioms for the integers, third graders with the field of rational numbers, and so on. Maybe it’s better not to attempt to read an article from beginning to end. Rather, scan over the whole thing, then scan again a little more deeply, then again… Pick out the easiest parts to understand and digest those first. Leave the most difficult parts for last. And question if necessary whether your priorities require 100% understanding of the entire article. Maybe at some point your time is better spent moving on to the next article.
Cal,
Thanks for the post. I’m very interested in reading the suggestions that get submitted. Further, do you plan to update your progress via the blog and / or synthesize the responsive material into a coherent “game plan”?
Daphne’s insight is a great start. Thanks Daphne.
I look forward to seeing this come together.
Thanks again,
Jeffrey
My first thought (maybe naive) was to ask for help, either:
a) Swap notes with colleagues and offer to help speed-read their papers.
b) Contact the author with one or two specific questions – like office hours for students
While I lecture at university, I’m not in academia so I don’t know how feasible either of these two options are.
Always inspired by your constant efforts to improve your work process.
All the best.
–Phil
Don’t underestimate Phil’s recommendations. I’m not an academic, but was recently reading a paper on a software toolkit I enjoy using. I asked the author some questions and he agreed to schedule a phone meeting to go through the paper. He was more interested in having me understand the ideas behind it than explaining all the tedious details, which is exactly what I needed. If I had more of the theoretical background I’m sure he would have taken the discussion deeper in those areas.
Presumably everyone in your field is in the same boat. Discussing with authors or colleagues would, I think, be the fastest way to learn the insights of a paper.
Hey Cal,
Your second point reminded me of Terence Tao’s essay on reading math papers, which I’m sure you’ve seen sometime before. Specifically, I was reminded of how what he wrote directly addresses some of the problems you’re having. I’ve copied the pertinent examples below.
2) Your words: “Often these ambiguities can be resolved in the larger context of the paper, but I have a tendency to remain where I get stuck, obsessing over the ambiguity until I get too tired to make any more progress.”
TT: “[When] one encounters the task of reading a technical mathematical paper for the first time, it is often the case that one loses much of one’s higher reading skills, reverting instead to a more formal and tedious line-by-line interpretation of the text. As a consequence, a single typo or undefined term in the paper can cause one’s comprehension of the paper to grind to a complete halt, in much the same way that it would to a computer.
“In many cases, such “compilation errors” can be resolved simply by reading ahead in the paper. In some cases, just reading the next one or two lines can shed a lot of light on the mysterious term that was just introduced, or the unexplained step in the logic. In other cases, one has to read a fair bit further ahead; if, for instance, the conclusion of Lemma 15 was difficult to understand, one can read ahead to the end of the proof of that Lemma (in which, presumably, the conclusion is obtained), or search ahead to, say, Proposition 23, in which Lemma 15 is invoked, to get more clues as to what Lemma 15 is trying to say. (The use of search functions in, say, a PDF reader, is particularly useful in this regard.)”
3) “This occurs because understanding someone else’s result requires that you keep organized and accessible in your mind an ever growing collection of definitions and intermediate claims — many of which can seem arbitrary at first.”
TT: “In extreme cases, one may have to go to a large blackboard and diagram all the logical dependencies of a paper (e.g. if Lemma 6 and Lemma 8 are used to prove Theorem 10, one can draw arrows between boxes bearing these names accordingly) to get some sense of what the key steps in the paper are.?” (which sounds like a specific tactic that you may already be using?)
(As for the essay itself: https://plus.google.com/u/0/114134834346472219368/posts/TGjjJPUdJjk )
I think your first issue’s also somewhat addressed in his writing–he writes about how it can help to “get a sense of what the author is trying to do with each statement or lemma in the paper, rather than focusing purely on the literal statements in the text.”
I’m sure that your proof-reading skills are far above those of the audience TT meant for this essay (math grad students), but because there was such a direct correlation between your issues and his solutions, I thought it might be of some help. 🙂
In economics at least (where I am a PhD student) a big step for me was understanding that academic papers are not meant to be read as novels from intro to conclusion, despite many authors insistence on writing them this way. In econ, the order of sections I read is : the conclusion, the model set up (methodology and data for empirical papers), the results, and then, if the paper is exceptionally interesting, I’ll read the intro and other sections.
It’s probably not as much of an issue in computer science, but the most important parts of an economics paper to read are the footnotes since there resides the flaws in the paper that the author is aware of, but hopes you don’t read.
For mathematical papers I’ve worked through, the best way I’ve found is to make a “problem set” out of the paper itself rather than just taking notes or recreating results. The “problem set” consists of a series of questions that subtly hint at the next steps as well as the motivation behind the ideas, this way you are forced to take the view point of the author figuring out the initial results. This method does take a considerable amount of time, but you will develop a deep understanding of the paper.
Read them the way you’d read a new chapter in a textbook, or how you’d write an exam. First read quickly to learn the layout and find road signs. Pretty much ignore the equations. Then read it a bit slower, to catch more details. Then more passes, each one taking one or two parts a bit deeper, ending with a “Yep, I got that” pass of the whole thing. It might take many passes, spread over a week so your subconscious can have a turn, but I often found that worked better than bashing away at the first equation in the article.
(That got me through thermodynamics of chemical reactions in batch and continuous rectors. Every variable affects every other. It’s also what they recommend to kids learning to read. Often the next few sentences will give clues as to whether Tom is the cat or the human, and a r-oh-k-ee-t might be a rocket.)
Cal, you’ve probably thought of this, but sometimes it helps to switch to “overview” mode when you find yourself stuck. That can give you a framework to hang details on. I’m not a professor, but I use it when I read or review technical documents.
For specifics, consider setting a time limit for getting stuck (e.g. five minutes without progress), and a similar (if difficult to quantify) limit on energy. When you hit the limit, move along looking for high points until you finish the paper, then go back for a deeper reading.
If I am really stuck on a particular point I would ask the author to explain. Most people are happy about interest in their research and happy to provide more information. Of course, there are often mistakes in papers. At least I see in economics that keeping the notation straight is hard. A lot of papers are just really badly written.
I’d follow your own advice for solving hard problem sets – if possible, seek out someone who knows the paper/area you are stuck on well, and ask them to explain it to you. I don’t know how easy or hard it may be to find someone though
Two things:
1: I find that it helps me a lot that I have an overall idea of what the point with a piece of math is. If you get stuck move on, and don’t obsess over a specific detail. Sometimes ambiguity is resolved, or only one of the possible meanings makes sense further along.
2: Following the proof on a piece of paper can sometimes help me understand what is going on. I find that the action of actually writing out the equations can trigger a deeper understanding.
Hello,
I still find problems when reading lots of material and “digesting” the results. I think there are two fundamental points that have helped me a lot and maybe you could try,
1.- I would try to do a fast review before start reading to see what the paper/article/book is about and see if it really interest me. This also helps me to see the parts that are more important for me and where is worth to spend more time if I do not completly understand it. (also it helps me ignore parts that can be interesting but have nothing to do with my objective to read the paper)
2.-When reading, I try to write down the ideas and follow the results on my own draft paper. The notes I take are usually a mess but they help me to organice the document in a way easier for me to understand and to keep the focus for a longer period of time.
I hope this can help.
Hello Cal,
I greatly admire your work, and very much empathize with your particular dilemma. As an assistant prof in the Faculty of Medicine here in a university in Canada, and an associate editor for one of our flagship journals, I also have to read and try to digest a lot of primary papers that can be heavy in the theory/math and make some sense of it. A few of my thoughts include (in no particular order of preference but rather, let you pick and choose at your discretion):
1. As some of the comments above suggest, getting a brief overview of the study by: getting a quick read by reading only the abstract, intro/background, and discussion. This is a filter for : is this paper relevant to my purpose?
2. For a good overview of a new topic: Searching for the latest review, and for increased rigor, ensuring that you find the latest systematic review for the particular topic. There are guidelines that categorize the scientific rigor of clinical papers, and such criteria might be found in the type of papers that you are looking at too. The reference section of these papers then provide a nice list of relevant papers that allow you can then get and read in the manner of 1. above. Of course, this does not keep us from getting bogged down in the minutiae!
3. For the actual math part or technical aspect, I look at my old textbooks for the basics or online resources. For the journal editorial work, we have a statistician that we can call upon if we need to. If this aspect of the journal article is still unclear, I would then contact the author (s) for further clarification. Academics are often flattered if you cared enough to reach out to them about their work!
4. If approach 3 is not satisfactory, I always think to myself, “if I am having trouble digesting this material, then others might also be having difficulty.” You could then contact the senior editor of the journal and see if she/he might have any suggestions on clarification of some of the points or their particular strategy, as they have the ultimate say on what does/does not get published and therefore have to review all the papers very quickly.
5. Asking an expert in your particular University (although does not have to be) who could explain the technical/mathematic aspects (I would get the profs who have won teaching awards as they can simplify topics/explain complex concepts to undergrads). Likely the papers that you will be reading will have similar math so once you build up your understanding, your reading of papers will get faster and faster.
6. To stay on top of relevant papers, there used to be a free service at our university called “uncover” that would provide weekly updates of scholarly articles to your inbox that fit the criteria that you specified eg. particular key words. Usually you will be quickly able to identify key authors in the field that you can also follow. A recent service that someone recommended to me was “ResearchGate” and you can updates on scientists with their new publications.
7. Reach out to your academic community for help. Also, a greatly untapped resource is the friendly university librarian, who often has several resources to help get you through the particular dilemma.
8. Search for this precise topic in the literature, or ask editors/grant reviewers how they overcome this problem too!!
I hope some of this helps. Keep up the wonderful work!
Ed
Hi Cal–these are such great suggestions and explanations. I immediately thought of the old saw about the biggest difference between an undergrad and a prof: the prof reads the footnotes first.
But what I wanted to ask is what do you consider a “good week” in terms of deep work? How many hours of “deep work” would you consider adequate to progress the way you have been progressing?
I don’t think there any shortcuts. Those papers are not written to be clearly understood. I find it helpful to just skip the math unless the math is directly relevant to answering a research question I have already formulated and that paper is the only place I can learn it. Otherwise I just read the intro and conclusion to get a big picture.
I think you may just be too hard on yourself and holding unrealistic expectations to think you can master the math in every paper you read. I don’t think it’s an efficient use of your time though. If you’re already publishing original research, it sounds like you already know what you need to know from other people. Don’t second guess yourself and think that you need to stay current on other people’s research. Let them do theirs and you focus on yours. If you must read their work, stick to the intro and conclusion and don’t get bogged down in details. It’s not worth it.
Hi Cal,
This is a topic I’ve become interested in recently as I try to improve my reading skills, although not specifically for academic papers in math.
There is an excellent book by Mortimer J. Adler called “How to Read a Book.” He breaks down the process of reading so the reader can gain as much as possible from the writing. It is aimed at reading books but may be adaptable to academic papers.
The most useful part for me so far relates to your second problem. He recommends doing a quick first reading after a systematic skimming of the preface (abstract), table of contents, and index. During the quick read you simply push forward if you don’t understand something. Just move on. You’ll have the opportunity to dig in to those areas on a second reading (the level of analytical reading) where you directly engage with the author and the writing. I used this strategy when reading The Black Swan, much of which went over my head on the first read.
Shane Parrish has a helpful summary of the process on his Farnam Street blog: http://www.farnamstreetblog.com/how-to-read-a-book/.
I’m interested to see the strategies you use to improve this skill and watch your progress.
Hi Cal,
In order to become better at understanding in great detail (line-by-line) a paper, you should exactly deliberately practice this exact skill every day to become better.
Yet in order to read faster papers and grasp the results, you could alternate between three modi:
1) detailed line-by-line analysis
2) structural reading: first grasping the main structure and then filling in the details
3) prediction/reconstruction: read the results and think back for yourself what is needed to obtain these results.
Each of these three modi is a different skill which could be deliberately practiced on its own. In my experience the exact combination needed to tackle a certain paper is different for each paper.
I often read theory papers in political economy and in complexity science. Perhaps not as math heavy but I think I relate to what you find tiring about the process.
When I’m really trying to understand a paper on a deep level, my approach is to read the model setup first and the results second. I try to visualize them in my brain and think about what might be the logic and mechanisms driving them. Then I check the proofs to see if the derivation squares with my understanding of the forces at play in the system. Maybe my experience differs from applied math and CS: the proofs in the papers I read often use a core intuition about the system of interest, so they’re not necessarily that hard to follow once time has been invested understanding the structure of the problem. The text often also spells out that core intuition.
Usually, that’s not enough. I often then need to explain it to someone and have a discussion about it (seminars specifically meant to read papers are great for that), or play with the results. I have sometimes tried to simplify the model to see if the same results come out if I leave out what seems to be a detail (that helps check that I understand what really drives the result), and have sometimes programmed the main equations.
As other commentators have pointed out, the process rarely ends up linear, but rather a journey through the paper.
Personally I try to figure out what the problem is that the paper solves and the technique it uses. If I’m still interested, I’ll try to answer these questions:
-What is the problem this paper tries to solve?
-Why is the problem important or interesting?
-imagine following the experimental steps yourself?-why are they doing it like this??
-what alternatives could they have used?
-what are the strengths/nice ideas of this
method??
-what are the weaknesses of this method/unresolved alternative explanations?
-how could you improve upon this
Cal: You just tapped on to my concern.
As I am planning to get back to academics after 2 years of industry work, my concerns are literally subsets of those two points.
I am concerned about my speed or productivity in an academic environment. I like hard problems. But time concern stresses me, and drops my productivity. In industry, you have clear goals. But in academics, I fear that you often don’t have clear goals and that affects your pace. I hope to discover this issue more as you dig deeper into it.
Hi Cal,
I’m a PhD Candidate (Organisastional Behaviour) and this was one of the first problems that I encountered. My approach is in line with some suggestions made earlier.
I like Cameron’s idea of turning reading into a problem set. I find that turning reading into a series of tasks lets you constantly monitor your progress. It also reminds you that this is “work” and not casual reading.
I agree with Daphne’s points: pushing yourself to read faster helps (I’m going to help out at the UK Open Memory Championships to get closer to people who have mastered this and related skills), avoid subvocalising (sometimes I put my tongue on the roof of my mouth) and always have a goal for each piece of reading.
The approach that I teach to my undergrads is the following:
Goals:
To answer the following questions for every paper read:
*What question/problem is the paper trying to address?
*Why is this question/problem important?
*What is their argument (theory, constructs)?
*What approach (methodology etc) did the authors take?
*What did they find?
Tasks:
1) Read the abstract, introduction and then the conclusion (in that order).
This helps to structure my thinking and get a birds eye view of what may have been missed, overlooked or deliberately omitted.
2) Have a quick look at the references.
This provides context for the paper and the literature that supports it. It also highlights gaps in my knowledge which I can later investigate.
3. Read the methodology, literature review etc.
4. Write a short summary that addresses the goal questions.
Using this approach makes boring papers less tedious and interesting papers a joy.
As you can imagine, using this in conjunction with your recommended active recall means that you are well placed to understand the papers in a meaningful way that is easy to communicate.
I hope this helps, I would love to hear any other ideas and happy to discuss further!
1. For mathematics by fiat – get a grad student to recreate the derivations that are abbreviated if they are only “semi-complex;” good practice for a grad student and could save you the time of working backwards in the middle of reading; it will take some pre-planning and a reliable grad student.
2. getting stuck on an ambiguity – I think you have already answered this yourself, but you have to move forward to see if the issue is solved later in the paper. Convince (trick) yourself mentally that you are not leaving the ambiguity, but investigating the ambiguity by moving forward in the paper. (you have to give your brain permission sometimes, like in a shut-down ritual)
3. losing concentration – map what you are reading on a sheet of paper. I know that is an extra step, but the more you hold simultaneously in your head, the slower you will move, and you’ll get distracted. This might be an instance in which you read while making notes and diagrams, rather than using the morse-code method.
hope that helps!
As a reading teacher, I am happy to see that people have recommended pre-reading strategies and post-reading strategies, which are things that expert readers do, but I’m surprised no one has suggested doing things during reading, such as annotating in the margins. I have my freshmen write main ideas in the left margins and write comments, definitions and questions on the right side to help them comprehend, focus and remember what they read.
When a text is difficult, we all become freshmen again, and I find that doing these elementary strategies helps me stay focused and delve deeper into the reading material.
Contacting the authors of the papers sounds reasonable.
Otherwise, one thing that works for me when I want to internalize a concept is to discuss it in a group setting. Other people can chime in with an insight or even a question that clarifies everything.
Another thing that helps me get different perspectives when I try to crack a problem is to simply think about it in different physical locations (i.e. not at my desk). I often found that thinking about the problem when sitting on a bench in the park, or even when travelling to meet a friend, uncovers new insights.
When I was in undergrad and in a challenging (even for the grad students) journal club, I would use the ‘frat house floor’ approach: with a really hard problem, the fastest way through can be to do the job twice. I would read the paper through once. I didn’t concern myself with comprehension on the first read through, just that I had read every word, looked at every equation, figure and table. About three days later, I would go through and do a more in-depth read of the paper. About 95% of what I would get stuck on made perfect sense and I could focus on figuring out the parts that were actually hard rather than vocabulary or flow issues. It is amazing what the brain can do if it is fed something and left alone.
I have this same problem Cal and like Daphne said above, I first have to remind myself that most academic papers are badly written. Especially the physics/math/engineering types. The authors don’t go out of their way to explain the equations and I find myself doing the same thing: Spending an hour re-reading the same paragraph over and over. The other day, I read a paper that so well written that I was pleasantly surprised. Especially since it was also a densely mathematical type of paper. I breezed through it in less than 30 minutes and had a ton of notes by the end.
Sometimes you can’t always blame yourself. However that hasn’t stopped me from cracking down on the math textbooks to freshen up on calculus.
I’m surprised this hasn’t been mentioned more than once (Leila’s post / TT): Blackboards.
The fastest learning of difficult subjects I’ve ever done has been with another grad student (or two) standing at a whiteboard with a paper. Not only does looking away from the paper to write out equations and work through steps give you much more “room” both on the board and in your head, but the second reader will almost invariably catch you right as you make a mistake working through a step, which speeds things immensely.
This exact approach is applied to coding as well. Big monitors and another programmer sitting beside you. You swap the keyboard when you get fatigued. Every single bug you miss just screams at the other coder, and seldom survives.
My main problem with this (and I suspect you will say the same) is that once you get sufficiently specialized there’s only a few people in the world that understand (and care) about that particular paper, and they’re almost never at the same institution. Finding that study-buddy is nigh-on impossible. But finding a whiteboard isn’t.
As a mathematician, the only thing that has ever worked for me with reading papers is
(1) be doing examples all the time, have examples in my head of every definition, theorem, computation (and counterexamples as well), the concrete stay in mind more easily than the abstract;
(2) if at all possible, read the paper with others. Math is more effective and always more fun that way, anyway.
Cal, I understand your challenge completely. Here’s how I handle it. In the last hour of the evening before I go to sleep I read over the material lightly, just trying to pick up the main points, understand the organization of the ideas and the conclusion. The next day when I sit down to really dig in to the material I’m usually more than half done. I also retain much better. Try it you’ll like it!
Can you link to an example paper on the arXiv, so one can get an impression on how those papers look like?
Look at the end of the blog post before this post. There is a link there to one of Cal’s drafts on arXiv.
Many people in a field share the same goal of trying to stay on top of new results. Why not start a new-papers-seminar / journal-club? Both of these approaches are successful in other discinplines, and trying to understand a technical paper is a lot easier when done in group (it’s easier to search the possibility space for the meaning of the author).
Brainstorming is horrible for deliberate practice or deep thinking, but I think it’s good for developing better understanding because one gets to see many different approaches.
Quickly read through each paper to gain its Big Picture ideas and to make connections between concepts located throughout the paper.
Create a Mind Map (or make notes) as you read that converts the story the other author is telling into your version of the story. List ambiguities during your quick read, but don’t try to resolve them.
Determine if the paper merits more time and go deeper if it does.
If multiple papers are available, quickly read and Mind Map each and prioritize them for deeper study.
Fascinating post and comments. Cal, about how much time do you spend per week doing this kind of meta thinking?
Re you tire out when reading: I like the suggestion of reading while pacing, since you can’t fall asleep while walking. Another thing to do is simply set a timer for 10 minutes (still awake?), then 15 minutes (still awake?), and test to see what your limit is for uninterrupted active reading. You probably get cumulatively fatigued after a time, so it won’t pay to read more than 50 minutes at a time.
I also would favor the idea of binge-reading: get a stack of these papers and blow through them as fast as you can in your allotted reading time. You can pretty quickly tell which are more interesting, which are uninteresting, and which are a mess. Then attack the interesting ones with your arsenal of thinking tools.
However, I’d like to take a step up from the tactical solutions that have been provided and look at the statement you made: “Indeed, I’m often frustrated with how long it takes me to digest interesting new results (and how often I end up aborting the process).”
Define your terms! as a professor of mine used to say.
What do you mean by “digest”? That you own the material such that you could explain it to a colleague? That you could lecture to the wall on it? That you could incorporate it into your current research?
What do you mean by “how long it takes”? What would be a reasonable time, in your estimation? What would be a reasonable time for an MIT PhD student? What would be a reasonable time for a new professor with classes to teach, a young child, an active research program, service obligations, etc? Maybe your brain takes about two weeks to really own the material — given everything else you ask it to do — and the process just can’t be sped up.
What do you mean by “interesting new results”? Interesting how? In what way? What are your criteria? How would you use these interesting new results?
I have the feeling that the commenters are replying to different aspects of your question because the question is too ambiguous.
What is your goal from a reading session (or a month of reading sessions), clearly stated?
It could also just be that this is the messy part of knowledge work and while there may be ways to make it less messy, there may be no way to make it stainless-steel systematic. Those researchers didn’t write their papers to give you interesting problems to solve; they wrote them for their own nefarious reasons and your transformation of their results into something that will benefit you is a creative act on your part that will necessarily take some time and work.
Keep us posted on your progress! You’ve touched a nerve!
Re you tire out when reading: I like the suggestion of reading while pacing, since you can’t fall asleep while walking. Another thing to do is simply set a timer for 10 minutes (still awake?), then 15 minutes (still awake?), and test to see what your limit is for uninterrupted active reading. You probably get cumulatively fatigued after a time, so it won’t pay to read more than 50 minutes at a time.
I also would favor the idea of binge-reading: get a stack of these papers and blow through them as fast as you can in your allotted reading time. You can pretty quickly tell which are more interesting, which are uninteresting, and which are a mess. Then attack the interesting ones with your arsenal of thinking tools.
However, I’d like to take a step up from the tactical solutions that have been provided and look at the statement you made: “Indeed, I’m often frustrated with how long it takes me to digest interesting new results (and how often I end up aborting the process).”
Define your terms! as a professor of mine used to say.
What do you mean by “digest”? That you own the material such that you could explain it to a colleague? That you could lecture to the wall on it? That you could incorporate it into your current research?
What do you mean by “how long it takes”? What would be a reasonable time, in your estimation? What would be a reasonable time for an MIT PhD student? What would be a reasonable time for a new professor with classes to teach, a young child, an active research program, service obligations, etc? Maybe your brain takes about two weeks to really own the material — given everything else you ask it to do — and the process just can’t be sped up.
What do you mean by “interesting new results”? Interesting how? In what way? What are your criteria? How would you use these interesting new results?
I have the feeling that the commenters are replying to different aspects of your question because the question is too ambiguous.
What is your goal from a reading session (or a month of reading sessions), clearly stated? What actionable result do you hope to achieve? (Or is that the wrong way to look at the problem?)
It could also just be that this is the messy part of knowledge work and while there may be ways to make it less messy, there may be no way to make it stainless-steel systematic. Those researchers didn’t write their papers to give you interesting problems to solve; they wrote them for their own nefarious reasons and your transformation of their results into something that will benefit you is a creative act on your part that will necessarily take some time and work.
Keep us posted on your progress! You’ve touched a nerve!
I would look into incremental reading:
https://en.wikipedia.org/wiki/Incremental_reading
I have found SRS to not only work well for language acquisition, but also internalizing other models of thought– like principles for marketing, or design or things like that. The goal in learning should not be to gain a huge quantity of information, but to gain a huge quantity of usable information– that is, knowledge that you have internalized to the point where it can interact and react with the ideas you already know– this is where you can start asking interesting questions and discover interesting problems to explore.
Here is the common sense approach of a PhD student in Theoretical Physics:
1) Pick a paper or topic that you would like to understand.
2) Skim trough the paper in about 10 to 15 minutes
3) On a seperate blanck sheet of paper write down what seems to be
the basic structure of the paper. Do this say by deviding the paper
into seperate block and by assigning a headline to each block.
Leave some space under each headline.
4) Below each headline write down circa three naive questions
that you would naturally expect to be answered after having read
the specific block in the paper. You can do this without looking at the paper.
5) Do a 1 hour intensive internet research collecting all the resources they may
seem to be in some way related to the paper that you want to understand.
6) Print all the resources and then shut down your computer!
7) Now assign to each of the headline at most two (the less the better) resources
that may be useful in dealing with the topic and answering the questions that
you have written down for that specific block. Initially it doesn’t really matter which
resources you pick – just choose the ones that appeal to you the most.
8) Remove everything from your desk. Just take the a stack of plain paper, the paper you want to understand and your favorite writing instrument.
9) Read through block 1. Carry out all the calculations of block 1 and write them on one of your blanck sheets of paper.
10) If you get stuck write down why you got stuck on a seperate sheet of paper. Try to fromulate precisely what is was that you didn’t understand. Then keep going. It’s not a big deal.
11) On a third sheet of paper write down the answers to your initial questions and the main points that seemed to be important to you.
12) After this procedure you should have three sheets: One with the detailed calculations, one with the points were you got stuck and one with the answers to your naive questions and the basic points of block 1.
13) Go to your two resources and check if they help you with the things that you did not understand.
14) If there are still any stucks on your lists refine them after heaving read the two resources.
15) Scan all of your three documents and save them as seperate documents on your computer
or the wiki of your choice.
16) Do the same with block 2 , 3, 4….
17) At the end you will have a set of all the detailed calculations, a list of stucks and
answers to your naive questions as well as an overview of the things that you think are important.
18) Write an Email to the authors of the papers and ask them if they can help you with your stucks. Don’t waste time explaining the stuff to other people. The guys who wrote the crap are the that are responsible!
19) If there is still stuff you don’t understand ask your professor, go on the internet, ask in forums, etc.
I hope this helps and I would be happy to here about improvements and comments to this method.
Best.
Crowdsource
As a professor, you can crowdsource it by assigning the proof to one of your classes. Let them sort out the details of the proof and explain it to you. Or assign it to a graduate student in that subject area.
It’s too bad that authors of these papers do not offer long-form versions of their proofs online for people who want to understand each step of the process.
Is there a popular math board for academics, like https://stackoverflow.com/ for coding? Let other people help you chew through the details.
I think the first question is why you are reading a given paper.
Unless you are reviewing the paper, I would suspect you are either interested in the result or some of the analytic machinery within the proof.
Focus on those and not the rest.
Excellent post.
My first idea echos a lot of the previous ones. I had to read a lot of biology papers during my thesis work, and after absorbing a lot of papers, you could then get to a place where reading the abstract and then cutting directly to the figures, then popping around to the other parts of the paper as needed (having the PDF is useful here, for word searches), was the most efficient route. But this seems to be not directly applicable to your field.
My second idea, which I got from a friend of mine, is to request ALL the theses from the groups that are publishing in areas you are interested in. (You can I think order them online from UMI Dissertation Services). Theses are not word-limited, and also contain a lot of detail that doesn’t make it into final papers (sometimes brutally honest detail, like “here’s what we tried, but didn’t work…”. Note: the stuff that didn’t work, or is half done, is often a good nucleus to start thinking of additional projects…) My friend did this from all his major competitors labs.
My third idea is to form a small group of people to help dissect papers. We used to have journal clubs where one person would present a paper from another laboratory, in detail, once a week. That took a lot of the burden off of each individual person to fully dissect current papers. But in your case, it might be better to get a small group of people to tackle one paper, and each person could be tasked with trying to figure out a particular part of that paper.
My fourth idea comes from working with this organizational program called DevonThink. It’s kind of like Evernote, but it has more sophisticated ways of linking different ideas together. Basically what I do is cut and paste (or rewrite) interesting pieces from PDF’s I’m reading, put them in the DevonThink database, along with a citation to the original paper. In your case it would enable you to pull up definitions very quickly, and also pull up similar ideas that come from two different papers. If you prefer working out stuff with pen and paper, you can just take a photo with a good quality smartphone and stick it in that way (would work with a whiteboard as well). I got the idea for using DevonThink for my research from this article “Capturing Annotations To Enhance Scientific Writing And Knowledge Retrieval” on Jeff Taekman’s blog. What I do is a bit stripped down from that… I know you are loath to add technology for novelties sake, but I have been happy at being able to pull up information easily, which is essential for my “deep thinking.”
I face a similar problem– I spend a lot of time these days with programming language specifications. My struggles remind me of my grad school days working on papers, so I’m hoping our techniques correlate enough to be effective. Two attacks have worked well for me recently:
1. Expect multiple passes. Scrape out what understanding you can get on each pass, but don’t get hung up on what you don’t understand yet. Get as far as you can go with your current understanding, push a little, and then drop it if you don’t make progress. It feels silly, but I find is much more globally time-efficient than trying to grind it out.
2. Restate everything. Rewrite it in your own words. Rewrite the prose in formulae. Rewrite the formulae in prose. Especially useful after a handful of passes (see point 1). Then rewrite those restatements on later passes.
If you really, really need to understand it in depth, play editor. How would you improve the paper? What would you cover first? What intermediate steps are important to the reader?
(That brings up a point hinted at earlier. Who is the reader and why are they reading?)
As for them not being well-written? Psych studies have found that hard-to-read fonts and badly-written texts actually help students learn. They can’t just skim the nicely-written summary, flip to the glossary when needed, speed-read the intermediate steps, and admire the pretty patterns the symbols in the equations make. They have to summarize the chapter, make and test the definitions, think about and write the intermediate steps, and see why the symbols follow patterns. (Numbers dance for me — a choreographed and complex dance rather than can’t stay in their places.) This assumes, of course, that the student is willing to put in the time.
The studies I have seen measure whether people remember things reasonably soon after the interaction with the text. The studies do not make claims about improved learning and understanding and they do not compare the results with the alternative which would be the same amount of effort spent on a well-written text.
I have been researching the exact same topic, because I am also realizing that my speed of comprehending new material can often be very slow.
I am a software engineer, but at the moment I am trying to learn the basics of philosophy (different from math, but some principles of comprehension may still apply). After reading some text, I feel foggy in my head. I kind of grasp what I read, but it’s not clear, like a low resolution TV with a lot of disturbance.
What is helping me is to keep notes. As I understand one fact I write it in my own words. So I am almost writing a parallel paper to help me follow along. The second paper is off course very compressed, contains main concepts only and is like a support to help me walk along.
Perhaps this technique could help you. Do let me know how it goes.
I too get tired of fighting my way through journal articles. The problem is that the traditional way of writing math uses a linear medium for what is really a hierarchically structured argument. Imagine reading a computer program line by line until you get to the end. Nearly impossible. What if every time a nonstandard notation was used it was hyper-linked to its definition? What if the logical structure of a proof was made explicit? I address these ideas very crudely here: https://jrh794.wordpress.com/2013/05/26/proofs-escaping-print-media/
This is just pie-in-sky thinking since I don’t make my living reading or writing proofs.
I’m a freelance editor for computer science papers (you know, the person Elsevir and Springer refer you to when your English is too poor for peer reviewers to review). I get a wide range of research topics, some of which I know nothing about, all written in eye-watering English, and usually less than 48 hours to turn it around. I’m so excited to see this project, and I look forward to any results (especially ones I could use as well). Here are my techniques which may be more or less relevant.
I don’t skim. I tried, but it only lengthened the time I spent confused by what was really going on. At any rate, the abstract should summarize sufficiently. Instead, I lay into deep reading/editing straight away. If I get stuck, I physically highlight it and go on. If I don’t do that, my mental stack of stuff to worry about gets too big to handle. If I need background material, I find key phrases and stick them verbatim into google. This usually brings up alternative explanations more quickly than hunting down references, review papers, or the authors. I use a Word macro to extract all the acronyms and symbols and then go through them one by one. Authors like to define acronyms because it sounds scientific, then forget (for whatever reason) and go back to the original phrase. Meanwhile, the reader is lost!
I don’t verify or edit the math beyond simple consistency checks with the text. It’s not my job. But I have noticed that after I make one deep pass through the text (at between 1000-4000 words per hour depending on the language) I can read the text again and understand the research to a surprising degree. We all know that plain English and active verbs are great, but until I learned how to do it (and it took a good six-nine months despite a PhD and published papers) I had no idea how powerful clarity could be.
I have a lot of difficulty maintaining the intense concentration needed to make that first pass. At the moment I go in chunks of 15 minutes with 2-3 minute breaks in between and mark my word count for each chunk. But I also have a wide range of sophisticated procrastination techniques and plenty of other stuff to do in my life, so beginning those 15 minute chunks can be hard. 🙂
Again, I look forward to reading about your progress and conclusions, Cal.
Kim, this is fascinating — really, one of the best blog comments I can remember reading. Thank you so much for posting.
I’m very curious about how you learned to do this. What are the principles you use to make the writing clearer — besides things like consistency with the acronyms — and how did you learn to employ them in practice?
Cal,
I think a great deal of the advice here is good in general, but I’d like to highlight what I think are some relevant factors of your work in particular:
1) The number of people who have seen and/or understand the paper already are likely to be few, so the efficiency of consulting others relies on their ability to do this faster than you.
2) You probably already know as much context as you’re going to when approaching the problem. Unless you’re reading papers way out of left field, I would think you’re as prepared as a person could be when going into a paper.
This narrows the scope of the problem and solution, because we can’t do more to prep and we can’t reliably reach out to others.
Just a guess, but I would think there’s a method of simultaneous note taking that would be helpful here. Perhaps writing out explicitly questions as you go along so you can be confident you’ll address all the vagaries before you move on from the paper? If the mental strain of keeping too many intermediate claims in mind is the problem, then writing them down in some fashion seems like a potential answer.
It also might be helpful to take a brief stab at how you would’ve solved the same problem yourself before trying to digest their solution, just to get in the right frame of mind.
I read through all comments and my blood pressure was increasing for almost every comment that I read. The advice given is good but the level typically targets undergraduates or early Ph.D students rather than someone who is 3+ years past their defence. David Wynn politely summarised things in comment #41.
There is often one or several “tricks” that make proofs go through, either in the problem setup or in the solution. I try to identify the idea(s) behind these tricks before jumping into the theorems and their proofs. Sometimes it is enough to know the trick to construct the theorem and/or proof yourself. Sometimes the authors explicitly state what they consider to be the key insight, and that saves a lot of time compared to reverse engineering the insights from the partial proof sketches.
Technical papers often have notation that fills the pages with ink. Bad notation hides important things and highlights useless things. I try to write down the same logical system that the paper presents, but changing some of the notation to use less ink. It does not matter which part of the notation I start to simplify, the important thing is to start with something. The outcome is one of the following:
1) I almost immediately get stuck because my chosen simplification can not express a particular inference rule. This tells me why a certain annotation is absolutely necessary and where it is used. Make a note of that. Restart the process.
2) I pinpoint a specific small part of the notation or system that I don’t understand. I try to find the answer to that question in at most 5 minutes. If no answer before timeout I make a note that I don’t understand this part. Restart the process.
3) I end up with a slightly simpler system that looks a bit less daunting. This means I uncovered some accidental complexity in the system presented. Use this simplified system from now on. Restart the process.
The same method can be applied to definitions and theorems. If the paper uses R for natural numbers: rewrite the same definition using the standard notation.
For me it is important to do this on a blackboard or with pen and paper. Writing on a computer does not force me to work with the material in the same way. Typesetting math is also dreadfully slow and I end up wasting time on looking up the perfect kind of arrow and how to make that arrow just a little bit longer.
If the paper is terrible and the above things have not worked but I still have not given up on the paper I might rephrase important paragraphs using my own words. That usually highlights gaps in both the paper and my understanding. This is my last resort and if I still don’t understand the paper after that I will try to discuss it with someone at the next conference. Very few papers are important enough to warrant this kind of attention for me: I could have spent the same amount of quality time and have a deep understanding of several other important papers instead.
That was my reading tricks. What are your writing tricks so that you do not write one of these papers by accident?
Would it help to mind map or outline the paper first so as to gain the whole structure? As an actuarial student who had to read many dense actuarial articles, it was fairly simple to get hung up on one sentence/typo in an actuarial paper. Maybe by organizing the paper into an outline/mind map and having a simple index card with 3 major points you need to get out of this article before you read it, you can be more focused. my 2 cents. I am very interested in the other comments as well.
Hello,
We are a not-for-profit educational organization founded by Mortimer Adler and we have recently made an exciting discovery—three years after writing the wonderfully expanded third edition of How to Read a Book, Mortimer Adler and Charles Van Doren made a series of thirteen 14-minute videos—lively discussing the art of reading. The videos were produced by Encyclopaedia Britannica. For reasons unknown, sometime after their original publication, these videos were lost.
Three hours with Mortimer Adler and Charles Van Doren, lively discussing the art of reading, on one DVD. A must for all readers, libraries and classroom teaching the art of reading.
I cannot exaggerate how instructive these programs are—we are so sure that you will agree, if you are not completely satisfied, we will refund your donation.
Please go here to see a clip and learn more:
https://www.thegreatideas.org/HowToReadABook.htm
ISBN: 978-1-61535-311-8
Thank you,
Max Weismann, Co-founder with Dr. Adler
Max,
I looked at the titles of the sections in the link you provided. A mathematical paper is probably not a story in the sense that your videos refer to. Could you elaborate on how those videos are applicable to the objective of reading mathematical proofs faster?
The paper may make reference to other papers that provide details the short paper wasn’t able to provide. Alternatively, if that short paper has already been cited, the citing paper may provide a better explanation.
In a similar vein, you could search for papers that use the same or similar concepts. This approach operates under the assumption that the steps being skipped are “standard,” at least in some field. The process of understanding the short paper becomes a matter of seeking out those papers dealing with similar mathematical tools and filling in the gaps.
Another approach is the following: if the question about the paper is specific and the author is a grad student/postdoc, you’re very likely to get a response to help fill in the gap.
Cal, have you tried incremental reading? It’s a valuable tool that the software supermemo has. Have a look 🙂
I think the majority of comments here make good sense for difficult readings in other areas, but not really for mathematical proofs. It’s tempting to say it’s the same, but I believe there are more differences than similarities. Mathematical proofs are not just really hard writing; it’s a difference in kind, not degree.
There seem to be two reasons for the difference: one is that the structure of mathematical proofs is far tighter than anything else, except maybe coding. The other is that, more than in any other type of writing, the mental process of doing and understanding the proof can diverge widely from what is finally written in a paper.
I have long been frustrated with how slowly I read proofs. I suspect that problem is (for me) unsolvable. However, I think speed and number of proofs read are the wrong metric; I think a better metric is, will I come away with new ideas, or new tools in my box that I can actually use on other things? In many instances I have spent a long time figuring out a paper, but never really internalized it, so it wasn’t actually useful to my work in the end, beyond simply knowing the results.
Measured by the latter metric, the most successful thing for me has been to determine what the authors are trying to prove, and then prove it myself as much as possible. And to really spend time on it before going back to their paper for hints. This lets me figure out what the key problem is, in my own internal language; when I return to the paper, I know what I am looking for. The mistake I often make is giving up too soon. The longer I spend working it out myself, the more I get out of it, and the more able I am to use it. It’s sometimes slow, but everything I’ve read and actually used has been read this way.
There’s a quote from Bill Thurston someplace where he says (roughly), when you buy a new toaster and it comes with a thirty page instruction manual, you don’t start off reading the manual cover to cover. You start trying to toast things and see if it works like toasters you’re familiar with, and retreat to the manual when something is confusing.
So, I’m a freelance editor, the kind that Elsevir and Springer send your stuff to when your English is too poor for peer reviewers to review. My remit does not quite extend to re-deriving proofs (thank god) but I must understand the paper and its methods/claims enough to ensure my edits retain the author’s original meaning. I only have only a few hours to do this, and often the subject is unfamiliar and the English is eye-watering. I’ll share the shortcuts I’ve learned, they may or may not extend to proofs (sorry there, Sam) but might be worth considering.
1) Scanning/pre-reading to get an “idea” of the subject is a waste of time (at least when I’ve tried it). The abstract summarizes the paper sufficiently, any thing else is procrastination.
2) Reviews and related papers are also an inefficient way to understand a specific paper. If I really need an alternative explanation, I google phrases from the paper verbatim and scan what Google Scholar throws up. Quickly.
3) As soon as I see inconsistent terms/acronyms used, I fix them throughout the paper before going on. I have a macro that extracts acronyms to a list. This sort of step is like the pre-sorting you do with puzzle pieces (all face up, edge pieces separated) and I imagine is critical for proofs as well. Authors are slack, for whatever reason, but at least this kind of confusion is easy to fix.
4) I deep edit all the way through once. If I don’t understand something, I highlight it. Usually on my second and final read, the ambiguity is cleared up and I can easily fix it. The physical highlighting assures my monkey brain that I will not forget it and keeps me focused on the present.
5) I go in 15 minute chunks with a timer. If I get distracted I notice it and mindfully turn my mind back to the paper, remembering the timer. In between, I take 2-3 minute breaks. Any longer and I lose the thread. I have to do this because research isn’t always the most exciting of reads – especially when lol-cats are only a click away. I’m lucky in a way because I am paid per thousand words and have hard deadlines, so the effort of intense concentration has immediate rewards.
Looking forward to your conclusions, Cal, and any further tips.
Hi Cal, have you read Bill Thurston’s thoughts on learning and communicating mathematics?
https://arxiv.org/abs/math/9404236
1. Identify and articulate the specific level of performance you are after.
2. Identify addressable people who have attained that level or beyond
3. Identify the skill sets that those people use regularly and are key to success (by talking to them)
3. Pick the most important skill from that list you can practice
4. Break that skill into to smaller sub skills AND the method you think improves them
5. Prove them out by getting feedback from ppl in step 3
6. If good, set a practice schedule practice only that skill in focused bursts
7. Measure progress to goal in step 1