Beyond Databases
Back in the early days of Study Hacks, I introduced the paper research database. The idea was to build a database of every quote you might need to cite in your paper. These citations could be sorted by date or type, and be linked to their matching source. The technique works because it helps you build and organize a comprehensive understanding of an event or idea before you start writing about it.
I should be clear: I love this technique. I used it to write two massive art history research papers while here at MIT. Recently, however, when I began the research process for my new book, I found myself drawn to a new strategy: the paper research wiki.
In this post I want to explain this approach, which has the potential to significantly improve the complexity and confidence of your written arguments.
The Basics of a Research Wiki
The research wiki I’m currently using is not my first attempt at this strategy. In fact, it’s the fourth research wiki I’ve started. The first three quickly faded in disuse. This last attempt, however, has become an incredible aid to my writing.
What’s the difference? It all comes down to structure…
If you jump blindly into a wiki, and start creating pages left and right, you’re unlikely to gain much benefit. When I attempted this approach with my first research wikis, I ended up building a page for every idea or piece of information, with few internal links. The site soon devolved into an overly complicated, wannabe notebook.
For my latest wiki, by contrast, I enforced structure. Specifically, I introduced a strict information hierarchy:
- At the bottom level, there are primary sources. Above them are second-level structures. Above them are third-level structures, and so on.
- My linking rule is simple: pages can only link to those from a lower level. Primary sources cannot link to any other pages. Second-level structures can link only to primary sources. Third-level structures can only link to second-level structures and primary sources. Etc.
As you ascend through the levels of this hierarchy, you increase the complexity of the ideas being captured. For example, here’s a screen shot of the home page for my book research wiki:
Notice, I have two types of primary sources: interview subjects and research papers. Each interview subject has his or her own page where I capture all of the relevant information — from contact information to interview transcripts. Each research paper has a page with a full citation and summary. These are the foundational blocks on top of which everything else about my book is built.
My second-level structures are ideas. For example, if you click on the ideas link you’ll see a list that includes the Failed Simulation Effect. The corresponding page describes the idea, linking back to the relevant primary sources, including the relevant research papers and students who exhibited the effect.
At the third-level, I have annotated outlines for each of the major parts of my books. The annotated outlines link heavily to both ideas and interview subjects.
When it comes to writing a part of the book, I can start with the relevant annotated outline and quickly drill down to the needed information. As you might imagine, this allows me to write with great confidence.
Applying the Technique to Your Paper
For a standard college research paper, I would suggest the following information hierarchy (this is only a suggestion, feel free to modify as needed):
- Have your primary sources include the actual primary sources: books, articles, interviews. Create one page for each such source. Include on the page the properly formatted citation and a list of the relevant quotes you might use from the source.
- Have your second-level structures include events and ideas. Create one page for each of these items. On the page, you can link every quote and fact in your description to the matching primary source.
- Have your third-level structures capture timelines and comparison charts. For example, you could have a page that orders and dates a sequence of important events (linking each to its matching second-level page), or a page that compares different related ideas (linking to the matching idea descriptions).
- Have your fourth-level structures capture large arguments. Here you can draw freely from all of the lower structures.
Notice, this wiki is different than an outline. Starting from the fourth-level argument pages, you should be able to easily drill down to the primary sources needed to build a standard flat outline. In other words, put most of your thinking into the wiki, then generate the pre-writing outline at the last minute.
The Advantages of a Wiki Approach
Wiki-driven writing enjoys two important advantages. First, the structure of the wiki helps you structure your research. Plugging your research into a clear information hierarchy is superior to simply creating a large pile of stuff. Second, working through these different levels forces you to do lots of high-level thinking before you get to the outlining and writing phase. In some sense, your paper research wiki requires you to master the nuances and complexity of the topic before you think about what you want to say about it. I can tell you from experiences, this is the approach that generates A* results.
Finding a Wiki
If you’re tech savvy, you could potentially setup your own wiki on a personal web hosting account. But I suggest just using PBworks (formerly PBwiki), which has every feature you need, works fine, and is free. It takes roughly 7 seconds to setup a PBworks wiki, so the required effort is minimal.
Practice Makes Perfect
It takes a little time to customize this technique to suit your own tastes. But if you’re serious about producing high quality papers, then I highly suggest experimenting with this strategy. I’ve been loving it.
Cal,
Great idea. I might try this for my dissertation research, which I’ll be starting over the summer. Seems like a much more manageable (and useful) system than my binder filled with my thesis research…
Jason
Oh my god, is that research for a new book I’m seeing?
Oops, sorry, I jumped ahead to the image without reading the entire article first.
Cal, are you familiar with the index-card method for writing research papers (easily adapted to MS Word), which I see, in many variations (some better than others), in a lot of how-to-study books?
Essentially, you write an outline for your paper (developed after scouring sources), you create a numbered list of your sources, and then you gather your data on “indexed” index cards (i.e., a single quote or fact on an index card, labeled with the source you found it in, the section of your paper it belongs to, and a little subtitle about the subject to which it pertains; some index cards basically just have a personal idea from your head). And then you arrange the cards in order and write the first draft of your paper from them.
I see this method suggested commonly in study-skills books, and am actually okay with it tho at most my papers are 30 to 40 pages long. Yet I encounter so many other suggestions on how to write research papers and wonder if they are actually an improvement on this method, improvements that I have failed to recognize. What is your evaluation of this technique? How does it compare and contrast with the techniques you espouse?
Wikis are still too cumbersome. There are much more user friendly software packages for this job. I like most the Evernote, which works and syncs on all platforms you may want to use.
My basic problem was that I wanted to progress to the next stage of the research process, before I have finished the preceding one. I wanted to achieve too much too quickly and end up with nothing. To solve this, I looked for a process that would allow me to:
– store structural ideas while doing the reading;
– note various ideas for possible structure, while linking to them snippets from readings
– trust that no idea will be lost.
Wiki would be one option, but it is often too cumbersome as one needs to think about how to record the idea not just simply record it. With Evernote this is much easier. Here is my current workflow for a literature review:
1. Create a separate Evernote account for the project
2. Gather all literature sources, create a new notebook for each source and drop all files there (you can also clip from web, mail yourself ideas, note them to your mobile it is all synced).
3. Read each source and create Notes from all potentially useful bits of information (select text + double-click CapsLock – that is all it takes to make a note from any PDF or webpage – much easier than wiki)
4. attach tags to all Notes; each Note may have multiple tags
5. Tags can be arranged into a hierarchy that gradually morphs into the structural outline of the paper. This is a key! Tagging and retagging creates the structure from a pile of clippings.
6. After all sources sorted and with workable tag-based outline, go through all notes – merge, delete, re-tag etc.
7. You are ready to open word processor and start writing, and copy-pasting (you can also export en masse from EN, but I did not find this useful).
This workflow mimicks the logic of the qualitative research process. On the pro level, this is usually supported by expensive software packages like Nvivo or Atlas.TI, but it can be done in Evernote, which is free and much, much more lightweight. People on Macs swear on DevonThink, but I run on Windows only.
Cal, thanks for this helpful article!
Personally, I’d be interested in getting an insight into your whole workflow when writing the two books mentioned.
And although I enjoy reading your blog, it seems to me that you often choose commercial solutions over free and open source alternatives which sometimes are even better.
E.g. as a wiki engine you could use tiddlywiki which runs from your harddisk or usb pendrive (https://tiddlywiki.com/) and can be kept in sync with an online version at tiddlyspot.com (where you can also collaborate with others, access earlier versions of your wiki etc.),plus you don’t need to be tech savvy for that.
Another great alternative would be the zim desktop wiki (https://zim-wiki.org/), a cross-plattform desktop wiki with versioning, to-do features, calendar and so on.
Overall, I’d like to see a greater amount of open source and linux related content as this certainly meets the financial needs of (doctoral) studends. (like this: https://academiclifehacker.wordpress.com/2009/02/19/academic-software-roundup-for-linux/)
Great tips, nonetheless. So thanks again,
Keith
I highly recommend it. At first, you’ll be a little frustrated that it’s taking you a little bit of time to enter things into the wiki. But once it grows past a certain size, you’ll start to love the structure it provides.
I’m planning on posting some more information and updates on the project soon. Stay tuned…
I think the paper research database, in some sense, is like an advanced index card system. The wiki, by contrast, adds structure and high-level thinking in a way that a primary source-centric approach does not.
This is insightful. Overcoming this issue was one of the most important things I’ve learned to do as a student interested in producing good writing.
That’s what I like about the Wiki, it forces me to spread out the thinking over the whole research process. It’s slightly more cumbersome while processing sources, but you end up performing a *lot* more thinking in total, hopefully generating a better result.
That being said, I’m not that familar with Evernote, so thank you for providing a good tutorial on how a student might use it as an alternative research system.
I tend to go with whatever is available and easy. I use pbworks because it was free and fast. I use excel and word because it’s already on my machine, etc. I rely on slightly more savvy readers like you to point me toward superior alternatives. Both tiddlywiki and zim sound cool. I’m going to check those out.
Great post, Cal. I’ve used pbworks for two other research projects in the past. One was a literature review and the other involved program development. Your tips on how to structure the wiki and the value of the thinking you do as you structure it are right on.
The additional value of using a tool like pbworks is as a team project management tool for research projects where you’re collaborating with others. The lit review project mentioned above involved two other students; we had widely varied schedules and had a hard time finding one day and time to meet that would work for everyone. The wiki allowed for us to do our individual work relative to the project, upload it to the wiki, and not experience any overlap. With three of us forcing ourselves to use the same logic and structure of the wiki we found ourselves easily following each other’s work. Of course, the wiki didn’t completely replace meeting in person. But it did help create a common organization for the project between all of us, keep us all marching in the same direction, and made it easy to transfer all of our work to our professor in one nice package when we were done.
The wiki also allows everyone to know who’s doing what, when, and how much. Therefore, it has a built in accountability function. Using it helped cut down on the social loafing I’ve experienced in the past from other group projects (and been guilty of myself).
I was in the process of getting another wiki set up for a new team research project and will add your suggestions into the structure.
Thanks again for the post.
So why don’t you just use endnote or even evernote? To me it would seem like an easier solution as most of the databasing can be done through the program. Just a thought.
Coincidentally, I was just assigned a paper that this wiki idea is perfect for. I’m going to try it out. Thanks.
Oh, beautiful. Just what I was looking for to do exam notes for law.
And the reason I think it will work really well for those as well as research papers (in case you’re interested) is basically because the structure of law school in common law countries is a cause of action in each class, or one element of a cause of action, with a lot of cases that illustrate how they work in particular circumstances for reading. Exams are a hypothetical fact situation. Generally you pass if you can work through the steps to determine whether each cause of action will work in the given situation, and you get the A* if you can comb through the detail and facts of each case to really analyse how relevant a legal principle is to the ones you’re looking at in the exam. I’ve never really been able to map any of your other study techniques onto law school without big tweaks because they just don’t fit the style, but this is probably something that could be maintained on a daily basis – you do the wiki pages (cases, legislation) for level one throughout semester, review once you’re done with a particular limb (one or two classes) by putting it into a level 2 page that explains this element, review once you’re done with a page that outlines the whole cause of action (maybe a month) with a level 3 page, and then stick the whole thing into a level 4 page at the end which basically lists them all and maybe gives some guidance on which one to go for for a given exam situation.
Cal,
Great post. I’m going back to grad school for another master’s degree this Fall and found the Red Book about a year ago. I wish I had had it in undergrad and during my first grad degree!
I’m curious how using the research wiki might fit into the “process” for writing research papers in the Red Book. Any guidance?
Thank you for your time and effort! It’s been a wonderful learning experience for me.
Yours,
Jason
A couple of other recommendations for personal wikis.
Firstly, I wouldn’t go with tiddlywiki for this as it gets bigger.
Better alternatives.
Wikidpad, which is free and is very powerful.
However if you have the money, I’d spring for the excellent ConnectedText, which is very powerful, fast and nice to use.
This seems like a great idea to bigger projects.
I’m so happy I found these kind of blogs, before I’m joining University next year, all these ideas, all these disciplines. I’m almost looking forward to a year of hard studying!
great post, thank you. I’m totally on your side. I love wiki too.
Cal,
This looks great!
One question – How do you pull together your bibliography when the time comes? Certain classes or professors require different bibliography styles – so would one copy the information from the Primary Sources in the wiki to another database for formatting into the proper style, or somehow make the wiki do the work?
Thanks!
Based on your screenshot and the ‘ideas’ link you mentioned in the post, I gather that the actual structure of the wiki is:
1)a wiki-wide directory
2)a level directory
3)the pages themselves
In other words, the wiki hierarchy is not just 4 levels high but three levels deep.
Does PB’s tagging option come in handy, at all?
Something like that seems to work well. Though it’s not fixed in stone.
I haven’t tried the tagging option yet, as I haven’t really seen the scenario where it would help.
Noticed that tagging a PB page ‘template’ makes it a template next time you use it, which is handy.
Thanks for the wonderful post, btw. I think this ‘node-based’ wiki approach (embedding a link page, and adding the links/pages as you go along) let’s you, the researcher, bring the complexity to the project, rather than imposing a structure a priori on your work. The underlying organizing principle being radically simple, but not simplistic.(Hearkens to mind an another MIT figure on language).
Can’t wait to see your ideas on the doctoral dissertation, but hurry my defense is coming up : )
At risk of seeming like a blog-hog, I’m posting again to give a little dirt on TiddlyWiki, which I found a better solution to PBworks. Wanted to illustrate what Keith posted about not needing to be “tech savvy,” because I surely ain’t:
My two step / two second solution to Tiddly-research-Wiki creation:
1. Write a wiki-wide TOC (table of contents) in the main menu on the front – and only – initial page,
2. Create the corresponding pages by adding double brackets as you write the page names inside the TOC.
That’s it, ya’ got yerself a wiki.
Btw, yes, PBworks can work this way too, but, it was significantly slower because you’re online with a delay (up to 7 seconds on my count), and it had too many clicks to get to the same point as Tiddly. If you further arm yourself with a cheat sheet here [https://www.linuxbeacon.com/doku.php/cheatsheets], and a FAQ there [https://www.giffmex.org/twfortherestofus.html], both for absolute beginners since there’s a lot of documentation, you can even do advanced things right off the bat (text formatting, printing, tables).
Hope that helps,
/Cas
I’m off Wikis altogether and on to Onenote! Just curious if anyone else has tried to apply the above system to onenote, and how it worked out?
For those of us stuck with windows, I can’t imagine a more perfect solution. In fact, I had no idea what to do with it until I started adding structure as per the description in the above in the post. Your wiki-like onenote pages because a virtual sandbox for ideas, but the nuts and bolts of organizing a ‘spine’ of citations and concepts garnered from texts can be done in a quote table supporting it.
Since its all integrated into MS Office, you can have your database cake and eat it (or your wiki at least), too.
Thanks for the great step by steps on how to create a research paper wiki. Well written.
your blog is an answer to a most desperate prayer. I’m a 3rd year “grad stud” working on a phd in biomedical sciences. I passed my orals in august, got married two weeks later and nothing in the lab has worked since… almost daily i pound my head against the wall asking why nothing works. i now have a few ideas how i might improve-but any additional ideas would be greatly appreciated.
Thanks for this wiki post, this is great! I have been all over the place with my dissertation notes and all the word docs I use to house them, and this sounds like a great way to organize and structure things better. I’m not at all familiar with wiki, but I’m going to go give this a try. Thank you!
Hey Cal! Great post here. I have one question though. I assume that due to your math background you are familiar with LaTex. I just started grad school and am brainstorming some ideas for paper tracking. I was wondering what you thought the advantages of a wiki were over say creating a directory structure and tex documents.
I like tex because eventually that is how all my papers and writings will be formatted and published. I also enjoy a good afternoon of vim and tex writing as well, but I see some of the positives about using a wiki. Most notably the fact that you can easily view and navigate the content. What argument would you provide in choosing the wiki over something like a good latex set up? I would also be interested to see your response to the comparison of using Evernote rather than a wiki as well.
Thanks for any comments and suggestions!
When working on mathematics research, I do use latex documents — one per potential problem — as my central organization tool. As you note, the ability to easily render math is essential for this specific type of work.
Great system! I think I’m going to start learning how to implement that for my senior high research paper, so that when I’m actually in college I will have the system down good!
Love it! Just implemented it and blogged about it.
I have the same question as Jason–how does this fit in to or correspond to the method in the red book? Do you not make photocopies in this method? I’m a beginning PhD student and just read the red book because I really like this blog, but I’m not sure if the photocopy method would be practical given the number of sources in my research papers. But maybe that makes it even more practical, I’m not sure.
So now I have gone through all of my sources and extracted the relevant information onto different pages in PBworks. I’m ready to start the structural work, but I’m frustrated at the prospect because many of my sources covered more than one of the ideas within my paper topic. So, when I link back to the primary sources, I will have to comb through the irrelevant quotes for that topic to get the relevant ones. Is there an aspect of this system that I’m not understanding? I’m starting to like your research paper database idea better. Perhaps it would work well to build a research paper database for primary sources, then create a wiki for the second and third levels,pasting the relevant quotes into the ideas and outline pages.
This is such an amazing approach to organizing the paper writing experience. I don’t know how I ended up on your site Cal, but my mind is blown. I have no doubt that your writing wisdom will change the way I approach my writing. As far as using a wiki…I’m very tech savvy, but I agree with the others that think starting a wiki for this task is just overkill. If you’ve got a mac, save up your coffee money for 2 weeks and get Scrivener. Does everything you need a Paper Research Wiki to do, all right out the box, no setup required, AND you can set up multiple ‘wiki’ styled researched situations within a single document. I don’t work for the Scrivener people or anything, just sharing what worked for me.
Here’s a slightly rambling but still great youtube video applying the research wiki technique using Zotero (a free addon for Firefox) which was created by academics for academics to make research easy. It’s clearly not just a reference manager. I found it very helpful to see the wiki strategy in use, rather than reading about it. It’s much clearer to me how to go about it now. Interestingly, he says that he’s sure that he has “radically oversimplified” the strategy from Study Hacks, but I am not so sure since the philosophy here is that of professed ‘radical simplicity’.
How does your use or application of the strategy jive with this one? Does any of you use Zotero for building a research wiki, or use it but organize things differently? Please share, especially if you have a video, screen shots or anything tangible. A picture is worth 1000 words.
Please forgive my old-school ignorance here- I am not as tech-savvy as the rest here: What is the advantage of wikis over documents in Microsoft Word and just making folders and sub-folders for related ideas? There must be a difference (other than the graphics aspect) that I am missing…thanks
Hi Cal,
I was wondering if you can recommend a particular program for organizing notes on research papers. Both in terms of organizing research papers and notes written about them.
Cal, do you think I could see your wiki myself? This is brilliant stuff, but I want to compare yours to mine manually to see if I need to fix anything.
We might be a little late to the party here, but I think we have created something that would interest you. Ref Ren allows users to create a bibliography citation on their mobile device by converting highlighted text. That citation is then saved where it can be saved, edited and exported for later use.
I’m writing a 10 page research paper on Joseph Stalin and there is so much info on him i’m not sure what’s important and what’s not anymore. And also what would be a good thesis? Mine right now is how he affected the soviet Union and the world during and after WWII, but I’m not sure if that’s good enough.
Now in 2022 the options for building a personal wiki are massive. Are you aware of this? Have you a personal one?