Using Tiddlywiki for project management and data analysis

I have been looking for a simple and flexible tool to manage projects, take notes from reading (e.g. literature and online), and track results of data analysis, e.g. EverNote, OneNote, Trac, JIRA, github, RMarkdown, etc. The tools are designed to develop software, but too complicated for daily usages. Recently, Tiddlywiki and Projectify pops up from random goggling. Tiddlywiki is a non-linear notebook for storing information. I found it is very useful and simple to manage my research projects and take notes of data analysis as a data scientist and digital agronomist.

Project management

I always work on multiple projects at the same time. Beyond the project management by project members, I prefer to keep a personal project management to track, hold and restore my activities related with a specific project. Projectify is a preconfigured Tiddlywiki for project management to capture thoughts, plan projects, and schedule tasks. Projectify manages projects through tags in Tiddlywiki. Adding tags into the new tiddler can easily associate it into one or multiple projects .

Reading notes

Literature and online materials are very important to scientific research. Refnotes plugin can add reference into any tiddlers using bibtex format. My reference management tool Zotero with extension Better Bibtext can easily export references in a collection into bibtex format and keep updates.

My notes are managed by tags and list according to topic.

Data analysis

R is the major programming language for data analysis. RMarkdown has been using for long period to document the data analysis and share outputs with my colleagues. [RMarkdown] documents provide powerful features to reproduce outputs of data analysis and very suitable to generate final outputs, which can be treated as a linear procedure from raw data to output. As a data scientist, however, I often branch data analysis to

  • Get new data from my own experiments or other colleagues. The new data require for quality checking and might or might not be useful for further analysis.
  • Test new ideas from the current analysis (branch from master analysis). The new ideas might be brilliant and merge into master analysis, or might be rubbish.
  • Check the previous analysis for any strange results. The checking procedure might find an error in the raw data and scripts which introduce a new fix into master analysis, or new understanding for science.

It is unreproducible, time-consuming and even impossible to write RMarkdown file for each branching data analysis as 1) the modified or dropped input files, 2) the too big intermediate results, 3) lost tracks of R scripts, etc. Branching data analysis is very common for scientific research and non-linear procedure.

As the way of natural thinking, the feature of tiddlywiki to create missing tiddler is very easy to branch master data analysis without leaving current tiddler just through using two brackets (e.g. [[New Idea]]). Tiddlywiki shows a link to create new tiddler.

Bangyou Zheng
Bangyou Zheng
Data Scientist / Digital Agronomist

a research scientist of digital agriculture at the CSIRO.

comments powered by Disqus