Denote Project Tasks
November 3, 2024Tracking project-specific tasks is something I've changed many times over the years. I've tried apps like Taskwarrior (which I loved), using in-line TODO comments throughout my code and collecting them with something like magit-todos, to now just a strategically-named plain org file I can just spawn when I need it. And recently I got a bug to try something new. I'm writing this post in parallel of me trying to implement it so it follows my train of thought and trials.
The idea
Use features of Denote to track all tasks related to a project where each task gets its own file. Org has a cool feature that Denote supports called Dynamic Blocks (dblocks) which allow you collect content from various Org (Denote) files into a block. Very nifty. I wanted to utilize this to enumerate all the pending tasks on a project (or maybe other "views" as well) based on the project-specific tasks.
Implementation Thoughts
I imagine that a main project file that contains the aforementioned dblock(s) could be tagged (keyword) with project and individual tasks simply task. But that begs the question… how do we know which task's are associated with which project? I've thought a little bit on how I could do this and here are some rough ideas I've come up with before I have written any code:
- I could manually link them in the dblock by utilizing some capture functionality that can append entries to the dblock. This would be very complicated and I can already foresee a ton of problems and would require me to use specific entrypoints to manage my tasks. That's not fun. The beauty of plain-text is being able to treat it as plain-text and everything still works. Doesn't seem viable.
- I could create a bespoke silo for each project. I kinda of like this idea because it would simplify a lot of stuff (there should only be one
projecttag). A downside is it removes it from my main note-taking "database". I don't really like that since some of my random notes I've collected might be useful, but ultimately seems like a decent solution. - I could hack some "sub-tag" convention where a project could have a generated value associated with it, like
project:1and associated tasks would havetask:1to tie them into that project. One question I'd have to answer is how do I find thisproject:#file while I'm actively editing the project? I could use some dir-local to set that as a local var, use a hash of the project name instead of an integer id, or maybe use a symlink into the project. Ultimately, I like this idea because it allows me to keep my project notes in my main database! I think it'll be my first attempt.
Let's start coding
I already have a command (pg/open-project-notes) I use to open the notes for my project. It's just a plain Org file, like I mentioned previously, so I need to tweak it a bit to use the new scheme. I generally create "workspace" directories with potentially multiple git repositories within them using my own treebundel package. Since this root workspace directory is flat, I can be pretty sure that the directory names within there are unique, at least as much as (denote-sluggify-title) will let me. I think I'll use this workspace name as my unique identifier… But now that I'm actually hacking on the code, I realize that I don't have any way to create a name for the note. Previously, I use the workspace name for the title, so I could keep that and leave the project tag the same… and nothing actually changes! Here's a snippet directly from my dotfiles.
(defun pg/open-project-notes ()
(interactive)
(when-let* ((workspace-name (or (when current-prefix-arg
(treebundel-read-workspace))
(treebundel-current-workspace)
(treebundel-read-workspace))))
(if-let ((project-notes (denote-directory-files
(concat "^.*" workspace-name "__project.org$"))))
(find-file-other-window (car project-notes))
(denote workspace-name '("project")))))Dynamic Blocks
Now I need to populate the tasks with dblocks. For now, I'll just manually create some dummy tasks for my "dotfiles" workspace for now. The files look like this and each have a simple Task # line in them:
20241030T200925--dotfiles__project.org 20241030T200935--dotfiles__task.org 20241030T200941--dotfiles__task.org 20241030T200946--dotfiles__task.org
Simple enough. Let's try to insert a dblock using (org-dynamic-block-insert-dblock). I select denote-files as the first option, --dotfiles.*__task as the regex match, and sort with identifier and it generates this:
#+BEGIN: denote-files :regexp "--dotfiles.*__task" :excluded-dirs-regexp nil :sort-by-component identifier :reverse-sort nil :no-front-matter nil :file-separator nil :add-links nil
#+title: dotfiles
#+date: [2024-10-30 Wed 20:09]
#+filetags: :task:
#+identifier: 20241030T200935
Task 1
#+title: dotfiles
#+date: [2024-10-30 Wed 20:09]
#+filetags: :task:
#+identifier: 20241030T200941
Task 2
#+title: dotfiles
#+date: [2024-10-30 Wed 20:09]
#+filetags: :task:
#+identifier: 20241030T200946
Task 3
#+ENDWhoa, that was easy, but it's a little too verbose. Let's condense it a bit by removing the front matter by setting no-front-matter to t:
#+BEGIN: denote-files :regexp "--dotfiles.*__task" :excluded-dirs-regexp nil :sort-by-component identifier :reverse-sort nil :no-front-matter t
Task 1
Task 2
Task 3
#+ENDMuch better, but there's a problem. I can't edit the tasks here since the changes don't get replicated back to the original file. I can add links to the original file though by adding the add-links parameter to the dblock. I also set the file-separator here to be a newline character just to make it look a little nicer.
#+BEGIN: denote-files :regexp "--dotfiles.*__task" :excluded-dirs-regexp nil :sort-by-component identifier :reverse-sort nil :no-front-matter t :add-links t :file-separator "\n"
- [[denote:20241030T200935][dotfiles]]
Task 1
- [[denote:20241030T200941][dotfiles]]
Task 2
- [[denote:20241030T200946][dotfiles]]
Task 3
#+ENDWell, that was a fun experiment, but the title of the task file being the same as the workspace kinda… sucks. All the links look the same. I think I'll revisit the "sub-tag" idea so I can free up the title field for actual task content. Then I could open the linked file for more information on that task.
Sub-tagging
Well, after a little sleuthing, sub-tagging won't work. The (denote-sluggify-keyword) function will remove any good special character to use as a separator. But in hindsight, it isn't necessary. We can just use multiple tags. I've renamed my task files have a proper title and tagged the files with task and the name of the project, dotfiles, using the (denote-rename-file-keywords) command:
20241030T200925--dotfiles__project.org 20241030T200935--Task 1__dotfiles_task.org 20241030T200941--Task 2__dotfiles_task.org 20241030T200946--Task 3__dotfiles_task.org
Now let's look at the dblock again with an new regex _dotfiles.*_task:
#+BEGIN: denote-files :regexp "_dotfiles.*_task" :excluded-dirs-regexp nil :sort-by-component identifier :reverse-sort nil :no-front-matter t :add-links t :file-separator "\n"
- [[denote:20241030T200935][Task 1]]
The content of task 1!
- [[denote:20241030T200941][Task 2]]
The content of task 2!
- [[denote:20241030T200946][Task 3]]
The content of task 3!
#+ENDUhh… well that's cool but the problem with this approach is that the keywords are sorted alphabetically which would fail the regex is the project name started with a letter after t. I think we'll need some fancier regex to match in any order. Here we use a the same capture group twice so we can match it twice. This is better, but in theory it would still match something like _dotfiles_dotfiles or _task_task. That should be fine since Denote shouldn't let you duplicate keywords.
#+BEGIN: denote-files :regexp "\\(_dotfiles\\|_task\\).*\\(_dotfiles\\|_task\\)" :no-front-matter t :add-links t :file-separator "\n"
- [[denote:20241030T200935][Task 1]]
The content of task 1!
- [[denote:20241030T200941][Task 2]]
The content of task 2!
- [[denote:20241030T200946][Task 3]]
The content of task 3!
#+ENDAt this moment, I had a realization that I probably should be using backlinks, but that would require me to include a link to the main project file in my task. I think I'll shelve this for now because I'm kinda having fun with this idea.
There's also the option to use denote-links instead. I like this better.
#+BEGIN: denote-links :regexp "\\(_dotfiles\\|_task\\).*\\(_dotfiles\\|_task\\)"
- [[denote:20241030T200935][Task 1]]
- [[denote:20241030T200941][Task 2]]
- [[denote:20241030T200946][Task 3]]
#+END:On that note, maybe there's a better way
Now onto marking a task as complete. Adding a done tag seems silly at this point. Maybe, I could tweak denote-links with my own implementation to do what I want to do in Elisp instead of complicated regular expressions. It looks like these callbacks are defined with the org-dblock-write: prefix. Let's try to make our own to say "hello".
(defun org-dblock-write:pg-hello (params)
"hello")#+BEGIN: pg-hello
hello
#+END:Pretty simple, but really, I want the denote-links functionality, but with an extra parameter to match the files I want. Let's wrap denote-links and inject our own regexp dynamically
(defun org-dblock-write:pg-tasks (params)
(let ((project-name (plist-get params :project-name)))
(plist-put params :regexp (format "\\(_%1$s\\|_task\\).*\\(_%1$s\\|_task\\)" project-name)))
(org-dblock-write:denote-links params))#+BEGIN: pg-tasks :project-name dotfiles - [[denote:20241030T200935][Task 1]] - [[denote:20241030T200941][Task 2]] - [[denote:20241030T200946][Task 3]] #+END:
Perfect! That was easy. I love Elisp. Now that we have a single place to edit our pattern matching and the power of Elisp, let's make it hide tasks marked as done by simply handling the done tag. I've added the done tag to Task 3.
20241030T200925--dotfiles__project.org 20241030T200935--task-1__dotfiles_task.org 20241030T200941--task-2__dotfiles_task.org 20241030T200946--task-3__done_dotfiles_task.org
But as I begin writing the regex, I realize I need to negatively match it and ensure the tag doesn't exist. I don't think the regex is robust enough yet. It seems like the problem I mentioned about Denote not allowing duplicate keywords might actually pose a problem. I should try to exhaustively match on the tags I want. But after trying to write a regular expression to exclude done but also include dotfiles and task, it doesn't seem reasonably doable. So instead I think I'll exclude the regex and try to provide a list of files to a lower-level function, (denote-link--insert-links), myself. Unfortunately, this is marked as "private" but it will have to do.
I decided to separate the regex match into 3 separate matches since it makes it significantly simpler. And since we're using less regex, I decided to tag all the project files with project. Here are the rules I setup for myself:
- All project-related notes have the
projecttag. - All project-related notes also have that sluggified project name as a tag.
- The main project file has its title as the project name.
- Tasks have have the
tasktag.
Here are the notes for my "dotfiles" repo.
20241030T200925--dotfiles__dotfiles_project.org 20241030T200935--task-1__dotfiles_project_task.org 20241030T200941--task-2__dotfiles_project_task.org 20241030T200946--task-3__done_dotfiles_project_task.org
And some Elisp to handle all of this.
(defun pg/denote-find-project-files (project-name)
"Find all notes related to PROJECT-NAME."
(seq-filter (lambda (file)
(and
;; Find only project files
(string-match "_project" file)
;; And everything tagged with project name
(string-match (concat "_" (denote-sluggify-keyword project-name)) file)))
(denote-directory-files)))
(defun org-dblock-write:pg-tasks (params)
(let* ((project-name (symbol-name (plist-get params :project-name)))
;; First, find ALL notes related to this project.
(files (pg/denote-find-project-files project-name))
(files (seq-filter (lambda (file)
(and
;; Then filter for tasks.
(string-match "_task" file)
;; But are not done
(not (string-match "_done" file))
;; or cancelled
(not (string-match "_cancelled" file))))
files)))
(denote-link--insert-links files 'org nil t)))#+BEGIN: pg-tasks :project-name dotfiles - [[denote:20241030T200935][Task 1]] - [[denote:20241030T200941][Task 2]] #+END:
It works!
Automation!
I think the basic framework is there, but setting this up is kinda cumbersome to use. What I need now is a way to automatically generate the main project file and be able to create tasks. I briefly experiment with org-capture templates since Denote supports them, but it seemed a little redundant for this use-case. Instead, I'll just use the (denote) function directly.
(defun pg/denote-task-capture ()
"Create a task for a treebundel workspace."
(interactive)
(let ((workspace (or (when current-prefix-arg (treebundel-read-workspace))
(treebundel-current-workspace)
(treebundel-read-workspace))))
(denote (denote-title-prompt nil "New task")
`(,workspace "task" "project"))))Then I Created a new task called "Captured task!" #+BEGIN: pg-tasks :project-name dotfiles - [[denote:20241030T200935][Task 1]] - [[denote:20241030T200941][Task 2]] - [[denote:20241102T113732][Captured task!]] #+END:
The main project file
Now to generate this project file automatically. This should be easy enough with a Denote template. It looks like the VALUE in the denote-templates variable accepts a function. I'll need this to detect the appropriate workspace. Normally, a template should return a string to be inserted into the document. I don't do that here and instead (insert …) the content manually, because I need to update the dblock after inserting so it's prefilled when I open the buffer. This has a potential break in the future so I'll probably revisit this
(defun pg/denote-project-template ()
(insert "#+BEGIN: pg-tasks :project-name "
(denote-sluggify-keyword (denote-retrieve-filename-title buffer-file-name))
"\n"
"#+END:")
(org-update-all-dblocks)
"")
(add-to-list 'denote-templates
'(pg-project . pg/denote-project-template))Now I need to update the original (pg/open-project-notes) command to use this new template.
(defun pg/open-project-notes ()
"Open the main notes file for the current or specified project."
(interactive)
(when-let* ((workspace-name (or (when current-prefix-arg
(treebundel-read-workspace))
(treebundel-current-workspace)
(treebundel-read-workspace))))
;; Find the workspace-name as a tag instead
(if-let ((project-notes (seq-find (lambda (file)
(string-match (concat "--" workspace-name) file))
(pg/denote-find-project-files workspace-name))))
;; Open it if it exists
(find-file-other-window project-notes)
;; Otherwise, create a new one with the new template
(denote workspace-name `(,workspace-name "project") nil nil nil 'pg-project))))Changing the state of a task
To change the state of a task it should be as simple as adding the done or cancelled tag. I'd like to just use (denote-rename-file-keywords), but that has too many prompts. I wrote a couple functions for toggling whether a task is done or cancelled. These functions are basically the same, but when they toggle the respective state on, then it removes the conflicting state first. If I had more than 2 state tags (which I might one day), I'll create a more generic function to handle all of this, but for right now, this is good enough.
(defun pg/denote-task-toggle-done ()
(interactive)
(when (denote-file-is-note-p buffer-file-name)
(let ((denote-rename-confirmations nil)
(keywords (denote-extract-keywords-from-path buffer-file-name)))
(setq keywords (remove "cancelled" keywords))
(if (member "done" keywords)
(setq keywords (remove "done" keywords))
(setq keywords (append keywords '("done"))))
(denote-rename-file buffer-file-name
'keep-current
keywords
'keep-current
'keep-current))))(defun pg/denote-task-toggle-cancelled ()
(interactive)
(when (denote-file-is-note-p buffer-file-name)
(let ((denote-rename-confirmations nil)
(keywords (denote-extract-keywords-from-path buffer-file-name)))
(setq keywords (remove "done" keywords))
(if (member "cancelled" keywords)
(setq keywords (remove "cancelled" keywords))
(setq keywords (append keywords '("cancelled"))))
(denote-rename-file buffer-file-name
'keep-current
keywords
'keep-current
'keep-current))))Altogether
With all of this together, I'm able to create or open a project file with (pg/open-project-notes), see the lists of pending tasks, open any of those tasks, and change their state to done or cancelled to remove them from the main project file. Here's all the code in a single snippet.
(defun pg/denote-find-project-files (project-name)
"Find all notes related to PROJECT-NAME."
(seq-filter (lambda (file)
(and
;; Find only project files
(string-match "_project" file)
;; And everything tagged with project name
(string-match (concat "_" (denote-sluggify-keyword project-name)) file)))
(denote-directory-files)))
(defun org-dblock-write:pg-tasks (params)
(let* ((project-name (symbol-name (plist-get params :project-name)))
;; First, find ALL notes related to this project.
(files (pg/denote-find-project-files project-name))
(files (seq-filter (lambda (file)
(and
;; Then filter for tasks.
(string-match "_task" file)
;; But are not done
(not (string-match "_done" file))
;; or cancelled
(not (string-match "_cancelled" file))))
files)))
(denote-link--insert-links files 'org nil t)))
(defun pg/denote-task-capture ()
"Create a new task for the current or specified treebundel
workspace."
(interactive)
(let ((workspace (or (when current-prefix-arg (treebundel-read-workspace))
(treebundel-current-workspace)
(treebundel-read-workspace))))
(denote (denote-title-prompt nil "New task")
`(,workspace "task" "project"))))
(defun pg/denote-project-template ()
(insert "#+BEGIN: pg-tasks :project-name "
(denote-sluggify-keyword (denote-retrieve-filename-title buffer-file-name))
"\n"
"#+END:")
(org-update-all-dblocks)
"")
(add-to-list 'denote-templates
'(pg-project . pg/denote-project-template))
(defun pg/open-project-notes ()
"Open the main notes file for the current or specified project."
(interactive)
(when-let* ((workspace-name (or (when current-prefix-arg
(treebundel-read-workspace))
(treebundel-current-workspace)
(treebundel-read-workspace))))
;; Find the workspace-name as a tag instead
(if-let ((project-notes (seq-find (lambda (file)
(string-match (concat "--" workspace-name) file))
(pg/denote-find-project-files workspace-name))))
;; Open it if it exists
(find-file-other-window project-notes)
;; Otherwise, create a new one with the new template
(denote workspace-name `(,workspace-name "project") nil nil nil 'pg-project))))
(defun pg/denote-task-toggle-done ()
(interactive)
(when (denote-file-is-note-p buffer-file-name)
(let ((denote-rename-confirmations nil)
(keywords (denote-extract-keywords-from-path buffer-file-name)))
(setq keywords (remove "cancelled" keywords))
(if (member "done" keywords)
(setq keywords (remove "done" keywords))
(setq keywords (append keywords '("done"))))
(denote-rename-file buffer-file-name
'keep-current
keywords
'keep-current
'keep-current))))
(defun pg/denote-task-toggle-cancelled ()
(interactive)
(when (denote-file-is-note-p buffer-file-name)
(let ((denote-rename-confirmations nil)
(keywords (denote-extract-keywords-from-path buffer-file-name)))
(setq keywords (remove "done" keywords))
(if (member "cancelled" keywords)
(setq keywords (remove "cancelled" keywords))
(setq keywords (append keywords '("cancelled"))))
(denote-rename-file buffer-file-name
'keep-current
keywords
'keep-current
'keep-current))))Final thoughts
This was a fun experiment. I'm going to be trying this out for a while but with only little experimental use, I'm not sure if it's robust enough for my needs (see next section). I still can't help but get the feeling that I might as well be using backlinks, but trying something new is always worth the time. Maybe as I try to use this I'll improve it and eventually make a part 2 to make it even better.
Future improvements
- More states
- It would be nice in the future to include more states, like whether a task has been delegate to someone else or if I'm actively working on it so I can quickly remind myself of what my current goal is.
- Related tasks
- Often, larger tasks need to be broken down into smaller tasks. Not sure how this could be done in this system (maybe backlinks?), but it would nice to include related tasks somehow. Maybe this could be done by adding custom front-matter to the note.
- Age sorting
- I mentioned earlier that I loved Taskwarrior. The thing I loved about it was that is will slowly increase the priority of your tasks depending on their age. This was great to encourage me to work on other tasks I've been ignoring. With this system, this seems possible to do.
- Project note detection
- In the
(pg/denote-task-toggle…)functions, I only check if the file is a Denote note, but not if it is also a project note. I should add my own predicate that checks if it also contains theprojecttag.
- Consistent naming scheme
- Since this was just an adventure to hack some code together, I wasn't very consistent with the names of the functions (or writing docstrings). A little polish there could help the readability a bit. Maybe I could even write a small package for it.
- Auto-update buffer
- When opening a project buffer, you have to manually update the dblock (
C-c C-x C-uby default). Should be easy enough to add a hook for this to automatically update all dblocks in project notes.
- Task title completion
- When creating a new task, Denote will try auto-complete the title for you. Handy, but annoying in this case. Title completion should be turned off
Meta
Writing this post at the same time as actually developing the subject was a neat experience. It helped me think more clearly about my direction and forced myself to do a little more research and dive in deeper before writing something down. I highly recommend it to anyone to give it a shot and I might do it more in the future.
Throughout this post I use the terms "workspace" and "project" a lot. While they technically have a distinct meaning in my workflow, as I briefly describe below, a "workspace" could be any project or repository on your system, as long they are uniquely named on your system (e.g. in the same directory).