Posts tagged "org-mode":

Followup on secrets in my work notes

I got the following question on my post on how I handle secrets in my work notes:

Sounds like a nice approach for other secrets but how about :dbconnection for Orgmode and sql-connection-alist?

I have to admit I'd never come across the variable sql-connection-alist before. I've never really used sql-mode for more than editing SQL queries and setting up code blocks for running them was one of the first things I used yasnippet for.

I did a little reading and unfortunately it looks like sql-connection-alist can only handle string values. However, there is a variable sql-password-search-wallet-function, with the default value of sql-auth-source-search-wallet, so using auth-source is already supported for the password itself.

There seems to be a lack of good tutorials for setting up sql-mode in a secure way – all articles I found place the password in clear-text in the config – filling that gap would be a nice way to contribute to the Emacs community. I'm sure it'd prompt me to re-evaluate incorporating sql-mode in my workflow.

Improving how I handle secrets in my work notes

At work I use org-mode to keep notes about useful ways to query our systems, mostly that involves using the built-in SQL support to access DBs and ob-http to send HTTP requests. In both cases I often need to provide credentials for the systems. I'm embarrassed to admit it, but for a long time I've taken the easy path and kept all credentials in clear text. Every time I've used one of those code blocks I've thought I really ought to find a better way of handling these secrets one of these days. Yesterday was that day.

I ended up with two functions that uses auth-source and its ~/.authinfo.gpg file.

(defun mes/auth-get-pwd (host)
  "Get the password for a host (authinfo.gpg)"
  (-> (auth-source-search :host host)
      car
      (plist-get :secret)
      funcall))

(defun mes/auth-get-key (host key)
  "Get a key's value for a host (authinfo.gpg)

Not usable for getting the password (:secret), use 'mes/auth-get-pwd'
for that."
  (-> (auth-source-search :host host)
      car
      (plist-get key)))

It turns out that the library can handle more keys than the documentation suggests so for DB entries I'm using a machine (:host) that's a bit shorter and easier to remember than the full AWS hostname. Then I keep the DB host and name in dbhost (:dbhost) and dbname (:dbname) respectively. That makes an entry look like this:

machine db.svc login user port port password pwd dbname dbname dbhost dbhost

If I use it in a property drawer it looks like this

:PROPERTIES:
:header-args:sql: :engine postgresql
:header-args:sql+: :dbhost (mes/auth-get-key "db.svc" :dbhost)
:header-args:sql+: :dbport (string-to-number (mes/auth-get-key "db.svc" :port))
:header-args:sql+: :dbuser (mes/auth-get-key "db.svc" :user)
:header-args:sql+: :dbpassword (mes/auth-get-pwd "db.svc")
:header-args:sql+: :database (mes/auth-get-key "db.svc" :dbname)
:END:

A function for jumping to a project TODO file

I've had org-projectile in my config since the beginning, and while it's worked nicely for me in my main config it gave me some grief when I played around with elpaca the other week.¹

I tried to get the install instructions to work, but kept on getting errors when loading my config. Given that I only use it for one thing, to open the file TODO.org in the current project's root, I decided to just write a function for doing that instead.

(defun mep-projectile-open-todo ()
  "Open the project's todo file."
  (interactive)
  (if-let* ((proj-dir (projectile-project-root))
            (proj-todo-file (f-join proj-dir "TODO.org")))
      (org-open-file proj-todo-file)
    (message "Not in a project")))

Annotate projects in Emacs

Every now and then I've wished to write comments on files in a project, but I've never found a good way to do that. annotate.el and org-annotate-file both collect annotations in a central place (in my $HOME), while marginalia puts annotations in files next to the source files but in a format that's rather cryptic and tends to be messed up when attached to multiple lines. None of them is ideal, I'd like the format to be org-mode, but not in a central file. At the same time having one annotation file per source file is simply too much.

I tried wrapping org-annotate-file, setting org-annotate-file-storage-file and taking advantage of elisp's dynamic binding. However, it opens the annotation file in the current window, and I'd really like to split the window and open the annotations the right. Rather than trying to sort of "work it out backwards" I decided to write a small package and use as much of the functionality in org-annotate-file.el as possible.

First off I decided that I want the annotation file to be called projectile-annotations.org.

(defvar org-projectile-annotate-file-name "projectile-annotations.org"
  "The name of the file to store project annotations.")

Then I wanted a slightly modified version of org-annotate-file-show-section, I wanted it to respect the root of the project.

(defun org-projectile-annotate--file-show-section (storage-file)
  "Add or show annotation entry in STORAGE-FILE and return the buffer."
  ;; modified version of org-annotate-file-show-section
  (let* ((proj-root (projectile-project-root))
         (filename (file-relative-name buffer-file-name proj-root))
         (line (buffer-substring-no-properties (point-at-bol) (point-at-eol)))
         (annotation-buffer (find-file-noselect storage-file)))
    (with-current-buffer annotation-buffer
      (org-annotate-file-annotate filename line))
    annotation-buffer))

The main function can then simply work out where the file with annotations should be located and call org-projectile-annotate--file-show-section.

(defun org-projectile-annotate ()
  (interactive)
  (let ((annot-fn (file-name-concat (projectile-project-root)
                                    org-projectile-annotate-file-name)))
    (set-window-buffer (split-window-right)
                       (org-projectile-annotate--file-show-section annot-fn))))

When testing it all out I noticed that org-store-link makes a link with a search text. In my case it would be much better to have links with line numbers. I found there's a hook to modify the behaviour of org-store-link, org-create-file-search-functions. So I wrote a function to get the kind of links I want, but only when the project annotation file is open in a buffer.

(defun org-projectile-annotate-file-search-func ()
  "A function returning the current line number when called in a
project while the project annotation file is open.

This function is designed for use in the hook
'org-create-file-search-functions'. It changes the behaviour of
'org-store-link' so it constructs a link with a line number
instead of a search string."
  ;; TODO: find a way to make the link description nicer
  (when (and (projectile-project-p)
             (get-buffer-window org-projectile-annotate-file-name))
    (number-to-string (line-number-at-pos))))

That's it, now I only have to wait until the next time I want to comment on a project to see if it improves my way of working.

Comments and org-static-blog

I'm using org-static-blog to generate the contents of this site. So far I'm very happy with it, but I've gotten a few emails from readers who've wanted to comment on something I've written and they always point out that it's not easy to do. It's actually not a coincidence that it's a bit difficult!

Yesterday I came up with a way that might make is slightly easier without involving JavaScript from a 3rd party. By making use of the built-in support for adding HTML code for comments. One slight limitation is that it's a single variable holding the code, and I'd really like to allow for both

• using a link to a discussion site, e.g. reddit, as well as
• my email address

As the comment support in org-static-blog comes in the form of a single variable this seems a bit difficult to accomplish. However, it isn't difficult at all to do in elisp due to the power of advice-add.

By using the following advice on org-static-blog-publish-file

(advice-add 'org-static-blog-publish-file :around
            (lambda (orig-fn filename &rest args)
              (let*  ((comments-url (with-temp-buffer
                                      (insert-file-contents filename)
                                      (or (cadar (org-collect-keywords '("commentsurl")))
                                          my-blog-default-comments-url)))
                      (org-static-blog-post-comments (concat "Comment <a href=" comments-url ">here</a>.")))
                (apply orig-fn filename args))))

and defining my-blog-default-comments-url to a mailto:... URL I get a link to use for commenting by either

1. set commentsurl to point to discussion about the post on reddit, or
2. not set commentsurl at all and get the mailto:... URL.

If you look at my previous post you see the result of the former, and if you look below you see the result of the latter.

Trimming newline on code block variable

Today I found ob-http and decided to try it out a little. I quickly ran into a problem of a trailing newline. Basically I tried to do something like this:

#+name: id
#+begin_src http :select .id :cache yes
POST /foo
Content-Type: application/json

{
  "foo": "toto",
  "bar": "tata"
}
#+end_src

#+RESULTS[c5fd99206822a2109d7ac1d140185e6ec3f4f1d9]: id
#+begin_example
48722051-f81b-433f-acb4-a65d961ec841
#+end_example

#+header: :var id=id
#+begin_src http
POST /foo/${id}/fix
#+end_src

The trailing newline messes up the URL though, and the second code block fails.

I found two ways to deal with it, using a table and using org-sbe

#+name: id
#+begin_src http :select .id :cache yes :results table
POST /foo
Content-Type: application/json

{
  "foo": "toto",
  "bar": "tata"
}
#+end_src

#+RESULTS[c5fd99206822a2109d7ac1d140185e6ec3f4f1d9]: id
#+begin_example
| 48722051-f81b-433f-acb4-a65d961ec841 |
#+end_example

#+header: :var id=id[0,0]
#+begin_src http
POST /foo/${id}/fix
#+end_src

#+name: id
#+begin_src http :select .id :cache yes
POST /foo
Content-Type: application/json

{
  "foo": "toto",
  "bar": "tata"
}
#+end_src

#+RESULTS[c5fd99206822a2109d7ac1d140185e6ec3f4f1d9]: id
#+begin_example
48722051-f81b-433f-acb4-a65d961ec841
#+end_example

#+header: :var id=(org-sbe id)
#+begin_src http
POST /foo/${id}/fix
#+end_src

Keeping todo items in org-roam v2

Org-roam v2 has been released and yes, it broke my config a bit. Unfortunately the v1-to-v2 upgrade wizard didn't work for me. I realized later that it might have been due to the roam-related functions I'd hooked into `'before-save-hook`. I didn't think about it until I'd already manually touched up almost all my files (there aren't that many) so I can't say anything for sure. However, I think it might be a good idea to keep hooks in mind if one runs into issues with upgrading.

The majority of the time I didn't spend on my notes though, but on the setup I've written about in an earlier post, Keeping todo items in org-roam. Due to some of the changes in v2, changes that I think make org-roam slightly more "org-y", that setup needed a bit of love.

The basis is still the same 4 functions I described in that post, only the details had to be changed.

I hope the following is useful, and as always I'm always happy to receive commends and suggestions for improvements.

(defun roam-extra:get-filetags ()
  (split-string (or (org-roam-get-keyword "filetags") "")))

(defun roam-extra:add-filetag (tag)
  (let* ((new-tags (cons tag (roam-extra:get-filetags)))
         (new-tags-str (combine-and-quote-strings new-tags)))
    (org-roam-set-keyword "filetags" new-tags-str)))

(defun roam-extra:del-filetag (tag)
  (let* ((new-tags (seq-difference (roam-extra:get-filetags) `(,tag)))
         (new-tags-str (combine-and-quote-strings new-tags)))
    (org-roam-set-keyword "filetags" new-tags-str)))

(defun roam-extra:todo-p ()
  "Return non-nil if current buffer has any TODO entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (org-element-map
      (org-element-parse-buffer 'headline)
      'headline
    (lambda (h)
      (eq (org-element-property :todo-type h)
          'todo))
    nil 'first-match))

(defun roam-extra:update-todo-tag ()
  "Update TODO tag in the current buffer."
  (when (and (not (active-minibuffer-window))
             (org-roam-file-p))
    (org-with-point-at 1
      (let* ((tags (roam-extra:get-filetags))
             (is-todo (roam-extra:todo-p)))
        (cond ((and is-todo (not (seq-contains-p tags "todo")))
               (roam-extra:add-filetag "todo"))
              ((and (not is-todo) (seq-contains-p tags "todo"))
               (roam-extra:del-filetag "todo")))))))

(defun roam-extra:todo-files ()
  "Return a list of roam files containing todo tag."
  (org-roam-db-sync)
  (let ((todo-nodes (seq-filter (lambda (n)
                                  (seq-contains-p (org-roam-node-tags n) "todo"))
                                 (org-roam-node-list))))
    (seq-uniq (seq-map #'org-roam-node-file todo-nodes))))

(defun roam-extra:update-todo-files (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (roam-extra:todo-files)))

(add-hook 'find-file-hook #'roam-extra:update-todo-tag)
(add-hook 'before-save-hook #'roam-extra:update-todo-tag)
(advice-add 'org-agenda :before #'roam-extra:update-todo-files)

Todo items in org-roam, an update

I got an email from Mr Z with a nice modification to the code in my post on keeping todo items in org-roam.

He already had a bunch of agenda files that he wanted to keep using (I had so few of them that I'd simply converted them to roam files). Here's the solution he shared with me:

(defvar roam-extra-original-org-agenda-files nil
  "Original value of  `org-agenda-files'.")

(defun roam-extra:update-todo-files (&rest _)
  "Update the value of `org-agenda-files'."
  (unless roam-extra-original-org-agenda-files
    (setq roam-extra-original-org-agenda-files org-agenda-files))
  (setq org-agenda-files
        (append roam-extra-original-org-agenda-files
                (roam-extra:todo-files))))

It's a rather nice modification I think. Thanks to Mr Z for agreeing to let me share it here.

Keeping todo items in org-roam

A while ago I made an attempt to improve my work habits by keeping a document with TODO items. It lasted only for a while, and I've since had the intention to make another attempt. Since then I've started using org-roam and I've managed to create a habit of writing daily journal notes using org-roam's daily-notes. A few times I've thought that it might fit me well to put TODO items in the notes, but that would mean that I'd have to somehow keep track of them. At first I manually added a tag to each journal fily containing a TODO item. That didn't work very well at all, which should have been obvious up front. Then I added the folders where I keep roam files and journals to org-agenda-files, that worked a lot better. I'd still be using that, even if I expected it to slow down considerably as the number of files grow, but then I found a post on dynamic and fast agenda with org-roam.

I adjusted it slightly to fit my own setup a bit better, i.e. I made a Spacemacs layer, roam-extra, I use the tag todo, and I use a different hook to get the tag added on opening an org-roam file.

The layer consists of a single file, layers/roam-extra/funcs.el. In it I define 4 functions (they are pretty much copies of the functions in the post linked above):

1. roam-extra:todo-p - returns non-nil if the current current buffer contains a TODO item.
2. roam-extra:update-todo-tag - updates the tags of the current buffer to reflect the presence of TODO items, i.e. ensure the the tag todo is present iff there's a TODO item.
3. roam-extra:todo-files - uses the org-roam DB to return a list of all files containing the tag todo.
4. roam-extra:update-todo-files - adjusts 'org-agenda-files to contain only the files with TODO items.

I've put the full contents of the file at the end of the post.

To ensure that the todo tag is correct in all org-mode files I've added roam-extra:update-todo-tag to hooks that are invoked on opening an org-ram file and when saving a file. (I would love to find a more specialise hook than before-save-hook, but it works for now.)

(add-hook 'org-roam-file-setup-hook #'roam-extra:update-todo-tag)
(add-hook 'before-save-hook #'roam-extra:update-todo-tag)

To ensure that the list of files with TODO items is kept up to date when I open I also wrap org-agenda in an advice so roam-extra:update-todo-files is called prior to the agenda being opened.

(advice-add 'org-agenda :before #'roam-extra:update-todo-files)

(defun roam-extra:todo-p ()
  "Return non-nil if current buffer has any TODO entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (org-element-map
      (org-element-parse-buffer 'headline)
      'headline
    (lambda (h)
      (eq (org-element-property :todo-type h)
          'todo))
    nil 'first-match))

(defun roam-extra:update-todo-tag ()
  "Update TODO tag in the current buffer."
  (when (and (not (active-minibuffer-window))
             (org-roam--org-file-p buffer-file-name))
    (let* ((file (buffer-file-name (buffer-base-buffer)))
           (all-tags (org-roam--extract-tags file))
           (prop-tags (org-roam--extract-tags-prop file))
           (tags prop-tags))
      (if (roam-extra:todo-p)
          (setq tags (seq-uniq (cons "todo" tags)))
        (setq tags (remove "todo" tags)))
      (unless (equal prop-tags tags)
        (org-roam--set-global-prop
         "roam_tags"
         (combine-and-quote-strings tags))))))

(defun roam-extra:todo-files ()
  "Return a list of note files containing todo tag."
  (seq-map
   #'car
   (org-roam-db-query
    [:select file
             :from tags
             :where (like tags (quote "%\"todo\"%"))])))

(defun roam-extra:update-todo-files (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (roam-extra:todo-files)))

Ditaa in Org mode

Just found out that Emacs ships with Babel support for ditaa (yes, I'm late to the party).

Sweet! That is yet another argument for converting all our README.md into README.org at work.

\

The changes I made to my Spacemacs config are

(defun dotspacemacs/user-config ()
  ...
  (with-eval-after-load 'org
    ...
    (add-to-list 'org-babel-load-languages '(ditaa . t))
    (setq org-ditaa-jar-path "/usr/share/java/ditaa/ditaa-0.11.jar"))
  ...)

Some OrgMode stuff

The last few days I've watched Rainer König's OrgMode videos. It's resulted in a few new settings that makes Org a little more useful.

TIL: prompt matters to org-mode

A workmate just embellished some shell code blocks I'd put in a shared org-mode file with :session s. When I tried to run the blocks with sessions my emacs just froze up though. I found a post on the emacs StackExchange that offered a possible cause for it: the prompt.

I'm using bash-it so my prompt is rather far from the default.

After inspecting the session buffer simply added the following to my ~/.bashrc

if [[ ${TERM} == "dumb" ]]; then
    export BASH_IT_THEME='standard'
else
    export BASH_IT_THEME='simple'
fi

and now I can finally run shell code blocks in sessions.