Posts tagged "emacs":

30 Mar 2023

More on tree-sitter and consult

Here's a few things that I've gotten help with figuring out during the last few days. Both things are related to my playing with tree-sitter that I've written about earlier, here and here.

You might also be interested in the two repositories where the full code is. (I've linked to the specific commits as of this writing.)

Anonymous nodes and matching in tree-sitter

In the grammar for Cabal I have a rule for sections that like this

sections: $ => repeat1(choice(
    $.benchmark,
    $.common,
    $.executable,
    $.flag,
    $.library,
    $.source_repository,
    $.test_suite,
)),

where each section followed this pattern

benchmark: $ => seq(
    repeat($.comment),
    'benchmark',
    field('name', $.section_name),
    field('properties', $.property_block),
),

This made it a little bit difficult to capture the relevant parts of each section to implement consult-cabal. I thought a pattern like this ought to work

(cabal
 (sections
  (_ _ @type
     name: (section_name)? @name)))

but it didn't; I got way too many things captured in type. Clearly I had misunderstood something about the wildcards, or the query syntax. I attempted to add a field name to the anonymous node, i.e. change the sections rules like this

benchmark: $ => seq(
    repeat($.comment),
    field('type', 'benchmark'),
    field('name', $.section_name),
    field('properties', $.property_block),
),

It was accepted by tree-sitter generate, but the field type was nowhere to be found in the parse tree.

Then I changed the query to list the anonymous nodes explicitly, like this

(cabal
 (sections
  (_ ["benchmark" "common" "executable" ...] @type
     name: (section_name)? @name)))

That worked, but listing all the sections like that in the query didn't sit right with me.

Luckily there's a discussions area in tree-sitters GitHub so a fairly short discussion later I had answers to why my query behaved like it did and a solution that would allow me to not list all the section types in the query. The trick is to wrap the string in a call to alias to make it a named node. After that it works to add a field name to it as well, of course. The section rules now look like this

benchmark: $ => seq(
    repeat($.comment),
    field('type', alias('benchmark', $.section_type)),
    field('name', $.section_name),
    field('properties', $.property_block),
),

and the final query looks like this

(cabal
 (sections
  (_
   type: (section_type) @type
   name: (section_name)? @name)))

With that in place I could improve on the function that collects all the items for consult-cabal so it now show the section's type and name instead of the string representation of the tree-sitter node.

State in a consult source for preview of lines in a buffer

I was struggling with figuring out how to make a good state function in order to preview the items in consult-cabal. The GitHub repo for consult doesn't have discussions enabled, but after a discussion in an issue I'd arrived at a state function that works very well.

The state function makes use of functions in consult and looks like this

(defun consult-cabal--state ()
  "Create a state function for previewing sections."
  (let ((state (consult--jump-state)))
    (lambda (action cand)
      (when cand
        (let ((pos (get-text-property 0 'section-pos cand)))
          (funcall state action pos))))))

The trick here was to figure out how the function returned by consult--jump-state actually works. On the surface it looks like it takes an action and a candidate, (lambda (action cand) ...). However, the argument cand shouldn't be the currently selected item, but rather a postion (ideally a marker), so I had to attach another text property on the items (section-pos, which is fetched in the inner lambda). This position is then what's passed to the function returned by consult--jump-state.

In hindsight it seems so easy, but I was struggling with this for an entire evening before finally asking the question the morning after.

Tags: consult emacs tree-sitter
27 Mar 2023

Cabal, tree-sitter, and consult

After my last post I thought I'd move on to implement the rest of the functions in haskell-mode's major mode for Cabal, functions like haskell-cabal-goto-library-section and haskell-cabal-goto-executable-section. Then I realised that what I really want is a way to quickly jump to any section, that is, I want consult-cabal!

What follows is very much a work-in-progress, but hopefully it'll show enough promise.

Listing the sections

As I have a tree-sitter parse tree to hand it is fairly easy to fetch all the nodes corresponding to sections. Since the last post I've made some improvements to the parser and now the parse tree looks like this (I can recommend the function treesit-explore-mode to expore the parse tree, I've found it invaluable ever since I realised it existed)

(cabal
 ...
 (properties ...)
 (sections
  (common common (section_name) ...)
  (library library ...)
  (executable executable (section_name) ...)
  ...))

That is, all the sections are children of the node called sections.

The function to use for fetching all the nodes is treesit-query-capture, it needs a node to start on, which this case should be the full parse tree, i.e. (treesit-buffer-root-node 'cabal) and a query string. Given the structure of the parse tree, and that I want to capture all children of sections, a query string like this one works

"(cabal (sections (_)* @section))"

Finally, by default treesit-query-capture returns a list of tuples of the form (<capture> . <node>), but in this case I only want the list of nodes, so the full call will look like this

(treesit-query-capture (treesit-buffer-root-node 'cabal)
                       "(cabal (sections (_)* @section))"
                       nil nil t)

Hooking it up to consult

As I envision adding more things to jump to in the future, I decided to make use of consult--multi. That in turn means I need to define a "source" for the sections. After a bit of digging and rummaging in the consult source I put together this

(defvar consult-cabal--source-section
  `(:name "Sections"
    :category location
    :action ,#'consult-cabal--section-action
    :items ,#'consult-cabal--section-items)
  "Definition of source for Cabal sections.")

which means I need two functions, consult-cabal--section-action and consult-cabal--section-items. I started with the latter.

Getting section nodes as items for consult

It took me a while to work understand how this would ever be able to work. The function that :items point to must return a list of strings, but how would I ever be able to use just a string to jump to the correct location?

The solution is in a comment in the documentation of consult--multi:

:items - List of strings to select from or function returning list of strings. Note that the strings can use text properties to carry metadata, which is then available to the :annotate, :action and :state functions.

I'd never come across text properties in Emacs before, so at first I completely missed those two words. Once I'd looked up the concept in the documentation everything fell into place. The function consult-cabal--section-items would simply attach the relevant node as a text property to the strings in the list.

My current version, obviously a work-in-progress, takes a list of nodes and turns them naïvely into a string and attaches the node. I split it into two functions, like this

(defun consult-cabal--section-to-string (section)
  "Convert a single SECTION node to a string."
  (propertize (format "%S" section)
              :treesit-node section))

(defun consult-cabal--section-items ()
  "Fetch all sections as a list of strings ."
  (let ((section-nodes (treesit-query-capture (treesit-buffer-root-node 'cabal)
                                              "(cabal (sections (_)* @section))"
                                              nil nil t)))
    (mapcar #'consult-cabal--section-to-string section-nodes)))

Implementing the action

The action function is called with the selected item, i.e. with the string and its properties. That means, to jump to the selected section the function needs to extract the node property, :treesit-node, and jump to the start of it. the function to use is get-text-property, and as all characters in the string will have to property I just picked the first one. The jumping itself I copied from the navigation functions I'd written before.

(defun consult-cabal--section-action (item)
  "Go to the section referenced by ITEM."
  (when-let* ((node (get-text-property 0 :treesit-node item))
              (new-pos (treesit-node-start node)))
    (goto-char new-pos)))

Tying it together with consult--multi

The final function, consult-cabal, looks like this

(defun consult-cabal ()
  "Choose a Cabal construct and jump to it."
  (interactive)
  (consult--multi '(consult-cabal--source-section)
                  :sort nil))

Conclusions and where to find the code

The end result works as intended, but it's very rough. I'll try to improve it a bit more. In particular I want

  1. better strings - (format "%S" node) is all right to start with, but in the long run I want strings that describe the sections, and
  2. preview as I navigate between items - AFAIU this is what the :state field is for, but I still haven't looked into how it works.

The source can be found here.

Tags: cabal consult emacs tree-sitter
22 Mar 2023

Making an Emacs major mode for Cabal using tree-sitter

A few days ago I posted on r/haskell that I'm attempting to put together a Cabal grammar for tree-sitter. Some things are still missing, but it covers enough to start doing what I initially intended: experiment with writing an alternative Emacs major mode for Cabal.

The documentation for the tree-sitter integration is very nice, and several of the major modes already have tree-sitter variants, called X-ts-mode where X is e.g. python, so putting together the beginning of a major mode wasn't too much work.

Configuring Emacs

First off I had to make sure the parser for Cabal was installed. The snippet for that looks like this1

(use-package treesit
  :straight nil
  :ensure nil
  :commands (treesit-install-language-grammar)
  :init
  (setq treesit-language-source-alist
        '((cabal . ("https://gitlab.com/magus/tree-sitter-cabal.git")))))

With that in place the parser is installed using M-x treesit-install-language-grammar and choosing cabal.

After that I removed my configuration for haskell-mode and added the following snippet to get my own major mode into my setup.

(use-package my-cabal-mode
  :straight (:type git
             :repo "git@gitlab.com:magus/my-emacs-pkgs.git"
             :branch "main"
             :files (:defaults "my-cabal-mode/*el")))

The major mode and font-locking

The built-in elisp documentation actually has a section on writing a major mode with tree-sitter, so it was easy to get started. Setting up the font-locking took a bit of trial-and-error, but once I had comments looking the way I wanted it was easy to add to the setup. Oh, and yes, there's a section on font-locking with tree-sitter in the documentation too. At the moment it looks like this

(defvar cabal--treesit-font-lock-setting
  (treesit-font-lock-rules
   :feature 'comment
   :language 'cabal
   '((comment) @font-lock-comment-face)

   :feature 'cabal-version
   :language 'cabal
   '((cabal_version _) @font-lock-constant-face)

   :feature 'field-name
   :language 'cabal
   '((field_name) @font-lock-keyword-face)

   :feature 'section-name
   :language 'cabal
   '((section_name) @font-lock-variable-name-face))
  "Tree-sitter font-lock settings.")

;;;###autoload
(define-derived-mode my-cabal-mode fundamental-mode "My Cabal"
  "My mode for Cabal files"

  (when (treesit-ready-p 'cabal)
    (treesit-parser-create 'cabal)
    ;; set up treesit
    (setq-local treesit-font-lock-feature-list
                '((comment field-name section-name)
                  (cabal-version)
                  () ()))
    (setq-local treesit-font-lock-settings cabal--treesit-font-lock-setting)
    (treesit-major-mode-setup)))

;;;###autoload
(add-to-list 'auto-mode-alist '("\\.cabal\\'" . my-cabal-mode))

Navigation

One of the reasons I want to experiment with tree-sitter is to use it for code navigation. My first attempt is to translate haskell-cabal-section-beginning (in haskell-mode, the source) to using tree-sitter. First a convenience function to recognise if a node is a section or not

(defun cabal--node-is-section-p (n)
  "Predicate to check if treesit node N is a Cabal section."
  (member (treesit-node-type n)
          '("benchmark" "common" "executable" "flag" "library" "test_suite")))

That makes it possible to use treesit-parent-until to traverse the nodes until hitting a section node

(defun cabal-goto-beginning-of-section ()
  "Go to the beginning of the current section."
  (interactive)
  (when-let* ((node-at-point (treesit-node-at (point)))
              (section-node (treesit-parent-until node-at-point #'cabal--node-is-section-p))
              (start-pos (treesit-node-start section-node)))
    (goto-char start-pos)))

And the companion function, to go to the end of a section is very similar

(defun cabal-goto-end-of-section ()
  "Go to the end of the current section."
  (interactive)
  (when-let* ((node-at-point (treesit-node-at (point)))
              (section-node (treesit-parent-until node-at-point #'cabal--node-is-section-p))
              (end-pos (treesit-node-end section-node)))
    (goto-char end-pos)))

Footnotes:

1

I'm using straight.el and use-package in my setup, but hopefully the snippets can easily be converted to other ways of configuring Emacs.

Tags: cabal emacs haskell tree-sitter
03 Mar 2023

Per-project xref history in Emacs

When I write code I jump around in the code quite a bit, as I'm sure many other developers do. The ability to jump to the definition of a function, or a type, is invaluable when trying to understand code. In Emacs the built-in xref package provides the basic functionality for this, with many other packages providing their custom functions for looking up identifiers. This works beautifully except for one thing, there's only one global stack for keeping track of how you've jumped around.

Well, that used to be the case.

As I tend to have multiple projects open at a time I used to find it very confusing when I pop positions off the xref stack and all of a sudden find myself in another project. It would be so much nicer to have a per-project stack.

I've only known of one solution for this, the perspective package, but as I've been building my own Emacs config I wanted to see if there were other options. It turns out there is one (almost) built into Emacs 29.

In Emacs 29 there's built-in support for having per-window xref stacks, and the way that's done allows one to extend it further. There's now a variable, xref-history-storage, that controls access to the xref stack. The default is still a global stack (when the variable is set to #'xref-global-history), but to get per-window stacks one sets it to #'xref-window-local-history.

After finding this out I thought I'd try to write my own, implementing per-project xref stacks (for projectile).

The function should take one optional argument, new-value, if it's provided the stack should be updated and if not, it should be returned. That is, something like this

(defun projectile-param-xref-history (&optional new-value)
  "Return project-local xref history for the current projectile.

Override existing value with NEW-VALUE if it's set."
  (if new-value
      (projectile-param-set-parameter 'xref--history new-value)
    (or (projectile-param-get-parameter 'xref--history)
        (projectile-param-set-parameter 'xref--history (xref--make-xref-history)))))

Now I only had to write the two functions projectile-param-get-parameter and projectile-param-set-parameter. I thought a rather straight forward option would be to use a hashtable and store values under a tuple comprising the project name and the parameter passed in.

(defvar projectile-params--store (make-hash-table :test 'equal)
  "The store of project parameters.")

(defun projectile-param-get-parameter (param)
  "Return project parameter PARAM, or nil if unset."
  (let ((key (cons (projectile-project-name) param)))
    (gethash key projectile-params--store nil)))

(defun projectile-param-set-parameter (param value)
  "Set the project parameter PARAM to VALUE."
  (let ((key (cons (projectile-project-name) param)))
    (puthash key value projectile-params--store))
  value)

Then I tried it out by setting xref-history-storage

(setq xref-history-storage #'projectile-param-xref-history)

and so far it's been working well.

The full code is here.

Tags: emacs xref projectile
24 Sep 2022

Annotate projects in Emacs

Every now and then I've wished to write comments on files in a project, but I've never found a good way to do that. annotate.el and org-annotate-file both collect annotations in a central place (in my $HOME), while marginalia puts annotations in files next to the source files but in a format that's rather cryptic and tends to be messed up when attached to multiple lines. None of them is ideal, I'd like the format to be org-mode, but not in a central file. At the same time having one annotation file per source file is simply too much.

I tried wrapping org-annotate-file, setting org-annotate-file-storage-file and taking advantage of elisp's dynamic binding. However, it opens the annotation file in the current window, and I'd really like to split the window and open the annotations the right. Rather than trying to sort of "work it out backwards" I decided to write a small package and use as much of the functionality in org-annotate-file.el as possible.

First off I decided that I want the annotation file to be called projectile-annotations.org.

(defvar org-projectile-annotate-file-name "projectile-annotations.org"
  "The name of the file to store project annotations.")

Then I wanted a slightly modified version of org-annotate-file-show-section, I wanted it to respect the root of the project.

(defun org-projectile-annotate--file-show-section (storage-file)
  "Add or show annotation entry in STORAGE-FILE and return the buffer."
  ;; modified version of org-annotate-file-show-section
  (let* ((proj-root (projectile-project-root))
         (filename (file-relative-name buffer-file-name proj-root))
         (line (buffer-substring-no-properties (point-at-bol) (point-at-eol)))
         (annotation-buffer (find-file-noselect storage-file)))
    (with-current-buffer annotation-buffer
      (org-annotate-file-annotate filename line))
    annotation-buffer))

The main function can then simply work out where the file with annotations should be located and call org-projectile-annotate--file-show-section.

(defun org-projectile-annotate ()
  (interactive)
  (let ((annot-fn (file-name-concat (projectile-project-root)
                                    org-projectile-annotate-file-name)))
    (set-window-buffer (split-window-right)
                       (org-projectile-annotate--file-show-section annot-fn))))

When testing it all out I noticed that org-store-link makes a link with a search text. In my case it would be much better to have links with line numbers. I found there's a hook to modify the behaviour of org-store-link, org-create-file-search-functions. So I wrote a function to get the kind of links I want, but only when the project annotation file is open in a buffer.

(defun org-projectile-annotate-file-search-func ()
  "A function returning the current line number when called in a
project while the project annotation file is open.

This function is designed for use in the hook
'org-create-file-search-functions'. It changes the behaviour of
'org-store-link' so it constructs a link with a line number
instead of a search string."
  ;; TODO: find a way to make the link description nicer
  (when (and (projectile-project-p)
             (get-buffer-window org-projectile-annotate-file-name))
    (number-to-string (line-number-at-pos))))

That's it, now I only have to wait until the next time I want to comment on a project to see if it improves my way of working.

Tags: emacs org-mode
09 Jul 2022

Playing with setting up Emacs

TL;DR: I've put together a small-ish Emacs configuration that I call MES. Hopefully it can be of use to someone out there.

My Emacs Setup - MES

The other day I started watching some videos in the Emacs From Scratch series from System Crafters. It looked like something that could be fun to play with so over the last couple of days I've been tinkering with putting together the beginnings of a configuration.

During the process I realised just how much work it'd be to put together something that comes close to the polish of Spacemacs, so I've currently no intention of actually using MES myself. It was fun though, and maybe it can serve as inspiration (or as a deterrent) for someone else.

The major parts are

Tags: emacs
15 Jun 2022

Power-mode in Spacemacs

I just found the Power Mode for Emacs. If you want to try it out in Spacemacs you can make sure that your ~/.spacemacs contains the following

dotspacemacs-additional-packages
'(
  ...
  (power-mode :location (recipe
                         :fetcher github
                         :repo "elizagamedev/power-mode.el"))
  )

After a restart Power Mode can be turned on using SPC SPC power-mode.

Unfortunately I found that it slows down rendering so badly that Emacs isn't keeping up with my typing. Even though I removed it right away again it was fun to try it out, and I did learn how to add package to Spacemacs that aren't on MELPA.

A useful resource is this reference on the recipe format.

Tags: emacs spacemacs
09 May 2022

Comments and org-static-blog

I'm using org-static-blog to generate the contents of this site. So far I'm very happy with it, but I've gotten a few emails from readers who've wanted to comment on something I've written and they always point out that it's not easy to do. It's actually not a coincidence that it's a bit difficult!

Yesterday I came up with a way that might make is slightly easier without involving JavaScript from a 3rd party. By making use of the built-in support for adding HTML code for comments. One slight limitation is that it's a single variable holding the code, and I'd really like to allow for both

As the comment support in org-static-blog comes in the form of a single variable this seems a bit difficult to accomplish. However, it isn't difficult at all to do in elisp due to the power of advice-add.

By using the following advice on org-static-blog-publish-file

(advice-add 'org-static-blog-publish-file :around
            (lambda (orig-fn filename &rest args)
              (let*  ((comments-url (with-temp-buffer
                                      (insert-file-contents filename)
                                      (or (cadar (org-collect-keywords '("commentsurl")))
                                          my-blog-default-comments-url)))
                      (org-static-blog-post-comments (concat "Comment <a href=" comments-url ">here</a>.")))
                (apply orig-fn filename args))))

and defining my-blog-default-comments-url to a mailto:... URL I get a link to use for commenting by either

  1. set commentsurl to point to discussion about the post on reddit, or
  2. not set commentsurl at all and get the mailto:... URL.

If you look at my previous post you see the result of the former, and if you look below you see the result of the latter.

Tags: emacs org-mode
30 Jan 2022

Keeping Projectile's cache tidy

A while back I added a function to find all projects recursively from a directory and add them to Projectile's cache (see the Populating Projectile's cache). Since then I've just made a tiny change to also include containing a .projectile file.

(defun projectile-extra-add-projects-in-subfolders (projects-root)
  (interactive (list (read-directory-name "Add to known projects: ")))
  (message "Searching for projects in %s..." projects-root)
  (let* ((proj-rx (rx (and line-start ?. (or "projectile" "git") line-end)))
         (dirs (seq-map
                'file-name-directory
                (directory-files-recursively projects-root
                                             proj-rx
                                             t))))
    (seq-do 'projectile-add-known-project dirs)
    (message "Added %d projects" (length dirs))))

Since then I've also found a need for tidying the cache, in my casse that means removing entries in the cache that no longer exist on disk. I didn't find a function for it, so I wrote one.

(defun projectile-extra-tidy-projects ()
  (interactive)
  (let ((missing-dirs (seq-remove 'file-directory-p projectile-known-projects)))
    (seq-do 'projectile-remove-known-project missing-dirs)
    (message "Tidied %d projects" (length missing-dirs))))
Tags: emacs elisp projectile
01 Jan 2022

Trimming newline on code block variable

Today I found ob-http and decided to try it out a little. I quickly ran into a problem of a trailing newline. Basically I tried to do something like this:

#+name: id
#+begin_src http :select .id :cache yes
POST /foo
Content-Type: application/json

{
  "foo": "toto",
  "bar": "tata"
}
#+end_src

#+RESULTS[c5fd99206822a2109d7ac1d140185e6ec3f4f1d9]: id
#+begin_example
48722051-f81b-433f-acb4-a65d961ec841
#+end_example

#+header: :var id=id
#+begin_src http
POST /foo/${id}/fix
#+end_src

The trailing newline messes up the URL though, and the second code block fails.

I found two ways to deal with it, using a table and using org-sbe

Using a table

#+name: id
#+begin_src http :select .id :cache yes :results table
POST /foo
Content-Type: application/json

{
  "foo": "toto",
  "bar": "tata"
}
#+end_src

#+RESULTS[c5fd99206822a2109d7ac1d140185e6ec3f4f1d9]: id
#+begin_example
| 48722051-f81b-433f-acb4-a65d961ec841 |
#+end_example

#+header: :var id=id[0,0]
#+begin_src http
POST /foo/${id}/fix
#+end_src

Using org-sbe

#+name: id
#+begin_src http :select .id :cache yes
POST /foo
Content-Type: application/json

{
  "foo": "toto",
  "bar": "tata"
}
#+end_src

#+RESULTS[c5fd99206822a2109d7ac1d140185e6ec3f4f1d9]: id
#+begin_example
48722051-f81b-433f-acb4-a65d961ec841
#+end_example

#+header: :var id=(org-sbe id)
#+begin_src http
POST /foo/${id}/fix
#+end_src
Tags: emacs org-mode
08 Dec 2021

Magit/forge and self-hosted GitLab

As I found the documentation for adding a self-hosted instance of GitLab to to magit/forge a bit difficult, I thought I'd write a note for my future self (and anyone else who might find it useful).

First put the following in `~/.gitconfig`

[gitlab "gitlab.private.com/api/v4"]
  user = my.username

Then create an access token on GitLab. I ticked api and write_repository, which seems to work fine so far. Put the token in ~/.authinfo.gpg

machine gitlab.private.com/api/v4 login my.user^forge password <token>

(Remember that a newline is needed at the end of the file.)

Finally, add the GitLab instance to 'forge-alist

(setq
 forge-alist
 '(("gitlab.private.com" "gitlab.private.com/api/v4" "gitlab.private.com" forge-gitlab-repository)
   ("github.com" "api.github.com" "github.com" forge-github-repository)
   ("gitlab.com" "gitlab.com/api/v4" "gitlab.com" forge-gitlab-repository))
 )

That's it!

Tags: emacs git magit
23 Jul 2021

Keeping todo items in org-roam v2

Org-roam v2 has been released and yes, it broke my config a bit. Unfortunately the v1-to-v2 upgrade wizard didn't work for me. I realized later that it might have been due to the roam-related functions I'd hooked into `'before-save-hook`. I didn't think about it until I'd already manually touched up almost all my files (there aren't that many) so I can't say anything for sure. However, I think it might be a good idea to keep hooks in mind if one runs into issues with upgrading.

The majority of the time I didn't spend on my notes though, but on the setup I've written about in an earlier post, Keeping todo items in org-roam. Due to some of the changes in v2, changes that I think make org-roam slightly more "org-y", that setup needed a bit of love.

The basis is still the same 4 functions I described in that post, only the details had to be changed.

I hope the following is useful, and as always I'm always happy to receive commends and suggestions for improvements.

Some tag helpers

The very handy functions for extracting tags as lists seem to be gone, in their place I found org-roam-{get,set}-keyword. Using these I wrote three wrappers that allow slightly nicer handling of tags.

(defun roam-extra:get-filetags ()
  (split-string (or (org-roam-get-keyword "filetags") "")))

(defun roam-extra:add-filetag (tag)
  (let* ((new-tags (cons tag (roam-extra:get-filetags)))
         (new-tags-str (combine-and-quote-strings new-tags)))
    (org-roam-set-keyword "filetags" new-tags-str)))

(defun roam-extra:del-filetag (tag)
  (let* ((new-tags (seq-difference (roam-extra:get-filetags) `(,tag)))
         (new-tags-str (combine-and-quote-strings new-tags)))
    (org-roam-set-keyword "filetags" new-tags-str)))

The layer

roam-extra:todo-p needed no changes at all. I'm including it here only for easy reference.

(defun roam-extra:todo-p ()
  "Return non-nil if current buffer has any TODO entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (org-element-map
      (org-element-parse-buffer 'headline)
      'headline
    (lambda (h)
      (eq (org-element-property :todo-type h)
          'todo))
    nil 'first-match))

As pretty much all functions I used in the old version of roam-extra:update-todo-tag are gone I took the opportunity to rework it completely. I think it ended up being slightly simpler. I suspect the the use of org-with-point-at 1 ... is unnecessary, but I haven't tested it yet so I'm leaving it in for now.

(defun roam-extra:update-todo-tag ()
  "Update TODO tag in the current buffer."
  (when (and (not (active-minibuffer-window))
             (org-roam-file-p))
    (org-with-point-at 1
      (let* ((tags (roam-extra:get-filetags))
             (is-todo (roam-extra:todo-p)))
        (cond ((and is-todo (not (seq-contains-p tags "todo")))
               (roam-extra:add-filetag "todo"))
              ((and (not is-todo) (seq-contains-p tags "todo"))
               (roam-extra:del-filetag "todo")))))))

In the previous version roam-extra:todo-files was built using an SQL query. That felt a little brittle to me, so despite that my original inspiration contains an updated SQL query I decided to go the route of using the org-roam API instead. The function org-roam-node-list makes it easy to get all nodes and then finding the files is just a matter of using seq-filter and seq-map. Now that headings may be nodes, and that heading-based nodes seem to inherit the top-level tags, a file may appear more than once, hence the call to seq-unique at the end.

Based on what I've seen V2 appears less eager to sync the DB, so to make sure all nodes are up-to-date it's best to start off with forcing a sync.

(defun roam-extra:todo-files ()
  "Return a list of roam files containing todo tag."
  (org-roam-db-sync)
  (let ((todo-nodes (seq-filter (lambda (n)
                                  (seq-contains-p (org-roam-node-tags n) "todo"))
                                 (org-roam-node-list))))
    (seq-uniq (seq-map #'org-roam-node-file todo-nodes))))

With that in place it turns out that also roam-extra:update-todo-files worked without any changes. I'm including it here for easy reference as well.

(defun roam-extra:update-todo-files (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (roam-extra:todo-files)))

Hooking it up

The variable org-roam-file-setup-hook is gone, so the the more general find-file-hook will have to be used instead.

(add-hook 'find-file-hook #'roam-extra:update-todo-tag)
(add-hook 'before-save-hook #'roam-extra:update-todo-tag)
(advice-add 'org-agenda :before #'roam-extra:update-todo-files)
Tags: emacs org-mode org-roam spacemacs
21 Mar 2021

Todo items in org-roam, an update

I got an email from Mr Z with a nice modification to the code in my post on keeping todo items in org-roam.

He already had a bunch of agenda files that he wanted to keep using (I had so few of them that I'd simply converted them to roam files). Here's the solution he shared with me:

(defvar roam-extra-original-org-agenda-files nil
  "Original value of  `org-agenda-files'.")

(defun roam-extra:update-todo-files (&rest _)
  "Update the value of `org-agenda-files'."
  (unless roam-extra-original-org-agenda-files
    (setq roam-extra-original-org-agenda-files org-agenda-files))
  (setq org-agenda-files
        (append roam-extra-original-org-agenda-files
                (roam-extra:todo-files))))

It's a rather nice modification I think. Thanks to Mr Z for agreeing to let me share it here.

Tags: emacs org-mode org-roam
14 Mar 2021

Keeping todo items in org-roam

A while ago I made an attempt to improve my work habits by keeping a document with TODO items. It lasted only for a while, and I've since had the intention to make another attempt. Since then I've started using org-roam and I've managed to create a habit of writing daily journal notes using org-roam's daily-notes. A few times I've thought that it might fit me well to put TODO items in the notes, but that would mean that I'd have to somehow keep track of them. At first I manually added a tag to each journal fily containing a TODO item. That didn't work very well at all, which should have been obvious up front. Then I added the folders where I keep roam files and journals to org-agenda-files, that worked a lot better. I'd still be using that, even if I expected it to slow down considerably as the number of files grow, but then I found a post on dynamic and fast agenda with org-roam.

I adjusted it slightly to fit my own setup a bit better, i.e. I made a Spacemacs layer, roam-extra, I use the tag todo, and I use a different hook to get the tag added on opening an org-roam file.

The layer consists of a single file, layers/roam-extra/funcs.el. In it I define 4 functions (they are pretty much copies of the functions in the post linked above):

  1. roam-extra:todo-p - returns non-nil if the current current buffer contains a TODO item.
  2. roam-extra:update-todo-tag - updates the tags of the current buffer to reflect the presence of TODO items, i.e. ensure the the tag todo is present iff there's a TODO item.
  3. roam-extra:todo-files - uses the org-roam DB to return a list of all files containing the tag todo.
  4. roam-extra:update-todo-files - adjusts 'org-agenda-files to contain only the files with TODO items.

I've put the full contents of the file at the end of the post.

To ensure that the todo tag is correct in all org-mode files I've added roam-extra:update-todo-tag to hooks that are invoked on opening an org-ram file and when saving a file. (I would love to find a more specialise hook than before-save-hook, but it works for now.)

(add-hook 'org-roam-file-setup-hook #'roam-extra:update-todo-tag)
(add-hook 'before-save-hook #'roam-extra:update-todo-tag)

To ensure that the list of files with TODO items is kept up to date when I open I also wrap org-agenda in an advice so roam-extra:update-todo-files is called prior to the agenda being opened.

(advice-add 'org-agenda :before #'roam-extra:update-todo-files)

The layer, layers/roam-extra/funcs.el

(defun roam-extra:todo-p ()
  "Return non-nil if current buffer has any TODO entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (org-element-map
      (org-element-parse-buffer 'headline)
      'headline
    (lambda (h)
      (eq (org-element-property :todo-type h)
          'todo))
    nil 'first-match))

(defun roam-extra:update-todo-tag ()
  "Update TODO tag in the current buffer."
  (when (and (not (active-minibuffer-window))
             (org-roam--org-file-p buffer-file-name))
    (let* ((file (buffer-file-name (buffer-base-buffer)))
           (all-tags (org-roam--extract-tags file))
           (prop-tags (org-roam--extract-tags-prop file))
           (tags prop-tags))
      (if (roam-extra:todo-p)
          (setq tags (seq-uniq (cons "todo" tags)))
        (setq tags (remove "todo" tags)))
      (unless (equal prop-tags tags)
        (org-roam--set-global-prop
         "roam_tags"
         (combine-and-quote-strings tags))))))

(defun roam-extra:todo-files ()
  "Return a list of note files containing todo tag."
  (seq-map
   #'car
   (org-roam-db-query
    [:select file
             :from tags
             :where (like tags (quote "%\"todo\"%"))])))

(defun roam-extra:update-todo-files (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (roam-extra:todo-files)))
Tags: emacs org-mode org-roam spacemacs
05 Mar 2021

Flycheck and HLS

I've been using LSP for most programming languages for a while now. HLS is really very good now, but I've found that it doesn't warn on quite all things I'd like it to so I find myself having to swap between the 'lsp and 'haskell-ghc checkers. However, since flycheck supports chaining checkers I thought there must be a way to have both checkers active at the same time.

The naive approach didn't work due to load order of things in Spacemacs so I had to experiment a bit to find something that works.

The first issue was to make sure that HLS is available at all. I use shell.nix together with direnv extensively and I had noticed that lsp-mode tried to load HLS before direnv had put it in the $PATH. I think the 'lsp-beforeinitialize-hook is the hook to use for this:

(add-hook 'lsp-before-initialize-hook #'direnv-update-environment))

I made a several attempt to chain the checkers but kept on getting errors due to the 'lsp checker not being defined yet. Another problem I ran into was that the checkers were chained too late, resulting in having to manually run flycheck-buffer on the first file I opened. (Deferred loading is a brilliant thing, but make some things really difficult to debug.) After quite a bit of experimenting and reading the description of various hooks I did find something that works:

(with-eval-after-load 'lsp-mode
  (defun magthe:lsp-next-checker ()
    (flycheck-add-next-checker 'lsp '(warning . haskell-ghc)))
  (add-hook 'lsp-lsp-haskell-after-open-hook
            #'magthe:lsp-next-checker))

Of course I have no idea if this is the easiest or most elegant solution but it does work for my testcases:

  1. Open a file in a project, SPC p l - choose project - choose a Haskell file.
  2. Open a project, SPC p l followed by C-d, and then open a Haskell file.

Suggestions for improvements are more than welcome, of course.

Tags: emacs haskell flycheck
22 Jun 2020

Better Nix setup for Spacemacs

In an earlier post I documented my setup for getting Spacemacs/Emacs to work with Nix. I've since found a much more elegant solution based on

No more Emacs packages for Nix and no need to defining functions that wrap executables in an invocation of nix-shell.

There's a nice bonus too, with this setup I don't need to run nix-shell, which always drops me at a bash prompt, instead I get a working setup in my shell of choice.

Setting up direnv

The steps for setting up direnv depends a bit on your setup, but luckily I found the official instructions for installing direnv to be very clear and easy to follow. There's not much I can add to that.

Setting up Spacemacs

Since emacs-direnv isn't included by default in Spacemacs I needed to do a bit of setup. I opted to create a layer for it, rather than just drop it in the list dotspacemacs-additional-packages. Yes, a little more complicated, but not difficult and I nurture an intention of submitting the layer for inclusion in Spacemacs itself at some point. I'll see where that goes.

For now, I put the following in the file ~/.emacs.d/private/layers/direnv/packages.el:

(defconst direnv-packages
      '(direnv))

(defun direnv/init-direnv ()
  (use-package direnv
    :init
    (direnv-mode)))

Setting up the project folders

In each project folder I then add the file .envrc containing a single line:

use_nix

Then I either run direnv allow from the command line, or run the function direnv-allow after opening the folder in Emacs.

Using it

It's as simple as moving into the folder in a shell – all required envvars are set up on entry and unset on exit.

In Emacs it's just as simple, just open a file in a project and the envvars are set. When switching to a buffer outside the project the envvars are unset.

There is only one little caveat, nix-build doesn't work inside a Nix shell. I found out that running

IN_NIX_SHELL= nix-build

does work though.

Tags: emacs nix spacemacs
07 Dec 2019

Nix setup for Spacemacs

Edit 2020-06-22: I've since found a better setup for this.

When using ghcide and LSP, as I wrote about in my post on Haskell, ghcide, and Spacemacs, I found myself ending up recompiling a little too often. This pushed me to finally start looking at Nix. After a bit of a fight I managed to get ghcide from Nix, which brought me the issue of setting up Spacemacs. Inspired by a gist from Samuel Evans-Powell and a guide to setting up an environment for Reflex by Thales Macedo Garitezi I ended up with the following setup:

(defun dotspacemacs/layers ()
  (setq-default
   ...
   dotspacemacs-additional-packages
   '(
     nix-sandbox
     nix-haskell-mode
     ...
     )
   ...
   ))
(defun dotspacemacs/user-config ()
  ...
  (add-hook 'haskell-mode-hook #'lsp)
  (add-hook 'haskell-mode-hook 'nix-haskell-mode)
  (add-hook 'haskell-mode-hook
            (lambda ()
              (setq-local flycheck-executable-find
                          (lambda (cmd)
                            (nix-executable-find (nix-current-sandbox) cmd)))
              (setq-local flycheck-command-wrapper-function
                          (lambda (argv)
                            (apply 'nix-shell-command (nix-current-sandbox) argv)))
              (setq-local haskell-process-wrapper-function
                          (lambda (argv)
                            (apply 'nix-shell-command (nix-current-sandbox) argv)))
              (setq-local lsp-haskell-process-wrapper-function
                          (lambda (argv)
                            `("nix-shell" "-I" "." "--command" "ghcide --lsp" ,(nix-current-sandbox))))))
  (add-hook 'haskell-mode-hook
            (lambda ()
              (flycheck-add-next-checker 'lsp-ui '(warning . haskell-stack-ghc))))
  ...
  )

It seems to work, but please let me know if you have suggestions for improvements.

Tags: emacs nix spacemacs
05 Nov 2019

Populating Projectile's cache

As I track the develop branch of Spacemacs I occasionally clean out my cache of projects known to Projectile. Every time it takes a while before I'm back at a stage where I very rarely have to visit something that isn't already in the cache.

However, today I found the function projectile-add-known-project, which prompted me to write the following function that'll help me quickly re-building the cache the next time I need to reset Spacemacs to a known state again.

(defun projectile-extra-add-projects-in-subfolders (projects-root)
  (interactive (list (read-directory-name "Add to known projects: ")))
  (message "Searching for projects in %s..." projects-root)
  (let ((dirs (seq-map 'file-name-directory (directory-files-recursively projects-root "^.git$" t))))
    (seq-do 'projectile-add-known-project dirs)
    (message "Added %d projects" (length dirs))))
Tags: emacs elisp projectile
20 Oct 2019

Ditaa in Org mode

Just found out that Emacs ships with Babel support for ditaa (yes, I'm late to the party).

Sweet! That is yet another argument for converting all our README.md into README.org at work.

giphy.gif\

The changes I made to my Spacemacs config are

(defun dotspacemacs/user-config ()
  ...
  (with-eval-after-load 'org
    ...
    (add-to-list 'org-babel-load-languages '(ditaa . t))
    (setq org-ditaa-jar-path "/usr/share/java/ditaa/ditaa-0.11.jar"))
  ...)
Tags: emacs org-mode
19 Sep 2019

Haskell, ghcide, and Spacemacs

The other day I read Chris Penner's post on Haskell IDE Support and thought I'd make an attempt to use it with Spacemacs.

After running stack build hie-bios ghcide haskell-lsp --copy-compiler-tool I had a look at the instructions on using haskell-ide-engine with Spacemacs. After a bit of trial and error I came up with these changes to my ~/.spacemacs:

(defun dotspacemacs/layers ()
  (setq-default
   dotspacemacs-configuration-layers
   '(
    ...
    lsp
    (haskell :variables
             haskell-completion-backend 'lsp
             )
    ...)
  )
)
(defun dotspacemacs/user-config ()
  (setq lsp-haskell-process-args-hie '("exec" "ghcide" "--" "--lsp")
        lsp-haskell-process-path-hie "stack"
        lsp-haskell-process-wrapper-function (lambda (argv) (cons (car argv) (cddr argv)))
        )
  (add-hook 'haskell-mode-hook
            #'lsp))

The slightly weird looking lsp-haskell-process-wrapper-function is removing the pesky --lsp inserted by this line.

That seems to work. Though I have to say I'm not ready to switch from intero just yet. Two things in particular didn't work with =ghcide=/LSP:

  1. Switching from one the Main.hs in one executable to the Main.hs of another executable in the same project didn't work as expected – I had hints and types in the first, but nothing in the second.
  2. Jump to the definition of a function defined in the package didn't work – I'm not willing to use GNU GLOBAL or some other source tagging system.
Tags: emacs haskell lsp spacemacs
07 May 2019

Some OrgMode stuff

The last few days I've watched Rainer König's OrgMode videos. It's resulted in a few new settings that makes Org a little more useful.

Variable Value Description
calendar-week-start-day 1 Weeks start on Monday!
org-modules (list) org-habit Support for tracking habits
org-modules (list) org-id Improved support for ID property
org-agenda-start-on-weekday 1 Weeks start on Monday, again!
org-log-into-drawer t Put notes (logs) into a drawer
org-enforce-todo-checkbox-dependencies t Checkboxes must be checked before a TODO can become DONE
org-id-link-to-org-use-id t Prefer use of ID property for links
Tags: emacs org-mode
16 Mar 2019

TIL: prompt matters to org-mode

A workmate just embellished some shell code blocks I'd put in a shared org-mode file with :session s. When I tried to run the blocks with sessions my emacs just froze up though. I found a post on the emacs StackExchange that offered a possible cause for it: the prompt.

I'm using bash-it so my prompt is rather far from the default.

After inspecting the session buffer simply added the following to my ~/.bashrc

if [[ ${TERM} == "dumb" ]]; then
    export BASH_IT_THEME='standard'
else
    export BASH_IT_THEME='simple'
fi

and now I can finally run shell code blocks in sessions.

Tags: emacs org-mode
28 Jan 2019

A missing piece in my Emacs/Spacemacs setup for Haskell development

With the help of a work mate I've finally found this gem that's been missing from my Spacemacs setup

(with-eval-after-load 'intero
  (flycheck-add-next-checker 'intero '(warning . haskell-hlint))
  (flycheck-add-next-checker 'intero '(warning . haskell-stack-ghc)))
Tags: haskell emacs spacemacs
14 Jul 2018

QuickCheck on a REST API

Since I'm working with web stuff nowadays I thought I'd play a little with translating my old post on using QuickCheck to test C APIs to the web.

The goal and how to reach it

I want to use QuickCheck to test a REST API, just like in the case of the C API the idea is to

  1. generate a sequence of API calls (a program), then
  2. run the sequence against a model, as well as
  3. run the sequence against the web service, and finally
  4. compare the resulting model against reality.

The REST API

I'll use a small web service I'm working on, and then concentrate on only a small part of the API to begin with.

The parts of the API I'll use for the programs at this stage are

Method Route Example in Example out
POST /users {"userId": 0, "userName": "Yogi Berra"} {"userId": 42, "userName": "Yogi Berra"}
DELETE /users/:id    

The following API calls will also be used, but not in the programs

Method Route Example in Example out
GET /users   [0,3,7]
GET /users/:id   {"userId": 42, "userName": "Yogi Berra"}
POST /reset    

Representing API calls

Given the information about the API above it seems the following is enough to represent the two calls of interest together with a constructor representing the end of a program

data ApiCall = AddUser Text
             | DeleteUser Int
             | EndProgram
             deriving (Show)

and a program is just a sequence of calls, so list of ApiCall will do. However, since I want to generate sequences of calls, i.e. implement Arbitrary, I'll wrap it in a newtype

newtype Program = Prog [ApiCall]

Running against a model (simulation)

First of all I need to decide what model to use. Based on the part of the API I'm using I'll use an ordinary dictionary of Int and Text

type Model = M.Map Int Text

Simulating execution of a program is simulating each call against a model that's updated with each step. I expect the final model to correspond to the state of the real service after the program is run for real. The simulation begins with an empty dictionary.

simulateProgram :: Program -> Model
simulateProgram (Prog cs) = foldl simulateCall M.empty cs

The simulation of the API calls must then be a function taking a model and a call, returning an updated model

simulateCall :: Model -> ApiCall -> Model
simulateCall m (AddUser t) = M.insert k t m
  where
    k = succ $ foldl max 0 (M.keys m)
simulateCall m (DeleteUser k) = M.delete k m
simulateCall m EndProgram = m

Here I have to make a few assumptions. First, I assume the indeces for the users start on 1. Second, that the next index used always is the successor of highest currently used index. We'll see how well this holds up to reality later on.

Running against the web service

Running the program against the actual web service follows the same pattern, but here I'm dealing with the real world, so it's a little more messy, i.e. IO is involved. First the running of a single call

runCall :: Manager -> ApiCall -> IO ()
runCall mgr (AddUser t) = do
  ireq <- parseRequest "POST http://localhost:3000/users"
  let req = ireq { requestBody = RequestBodyLBS (encode $ User 0 t)}
  resp <- httpLbs req mgr
  guard (status201 == responseStatus resp)

runCall mgr (DeleteUser k) = do
  req <- parseRequest $ "DELETE http://localhost:3000/users/" ++ show k
  resp <- httpNoBody req mgr
  guard (status200 == responseStatus resp)

runCall _ EndProgram = return ()

The running of a program is slightly more involved. Of course I have to set up the Manager needed for the HTTP calls, but I also need to

  1. ensure that the web service is in a well-known state before starting, and
  2. extract the state of the web service after running the program, so I can compare it to the model
runProgram :: Program -> IO Model
runProgram (Prog cs) = do
  mgr <- newManager defaultManagerSettings
  resetReq <- parseRequest "POST http://localhost:3000/reset"
  httpNoBody resetReq mgr
  mapM_ (runCall mgr) cs
  model <- extractModel mgr
  return model

The call to POST /reset resets the web service. I would have liked to simply restart the service completely, but I failed in automating it. I think I'll have to take a closer look at the implementation of scotty to find a way.

Extracting the web service state and packaging it in a Model is a matter of calling GET /users and then repeatedly calling GET /users/:id with each id gotten from the first call

extractModel :: Manager -> IO Model
extractModel mgr = do
  req <- parseRequest "http://localhost:3000/users"
  resp <- httpLbs req mgr
  let (Just ids) = decode (responseBody resp) :: Maybe [Int]
  users <- forM ids $ \ id -> do
    req <- parseRequest $ "http://localhost:3000/users/" ++ show id
    resp <- httpLbs req mgr
    let (Just (user:_)) = decode (responseBody resp) :: Maybe [User]
    return user
  return $ foldl (\ map (User id name) -> M.insert id name map) M.empty users

Generating programs

My approach to generating a program is based on the idea that given a certain state there is only a limited number of possible calls that make sense. Given a model m it makes sense to make one of the following calls:

  • add a new user
  • delete an existing user
  • end the program

Based on this writing genProgram is rather straight forward

genProgram :: Gen Program
genProgram = Prog <$> go M.empty
  where
    possibleAddUser _ = [AddUser <$> arbitrary]
    possibleDeleteUser m = map (return . DeleteUser) (M.keys m)
    possibleEndProgram _ = [return EndProgram]

    go m = do
      let possibles = possibleDeleteUser m ++ possibleAddUser m ++ possibleEndProgram m
      s <- oneof possibles
      let m' = simulateCall m s
      case s of
        EndProgram -> return []
        _ -> (s:) <$> go m'

Armed with that the Arbitrary instance for Program can be implemented as1

instance Arbitrary Program where
  arbitrary = genProgram
  shrink p = []

The property of an API

The steps in the first section can be used as a recipe for writing the property

prop_progCorrectness :: Program -> Property
prop_progCorrectness program = monadicIO $ do
  let simulatedModel = simulateProgram program
  runModel <- run $ runProgram program
  assert $ simulatedModel == runModel

What next?

There are some improvements that I'd like to make:

  • Make the generation of Program better in the sense that the programs become longer. I think this is important as I start tackling larger APIs.
  • Write an implementation of shrink for Program. With longer programs it's of course more important to actually implement shrink.

I'd love to hear if others are using QuickCheck to test REST APIs in some way, if anyone has suggestions for improvements, and of course ideas for how to implement shrink in a nice way.

Footnotes:

1

Yes, I completely skip the issue of shrinking programs at this point. This is OK at this point though, because the generated =Programs=s do end up to be very short indeed.

Tags: emacs haskell flycheck
Other posts