Posts tagged "emacs":
More on tree-sitter and consult
Here's a few things that I've gotten help with figuring out during the last few days. Both things are related to my playing with tree-sitter that I've written about earlier, here and here.
You might also be interested in the two repositories where the full code is. (I've linked to the specific commits as of this writing.)
Anonymous nodes and matching in tree-sitter
In the grammar for Cabal I have a rule for sections that like this
sections: $ => repeat1(choice( $.benchmark, $.common, $.executable, $.flag, $.library, $.source_repository, $.test_suite, )),
where each section followed this pattern
benchmark: $ => seq( repeat($.comment), 'benchmark', field('name', $.section_name), field('properties', $.property_block), ),
This made it a little bit difficult to capture the relevant parts of each
section to implement consult-cabal. I thought a pattern like this ought to
work
(cabal (sections (_ _ @type name: (section_name)? @name)))
but it didn't; I got way too many things captured in type. Clearly I had
misunderstood something about the wildcards, or the query syntax. I attempted to
add a field name to the anonymous node, i.e. change the sections rules like this
benchmark: $ => seq( repeat($.comment), field('type', 'benchmark'), field('name', $.section_name), field('properties', $.property_block), ),
It was accepted by tree-sitter generate, but the field type was nowhere to
be found in the parse tree.
Then I changed the query to list the anonymous nodes explicitly, like this
(cabal (sections (_ ["benchmark" "common" "executable" ...] @type name: (section_name)? @name)))
That worked, but listing all the sections like that in the query didn't sit right with me.
Luckily there's a discussions area in tree-sitters GitHub so a fairly short
discussion later I had answers to why my query behaved like it did and a
solution that would allow me to not list all the section types in the query. The
trick is to wrap the string in a call to alias to make it a named node. After
that it works to add a field name to it as well, of course. The section rules
now look like this
benchmark: $ => seq( repeat($.comment), field('type', alias('benchmark', $.section_type)), field('name', $.section_name), field('properties', $.property_block), ),
and the final query looks like this
(cabal (sections (_ type: (section_type) @type name: (section_name)? @name)))
With that in place I could improve on the function that collects all the items
for consult-cabal so it now show the section's type and name instead of the
string representation of the tree-sitter node.
State in a consult source for preview of lines in a buffer
I was struggling with figuring out how to make a good state function in order
to preview the items in consult-cabal. The GitHub repo for consult doesn't
have discussions enabled, but after a discussion in an issue I'd arrived at a
state function that works very well.
The state function makes use of functions in consult and looks like this
(defun consult-cabal--state () "Create a state function for previewing sections." (let ((state (consult--jump-state))) (lambda (action cand) (when cand (let ((pos (get-text-property 0 'section-pos cand))) (funcall state action pos))))))
The trick here was to figure out how the function returned by
consult--jump-state actually works. On the surface it looks like it takes an
action and a candidate, (lambda (action cand) ...). However, the argument
cand shouldn't be the currently selected item, but rather a postion (ideally a
marker), so I had to attach another text property on the items (section-pos,
which is fetched in the inner lambda). This position is then what's passed to
the function returned by consult--jump-state.
In hindsight it seems so easy, but I was struggling with this for an entire evening before finally asking the question the morning after.
Cabal, tree-sitter, and consult
After my last post I thought I'd move on to implement the rest of the functions
in haskell-mode's major mode for Cabal, functions like
haskell-cabal-goto-library-section and
haskell-cabal-goto-executable-section. Then I realised that what I really
want is a way to quickly jump to any section, that is, I want consult-cabal!
What follows is very much a work-in-progress, but hopefully it'll show enough promise.
Listing the sections
As I have a tree-sitter parse tree to hand it is fairly easy to fetch all the
nodes corresponding to sections. Since the last post I've made some
improvements to the parser and now the parse tree looks like this (I can
recommend the function treesit-explore-mode to expore the parse tree, I've
found it invaluable ever since I realised it existed)
(cabal ... (properties ...) (sections (common common (section_name) ...) (library library ...) (executable executable (section_name) ...) ...))
That is, all the sections are children of the node called sections.
The function to use for fetching all the nodes is treesit-query-capture, it
needs a node to start on, which this case should be the full parse tree,
i.e. (treesit-buffer-root-node 'cabal) and a query string. Given the
structure of the parse tree, and that I want to capture all children of
sections, a query string like this one works
"(cabal (sections (_)* @section))"
Finally, by default treesit-query-capture returns a list of tuples of the form
(<capture> . <node>), but in this case I only want the list of nodes, so the
full call will look like this
(treesit-query-capture (treesit-buffer-root-node 'cabal) "(cabal (sections (_)* @section))" nil nil t)
Hooking it up to consult
As I envision adding more things to jump to in the future, I decided to make use
of consult--multi. That in turn means I need to define a "source" for the
sections. After a bit of digging and rummaging in the consult source I put
together this
(defvar consult-cabal--source-section `(:name "Sections" :category location :action ,#'consult-cabal--section-action :items ,#'consult-cabal--section-items) "Definition of source for Cabal sections.")
which means I need two functions, consult-cabal--section-action and
consult-cabal--section-items. I started with the latter.
Getting section nodes as items for consult
It took me a while to work understand how this would ever be able to work. The
function that :items point to must return a list of strings, but how would I
ever be able to use just a string to jump to the correct location?
The solution is in a comment in the documentation of consult--multi:
:items - List of strings to select from or function returning list of strings. Note that the strings can use text properties to carry metadata, which is then available to the :annotate, :action and :state functions.
I'd never come across text properties in Emacs before, so at first I
completely missed those two words. Once I'd looked up the concept in the
documentation everything fell into place. The function
consult-cabal--section-items would simply attach the relevant node as a text
property to the strings in the list.
My current version, obviously a work-in-progress, takes a list of nodes and turns them naïvely into a string and attaches the node. I split it into two functions, like this
(defun consult-cabal--section-to-string (section) "Convert a single SECTION node to a string." (propertize (format "%S" section) :treesit-node section)) (defun consult-cabal--section-items () "Fetch all sections as a list of strings ." (let ((section-nodes (treesit-query-capture (treesit-buffer-root-node 'cabal) "(cabal (sections (_)* @section))" nil nil t))) (mapcar #'consult-cabal--section-to-string section-nodes)))
Implementing the action
The action function is called with the selected item, i.e. with the string and
its properties. That means, to jump to the selected section the function needs
to extract the node property, :treesit-node, and jump to the start of it. the
function to use is get-text-property, and as all characters in the string will
have to property I just picked the first one. The jumping itself I copied from
the navigation functions I'd written before.
(defun consult-cabal--section-action (item) "Go to the section referenced by ITEM." (when-let* ((node (get-text-property 0 :treesit-node item)) (new-pos (treesit-node-start node))) (goto-char new-pos)))
Tying it together with consult--multi
The final function, consult-cabal, looks like this
(defun consult-cabal () "Choose a Cabal construct and jump to it." (interactive) (consult--multi '(consult-cabal--source-section) :sort nil))
Conclusions and where to find the code
The end result works as intended, but it's very rough. I'll try to improve it a bit more. In particular I want
- better strings -
(format "%S" node)is all right to start with, but in the long run I want strings that describe the sections, and - preview as I navigate between items - AFAIU this is what the
:statefield is for, but I still haven't looked into how it works.
The source can be found here.
Making an Emacs major mode for Cabal using tree-sitter
A few days ago I posted on r/haskell that I'm attempting to put together a Cabal grammar for tree-sitter. Some things are still missing, but it covers enough to start doing what I initially intended: experiment with writing an alternative Emacs major mode for Cabal.
The documentation for the tree-sitter integration is very nice, and several of
the major modes already have tree-sitter variants, called X-ts-mode where X
is e.g. python, so putting together the beginning of a major mode wasn't too
much work.
Configuring Emacs
First off I had to make sure the parser for Cabal was installed. The snippet for that looks like this1
(use-package treesit :straight nil :ensure nil :commands (treesit-install-language-grammar) :init (setq treesit-language-source-alist '((cabal . ("https://gitlab.com/magus/tree-sitter-cabal.git")))))
With that in place the parser is installed using M-x
treesit-install-language-grammar and choosing cabal.
After that I removed my configuration for haskell-mode and added the following
snippet to get my own major mode into my setup.
(use-package my-cabal-mode :straight (:type git :repo "git@gitlab.com:magus/my-emacs-pkgs.git" :branch "main" :files (:defaults "my-cabal-mode/*el")))
The major mode and font-locking
The built-in elisp documentation actually has a section on writing a major mode with tree-sitter, so it was easy to get started. Setting up the font-locking took a bit of trial-and-error, but once I had comments looking the way I wanted it was easy to add to the setup. Oh, and yes, there's a section on font-locking with tree-sitter in the documentation too. At the moment it looks like this
(defvar cabal--treesit-font-lock-setting (treesit-font-lock-rules :feature 'comment :language 'cabal '((comment) @font-lock-comment-face) :feature 'cabal-version :language 'cabal '((cabal_version _) @font-lock-constant-face) :feature 'field-name :language 'cabal '((field_name) @font-lock-keyword-face) :feature 'section-name :language 'cabal '((section_name) @font-lock-variable-name-face)) "Tree-sitter font-lock settings.") ;;;###autoload (define-derived-mode my-cabal-mode fundamental-mode "My Cabal" "My mode for Cabal files" (when (treesit-ready-p 'cabal) (treesit-parser-create 'cabal) ;; set up treesit (setq-local treesit-font-lock-feature-list '((comment field-name section-name) (cabal-version) () ())) (setq-local treesit-font-lock-settings cabal--treesit-font-lock-setting) (treesit-major-mode-setup))) ;;;###autoload (add-to-list 'auto-mode-alist '("\\.cabal\\'" . my-cabal-mode))
Navigation
One of the reasons I want to experiment with tree-sitter is to use it for code
navigation. My first attempt is to translate haskell-cabal-section-beginning
(in haskell-mode, the source) to using tree-sitter. First a convenience
function to recognise if a node is a section or not
(defun cabal--node-is-section-p (n) "Predicate to check if treesit node N is a Cabal section." (member (treesit-node-type n) '("benchmark" "common" "executable" "flag" "library" "test_suite")))
That makes it possible to use treesit-parent-until to traverse the nodes until
hitting a section node
(defun cabal-goto-beginning-of-section () "Go to the beginning of the current section." (interactive) (when-let* ((node-at-point (treesit-node-at (point))) (section-node (treesit-parent-until node-at-point #'cabal--node-is-section-p)) (start-pos (treesit-node-start section-node))) (goto-char start-pos)))
And the companion function, to go to the end of a section is very similar
(defun cabal-goto-end-of-section () "Go to the end of the current section." (interactive) (when-let* ((node-at-point (treesit-node-at (point))) (section-node (treesit-parent-until node-at-point #'cabal--node-is-section-p)) (end-pos (treesit-node-end section-node))) (goto-char end-pos)))
Footnotes:
I'm using straight.el and use-package in my setup, but hopefully the
snippets can easily be converted to other ways of configuring Emacs.
Per-project xref history in Emacs
When I write code I jump around in the code quite a bit, as I'm sure many other developers do. The ability to jump to the definition of a function, or a type, is invaluable when trying to understand code. In Emacs the built-in xref package provides the basic functionality for this, with many other packages providing their custom functions for looking up identifiers. This works beautifully except for one thing, there's only one global stack for keeping track of how you've jumped around.
Well, that used to be the case.
As I tend to have multiple projects open at a time I used to find it very
confusing when I pop positions off the xref stack and all of a sudden find
myself in another project. It would be so much nicer to have a per-project
stack.
I've only known of one solution for this, the perspective package, but as I've been building my own Emacs config I wanted to see if there were other options. It turns out there is one (almost) built into Emacs 29.
In Emacs 29 there's built-in support for having per-window xref stacks, and
the way that's done allows one to extend it further. There's now a variable,
xref-history-storage, that controls access to the xref stack. The default is
still a global stack (when the variable is set to #'xref-global-history), but
to get per-window stacks one sets it to #'xref-window-local-history.
After finding this out I thought I'd try to write my own, implementing
per-project xref stacks (for projectile).
The function should take one optional argument, new-value, if it's provided
the stack should be updated and if not, it should be returned. That is,
something like this
(defun projectile-param-xref-history (&optional new-value) "Return project-local xref history for the current projectile. Override existing value with NEW-VALUE if it's set." (if new-value (projectile-param-set-parameter 'xref--history new-value) (or (projectile-param-get-parameter 'xref--history) (projectile-param-set-parameter 'xref--history (xref--make-xref-history)))))
Now I only had to write the two functions projectile-param-get-parameter and
projectile-param-set-parameter. I thought a rather straight forward option
would be to use a hashtable and store values under a tuple comprising the
project name and the parameter passed in.
(defvar projectile-params--store (make-hash-table :test 'equal) "The store of project parameters.") (defun projectile-param-get-parameter (param) "Return project parameter PARAM, or nil if unset." (let ((key (cons (projectile-project-name) param))) (gethash key projectile-params--store nil))) (defun projectile-param-set-parameter (param value) "Set the project parameter PARAM to VALUE." (let ((key (cons (projectile-project-name) param))) (puthash key value projectile-params--store)) value)
Then I tried it out by setting xref-history-storage
(setq xref-history-storage #'projectile-param-xref-history)
and so far it's been working well.
The full code is here.
Annotate projects in Emacs
Every now and then I've wished to write comments on files in a project, but I've
never found a good way to do that. annotate.el and org-annotate-file both
collect annotations in a central place (in my $HOME), while marginalia puts
annotations in files next to the source files but in a format that's rather
cryptic and tends to be messed up when attached to multiple lines. None of them
is ideal, I'd like the format to be org-mode, but not in a central file. At the
same time having one annotation file per source file is simply too much.
I tried wrapping org-annotate-file, setting org-annotate-file-storage-file
and taking advantage of elisp's dynamic binding. However, it opens the
annotation file in the current window, and I'd really like to split the window
and open the annotations the right. Rather than trying to sort of "work it out
backwards" I decided to write a small package and use as much of the
functionality in org-annotate-file.el as possible.
First off I decided that I want the annotation file to be called
projectile-annotations.org.
(defvar org-projectile-annotate-file-name "projectile-annotations.org" "The name of the file to store project annotations.")
Then I wanted a slightly modified version of org-annotate-file-show-section, I
wanted it to respect the root of the project.
(defun org-projectile-annotate--file-show-section (storage-file) "Add or show annotation entry in STORAGE-FILE and return the buffer." ;; modified version of org-annotate-file-show-section (let* ((proj-root (projectile-project-root)) (filename (file-relative-name buffer-file-name proj-root)) (line (buffer-substring-no-properties (point-at-bol) (point-at-eol))) (annotation-buffer (find-file-noselect storage-file))) (with-current-buffer annotation-buffer (org-annotate-file-annotate filename line)) annotation-buffer))
The main function can then simply work out where the file with annotations
should be located and call org-projectile-annotate--file-show-section.
(defun org-projectile-annotate () (interactive) (let ((annot-fn (file-name-concat (projectile-project-root) org-projectile-annotate-file-name))) (set-window-buffer (split-window-right) (org-projectile-annotate--file-show-section annot-fn))))
When testing it all out I noticed that org-store-link makes a link with a
search text. In my case it would be much better to have links with line numbers.
I found there's a hook to modify the behaviour of org-store-link,
org-create-file-search-functions. So I wrote a function to get the kind of
links I want, but only when the project annotation file is open in a buffer.
(defun org-projectile-annotate-file-search-func () "A function returning the current line number when called in a project while the project annotation file is open. This function is designed for use in the hook 'org-create-file-search-functions'. It changes the behaviour of 'org-store-link' so it constructs a link with a line number instead of a search string." ;; TODO: find a way to make the link description nicer (when (and (projectile-project-p) (get-buffer-window org-projectile-annotate-file-name)) (number-to-string (line-number-at-pos))))
That's it, now I only have to wait until the next time I want to comment on a project to see if it improves my way of working.
Playing with setting up Emacs
TL;DR: I've put together a small-ish Emacs configuration that I call MES. Hopefully it can be of use to someone out there.
My Emacs Setup - MES
The other day I started watching some videos in the Emacs From Scratch series from System Crafters. It looked like something that could be fun to play with so over the last couple of days I've been tinkering with putting together the beginnings of a configuration.
During the process I realised just how much work it'd be to put together something that comes close to the polish of Spacemacs, so I've currently no intention of actually using MES myself. It was fun though, and maybe it can serve as inspiration (or as a deterrent) for someone else.
The major parts are
Power-mode in Spacemacs
I just found the Power Mode for Emacs. If you want to try it out in Spacemacs
you can make sure that your ~/.spacemacs contains the following
dotspacemacs-additional-packages '( ... (power-mode :location (recipe :fetcher github :repo "elizagamedev/power-mode.el")) )
After a restart Power Mode can be turned on using SPC SPC power-mode.
Unfortunately I found that it slows down rendering so badly that Emacs isn't keeping up with my typing. Even though I removed it right away again it was fun to try it out, and I did learn how to add package to Spacemacs that aren't on MELPA.
A useful resource is this reference on the recipe format.
Comments and org-static-blog
I'm using org-static-blog to generate the contents of this site. So far I'm very happy with it, but I've gotten a few emails from readers who've wanted to comment on something I've written and they always point out that it's not easy to do. It's actually not a coincidence that it's a bit difficult!
Yesterday I came up with a way that might make is slightly easier without involving JavaScript from a 3rd party. By making use of the built-in support for adding HTML code for comments. One slight limitation is that it's a single variable holding the code, and I'd really like to allow for both
- using a link to a discussion site, e.g. reddit, as well as
- my email address
As the comment support in org-static-blog comes in the form of a single variable
this seems a bit difficult to accomplish. However, it isn't difficult at all to
do in elisp due to the power of advice-add.
By using the following advice on org-static-blog-publish-file
(advice-add 'org-static-blog-publish-file :around (lambda (orig-fn filename &rest args) (let* ((comments-url (with-temp-buffer (insert-file-contents filename) (or (cadar (org-collect-keywords '("commentsurl"))) my-blog-default-comments-url))) (org-static-blog-post-comments (concat "Comment <a href=" comments-url ">here</a>."))) (apply orig-fn filename args))))
and defining my-blog-default-comments-url to a mailto:... URL I get a link
to use for commenting by either
- set
commentsurlto point to discussion about the post on reddit, or - not set
commentsurlat all and get themailto:...URL.
If you look at my previous post you see the result of the former, and if you look below you see the result of the latter.
Keeping Projectile's cache tidy
A while back I added a function to find all projects recursively from a
directory and add them to Projectile's cache (see the Populating Projectile's
cache). Since then I've just made a tiny change to also include containing a
.projectile file.
(defun projectile-extra-add-projects-in-subfolders (projects-root) (interactive (list (read-directory-name "Add to known projects: "))) (message "Searching for projects in %s..." projects-root) (let* ((proj-rx (rx (and line-start ?. (or "projectile" "git") line-end))) (dirs (seq-map 'file-name-directory (directory-files-recursively projects-root proj-rx t)))) (seq-do 'projectile-add-known-project dirs) (message "Added %d projects" (length dirs))))
Since then I've also found a need for tidying the cache, in my casse that means removing entries in the cache that no longer exist on disk. I didn't find a function for it, so I wrote one.
(defun projectile-extra-tidy-projects () (interactive) (let ((missing-dirs (seq-remove 'file-directory-p projectile-known-projects))) (seq-do 'projectile-remove-known-project missing-dirs) (message "Tidied %d projects" (length missing-dirs))))
Trimming newline on code block variable
Today I found ob-http and decided to try it out a little. I quickly ran into a problem of a trailing newline. Basically I tried to do something like this:
#+begin_src http :select .id :cache yes POST /foo Content-Type: application/json { "foo": "toto", "bar": "tata" } #+end_src #+begin_example 48722051-f81b-433f-acb4-a65d961ec841 #+end_example #+begin_src http POST /foo/${id}/fix #+end_src
The trailing newline messes up the URL though, and the second code block fails.
I found two ways to deal with it, using a table and using org-sbe
Using a table
#+begin_src http :select .id :cache yes :results table POST /foo Content-Type: application/json { "foo": "toto", "bar": "tata" } #+end_src #+begin_example | 48722051-f81b-433f-acb4-a65d961ec841 | #+end_example #+begin_src http POST /foo/${id}/fix #+end_src
Using org-sbe
#+begin_src http :select .id :cache yes POST /foo Content-Type: application/json { "foo": "toto", "bar": "tata" } #+end_src #+begin_example 48722051-f81b-433f-acb4-a65d961ec841 #+end_example #+begin_src http POST /foo/${id}/fix #+end_src
Magit/forge and self-hosted GitLab
As I found the documentation for adding a self-hosted instance of GitLab to to magit/forge a bit difficult, I thought I'd write a note for my future self (and anyone else who might find it useful).
First put the following in `~/.gitconfig`
[gitlab "gitlab.private.com/api/v4"] user = my.username
Then create an access token on GitLab. I ticked api and write_repository,
which seems to work fine so far. Put the token in ~/.authinfo.gpg
machine gitlab.private.com/api/v4 login my.user^forge password <token>
(Remember that a newline is needed at the end of the file.)
Finally, add the GitLab instance to 'forge-alist
(setq forge-alist '(("gitlab.private.com" "gitlab.private.com/api/v4" "gitlab.private.com" forge-gitlab-repository) ("github.com" "api.github.com" "github.com" forge-github-repository) ("gitlab.com" "gitlab.com/api/v4" "gitlab.com" forge-gitlab-repository)) )
That's it!
Keeping todo items in org-roam v2
Org-roam v2 has been released and yes, it broke my config a bit. Unfortunately the v1-to-v2 upgrade wizard didn't work for me. I realized later that it might have been due to the roam-related functions I'd hooked into `'before-save-hook`. I didn't think about it until I'd already manually touched up almost all my files (there aren't that many) so I can't say anything for sure. However, I think it might be a good idea to keep hooks in mind if one runs into issues with upgrading.
The majority of the time I didn't spend on my notes though, but on the setup I've written about in an earlier post, Keeping todo items in org-roam. Due to some of the changes in v2, changes that I think make org-roam slightly more "org-y", that setup needed a bit of love.
The basis is still the same 4 functions I described in that post, only the details had to be changed.
I hope the following is useful, and as always I'm always happy to receive commends and suggestions for improvements.
Some tag helpers
The very handy functions for extracting tags as lists seem to be gone, in their
place I found org-roam-{get,set}-keyword. Using these I wrote three wrappers
that allow slightly nicer handling of tags.
(defun roam-extra:get-filetags () (split-string (or (org-roam-get-keyword "filetags") ""))) (defun roam-extra:add-filetag (tag) (let* ((new-tags (cons tag (roam-extra:get-filetags))) (new-tags-str (combine-and-quote-strings new-tags))) (org-roam-set-keyword "filetags" new-tags-str))) (defun roam-extra:del-filetag (tag) (let* ((new-tags (seq-difference (roam-extra:get-filetags) `(,tag))) (new-tags-str (combine-and-quote-strings new-tags))) (org-roam-set-keyword "filetags" new-tags-str)))
The layer
roam-extra:todo-p needed no changes at all. I'm including it here only for
easy reference.
(defun roam-extra:todo-p () "Return non-nil if current buffer has any TODO entry. TODO entries marked as done are ignored, meaning the this function returns nil if current buffer contains only completed tasks." (org-element-map (org-element-parse-buffer 'headline) 'headline (lambda (h) (eq (org-element-property :todo-type h) 'todo)) nil 'first-match))
As pretty much all functions I used in the old version of
roam-extra:update-todo-tag are gone I took the opportunity to rework it
completely. I think it ended up being slightly simpler. I suspect the the use of
org-with-point-at 1 ... is unnecessary, but I haven't tested it yet so I'm
leaving it in for now.
(defun roam-extra:update-todo-tag () "Update TODO tag in the current buffer." (when (and (not (active-minibuffer-window)) (org-roam-file-p)) (org-with-point-at 1 (let* ((tags (roam-extra:get-filetags)) (is-todo (roam-extra:todo-p))) (cond ((and is-todo (not (seq-contains-p tags "todo"))) (roam-extra:add-filetag "todo")) ((and (not is-todo) (seq-contains-p tags "todo")) (roam-extra:del-filetag "todo")))))))
In the previous version roam-extra:todo-files was built using an SQL query.
That felt a little brittle to me, so despite that my original inspiration
contains an updated SQL query I decided to go the route of using the org-roam
API instead. The function org-roam-node-list makes it easy to get all nodes
and then finding the files is just a matter of using seq-filter and seq-map.
Now that headings may be nodes, and that heading-based nodes seem to inherit the
top-level tags, a file may appear more than once, hence the call to seq-unique
at the end.
Based on what I've seen V2 appears less eager to sync the DB, so to make sure all nodes are up-to-date it's best to start off with forcing a sync.
(defun roam-extra:todo-files () "Return a list of roam files containing todo tag." (org-roam-db-sync) (let ((todo-nodes (seq-filter (lambda (n) (seq-contains-p (org-roam-node-tags n) "todo")) (org-roam-node-list)))) (seq-uniq (seq-map #'org-roam-node-file todo-nodes))))
With that in place it turns out that also roam-extra:update-todo-files worked
without any changes. I'm including it here for easy reference as well.
(defun roam-extra:update-todo-files (&rest _) "Update the value of `org-agenda-files'." (setq org-agenda-files (roam-extra:todo-files)))
Hooking it up
The variable org-roam-file-setup-hook is gone, so the the more general
find-file-hook will have to be used instead.
(add-hook 'find-file-hook #'roam-extra:update-todo-tag) (add-hook 'before-save-hook #'roam-extra:update-todo-tag) (advice-add 'org-agenda :before #'roam-extra:update-todo-files)
Todo items in org-roam, an update
I got an email from Mr Z with a nice modification to the code in my post on keeping todo items in org-roam.
He already had a bunch of agenda files that he wanted to keep using (I had so few of them that I'd simply converted them to roam files). Here's the solution he shared with me:
(defvar roam-extra-original-org-agenda-files nil "Original value of `org-agenda-files'.") (defun roam-extra:update-todo-files (&rest _) "Update the value of `org-agenda-files'." (unless roam-extra-original-org-agenda-files (setq roam-extra-original-org-agenda-files org-agenda-files)) (setq org-agenda-files (append roam-extra-original-org-agenda-files (roam-extra:todo-files))))
It's a rather nice modification I think. Thanks to Mr Z for agreeing to let me share it here.
Keeping todo items in org-roam
A while ago I made an attempt to improve my work habits by keeping a document
with TODO items. It lasted only for a while, and I've since had the intention to
make another attempt. Since then I've started using org-roam and I've managed to
create a habit of writing daily journal notes using org-roam's daily-notes. A
few times I've thought that it might fit me well to put TODO items in the notes,
but that would mean that I'd have to somehow keep track of them. At first I
manually added a tag to each journal fily containing a TODO item. That didn't
work very well at all, which should have been obvious up front. Then I added the
folders where I keep roam files and journals to org-agenda-files, that worked
a lot better. I'd still be using that, even if I expected it to slow down
considerably as the number of files grow, but then I found a post on dynamic and
fast agenda with org-roam.
I adjusted it slightly to fit my own setup a bit better, i.e. I made a Spacemacs
layer, roam-extra, I use the tag todo, and I use a different hook to get the
tag added on opening an org-roam file.
The layer consists of a single file, layers/roam-extra/funcs.el. In it I
define 4 functions (they are pretty much copies of the functions in the post
linked above):
roam-extra:todo-p- returns non-nilif the current current buffer contains a TODO item.roam-extra:update-todo-tag- updates the tags of the current buffer to reflect the presence of TODO items, i.e. ensure the the tagtodois present iff there's a TODO item.roam-extra:todo-files- uses the org-roam DB to return a list of all files containing the tagtodo.roam-extra:update-todo-files- adjusts'org-agenda-filesto contain only the files with TODO items.
I've put the full contents of the file at the end of the post.
To ensure that the todo tag is correct in all org-mode files I've added
roam-extra:update-todo-tag to hooks that are invoked on opening an org-ram
file and when saving a file. (I would love to find a more specialise hook than
before-save-hook, but it works for now.)
(add-hook 'org-roam-file-setup-hook #'roam-extra:update-todo-tag) (add-hook 'before-save-hook #'roam-extra:update-todo-tag)
To ensure that the list of files with TODO items is kept up to date when I open
I also wrap org-agenda in an advice so roam-extra:update-todo-files is
called prior to the agenda being opened.
(advice-add 'org-agenda :before #'roam-extra:update-todo-files)
The layer, layers/roam-extra/funcs.el
(defun roam-extra:todo-p () "Return non-nil if current buffer has any TODO entry. TODO entries marked as done are ignored, meaning the this function returns nil if current buffer contains only completed tasks." (org-element-map (org-element-parse-buffer 'headline) 'headline (lambda (h) (eq (org-element-property :todo-type h) 'todo)) nil 'first-match)) (defun roam-extra:update-todo-tag () "Update TODO tag in the current buffer." (when (and (not (active-minibuffer-window)) (org-roam--org-file-p buffer-file-name)) (let* ((file (buffer-file-name (buffer-base-buffer))) (all-tags (org-roam--extract-tags file)) (prop-tags (org-roam--extract-tags-prop file)) (tags prop-tags)) (if (roam-extra:todo-p) (setq tags (seq-uniq (cons "todo" tags))) (setq tags (remove "todo" tags))) (unless (equal prop-tags tags) (org-roam--set-global-prop "roam_tags" (combine-and-quote-strings tags)))))) (defun roam-extra:todo-files () "Return a list of note files containing todo tag." (seq-map #'car (org-roam-db-query [:select file :from tags :where (like tags (quote "%\"todo\"%"))]))) (defun roam-extra:update-todo-files (&rest _) "Update the value of `org-agenda-files'." (setq org-agenda-files (roam-extra:todo-files)))
Flycheck and HLS
I've been using LSP for most programming languages for a while now. HLS is
really very good now, but I've found that it doesn't warn on quite all things
I'd like it to so I find myself having to swap between the 'lsp and
'haskell-ghc checkers. However, since flycheck supports chaining checkers I
thought there must be a way to have both checkers active at the same time.
The naive approach didn't work due to load order of things in Spacemacs so I had to experiment a bit to find something that works.
The first issue was to make sure that HLS is available at all. I use shell.nix
together with direnv extensively and I had noticed that lsp-mode tried to load
HLS before direnv had put it in the $PATH. I think the
'lsp-beforeinitialize-hook is the hook to use for this:
(add-hook 'lsp-before-initialize-hook #'direnv-update-environment))
I made a several attempt to chain the checkers but kept on getting
errors due to the 'lsp checker not being defined yet. Another problem
I ran into was that the checkers were chained too late, resulting in
having to manually run flycheck-buffer on the first file I opened.
(Deferred loading is a brilliant thing, but make some things really
difficult to debug.) After quite a bit of experimenting and reading the
description of various hooks I did find something that works:
(with-eval-after-load 'lsp-mode (defun magthe:lsp-next-checker () (flycheck-add-next-checker 'lsp '(warning . haskell-ghc))) (add-hook 'lsp-lsp-haskell-after-open-hook #'magthe:lsp-next-checker))
Of course I have no idea if this is the easiest or most elegant solution but it does work for my testcases:
- Open a file in a project,
SPC p l- choose project - choose a Haskell file. - Open a project,
SPC p lfollowed byC-d, and then open a Haskell file.
Suggestions for improvements are more than welcome, of course.
Better Nix setup for Spacemacs
In an earlier post I documented my setup for getting Spacemacs/Emacs to work with Nix. I've since found a much more elegant solution based on
- direnv, and
- emacs-direnv
No more Emacs packages for Nix and no need to defining functions that wrap
executables in an invocation of nix-shell.
There's a nice bonus too, with this setup I don't need to run nix-shell, which
always drops me at a bash prompt, instead I get a working setup in my shell of
choice.
Setting up direnv
The steps for setting up direnv depends a bit on your setup, but luckily I
found the official instructions for installing direnv to be very clear and
easy to follow. There's not much I can add to that.
Setting up Spacemacs
Since emacs-direnv isn't included by default in Spacemacs I needed to do a bit
of setup. I opted to create a layer for it, rather than just drop it in the list
dotspacemacs-additional-packages. Yes, a little more complicated, but not
difficult and I nurture an intention of submitting the layer for inclusion in
Spacemacs itself at some point. I'll see where that goes.
For now, I put the following in the file
~/.emacs.d/private/layers/direnv/packages.el:
(defconst direnv-packages '(direnv)) (defun direnv/init-direnv () (use-package direnv :init (direnv-mode)))
Setting up the project folders
In each project folder I then add the file .envrc containing a single line:
use_nix
Then I either run direnv allow from the command line, or run the
function direnv-allow after opening the folder in Emacs.
Using it
It's as simple as moving into the folder in a shell – all required envvars are set up on entry and unset on exit.
In Emacs it's just as simple, just open a file in a project and the envvars are set. When switching to a buffer outside the project the envvars are unset.
There is only one little caveat, nix-build doesn't work inside a Nix shell. I
found out that running
IN_NIX_SHELL= nix-build
does work though.
Nix setup for Spacemacs
Edit 2020-06-22: I've since found a better setup for this.
When using ghcide and LSP, as I wrote about in my post on Haskell, ghcide, and
Spacemacs, I found myself ending up recompiling a little too often. This pushed
me to finally start looking at Nix. After a bit of a fight I managed to
get ghcide from Nix,
which brought me the issue of setting up Spacemacs. Inspired by a gist from
Samuel Evans-Powell and a guide to setting up an environment for Reflex by
Thales Macedo Garitezi I ended up with the following setup:
(defun dotspacemacs/layers () (setq-default ... dotspacemacs-additional-packages '( nix-sandbox nix-haskell-mode ... ) ... ))
(defun dotspacemacs/user-config () ... (add-hook 'haskell-mode-hook #'lsp) (add-hook 'haskell-mode-hook 'nix-haskell-mode) (add-hook 'haskell-mode-hook (lambda () (setq-local flycheck-executable-find (lambda (cmd) (nix-executable-find (nix-current-sandbox) cmd))) (setq-local flycheck-command-wrapper-function (lambda (argv) (apply 'nix-shell-command (nix-current-sandbox) argv))) (setq-local haskell-process-wrapper-function (lambda (argv) (apply 'nix-shell-command (nix-current-sandbox) argv))) (setq-local lsp-haskell-process-wrapper-function (lambda (argv) `("nix-shell" "-I" "." "--command" "ghcide --lsp" ,(nix-current-sandbox)))))) (add-hook 'haskell-mode-hook (lambda () (flycheck-add-next-checker 'lsp-ui '(warning . haskell-stack-ghc)))) ... )
It seems to work, but please let me know if you have suggestions for improvements.
Populating Projectile's cache
As I track the develop branch of Spacemacs I occasionally clean out my cache of projects known to Projectile. Every time it takes a while before I'm back at a stage where I very rarely have to visit something that isn't already in the cache.
However, today I found the function projectile-add-known-project, which
prompted me to write the following function that'll help me quickly re-building
the cache the next time I need to reset Spacemacs to a known state again.
(defun projectile-extra-add-projects-in-subfolders (projects-root) (interactive (list (read-directory-name "Add to known projects: "))) (message "Searching for projects in %s..." projects-root) (let ((dirs (seq-map 'file-name-directory (directory-files-recursively projects-root "^.git$" t)))) (seq-do 'projectile-add-known-project dirs) (message "Added %d projects" (length dirs))))
Ditaa in Org mode
Just found out that Emacs ships with Babel support for ditaa (yes, I'm late to the party).
Sweet! That is yet another argument for converting all our README.md into
README.org at work.
\
The changes I made to my Spacemacs config are
(defun dotspacemacs/user-config () ... (with-eval-after-load 'org ... (add-to-list 'org-babel-load-languages '(ditaa . t)) (setq org-ditaa-jar-path "/usr/share/java/ditaa/ditaa-0.11.jar")) ...)
Haskell, ghcide, and Spacemacs
The other day I read Chris Penner's post on Haskell IDE Support and thought I'd make an attempt to use it with Spacemacs.
After running stack build hie-bios ghcide haskell-lsp --copy-compiler-tool I
had a look at the instructions on using haskell-ide-engine with Spacemacs.
After a bit of trial and error I came up with these changes to my
~/.spacemacs:
(defun dotspacemacs/layers () (setq-default dotspacemacs-configuration-layers '( ... lsp (haskell :variables haskell-completion-backend 'lsp ) ...) ) )
(defun dotspacemacs/user-config () (setq lsp-haskell-process-args-hie '("exec" "ghcide" "--" "--lsp") lsp-haskell-process-path-hie "stack" lsp-haskell-process-wrapper-function (lambda (argv) (cons (car argv) (cddr argv))) ) (add-hook 'haskell-mode-hook #'lsp))
The slightly weird looking lsp-haskell-process-wrapper-function is removing
the pesky --lsp inserted by this line.
That seems to work. Though I have to say I'm not ready to switch from intero just yet. Two things in particular didn't work with =ghcide=/LSP:
- Switching from one the
Main.hsin one executable to theMain.hsof another executable in the same project didn't work as expected – I had hints and types in the first, but nothing in the second. - Jump to the definition of a function defined in the package didn't work – I'm not willing to use GNU GLOBAL or some other source tagging system.
Some OrgMode stuff
The last few days I've watched Rainer König's OrgMode videos. It's resulted in a few new settings that makes Org a little more useful.
| Variable | Value | Description |
|---|---|---|
calendar-week-start-day |
1 |
Weeks start on Monday! |
org-modules (list) |
org-habit |
Support for tracking habits |
org-modules (list) |
org-id |
Improved support for ID property |
org-agenda-start-on-weekday |
1 |
Weeks start on Monday, again! |
org-log-into-drawer |
t |
Put notes (logs) into a drawer |
org-enforce-todo-checkbox-dependencies |
t |
Checkboxes must be checked before a TODO can become DONE |
org-id-link-to-org-use-id |
t |
Prefer use of ID property for links |
TIL: prompt matters to org-mode
A workmate just embellished some shell code blocks I'd put in a shared org-mode
file with :session s. When I tried to run the blocks with sessions my emacs
just froze up though. I found a post on the emacs StackExchange that offered a
possible cause for it: the prompt.
I'm using bash-it so my prompt is rather far from the default.
After inspecting the session buffer simply added the following to my ~/.bashrc
if [[ ${TERM} == "dumb" ]]; then export BASH_IT_THEME='standard' else export BASH_IT_THEME='simple' fi
and now I can finally run shell code blocks in sessions.
A missing piece in my Emacs/Spacemacs setup for Haskell development
With the help of a work mate I've finally found this gem that's been missing from my Spacemacs setup
(with-eval-after-load 'intero (flycheck-add-next-checker 'intero '(warning . haskell-hlint)) (flycheck-add-next-checker 'intero '(warning . haskell-stack-ghc)))
QuickCheck on a REST API
Since I'm working with web stuff nowadays I thought I'd play a little with translating my old post on using QuickCheck to test C APIs to the web.
The goal and how to reach it
I want to use QuickCheck to test a REST API, just like in the case of the C API the idea is to
- generate a sequence of API calls (a program), then
- run the sequence against a model, as well as
- run the sequence against the web service, and finally
- compare the resulting model against reality.
The REST API
I'll use a small web service I'm working on, and then concentrate on only a small part of the API to begin with.
The parts of the API I'll use for the programs at this stage are
| Method | Route | Example in | Example out |
|---|---|---|---|
POST |
/users |
{"userId": 0, "userName": "Yogi Berra"} |
{"userId": 42, "userName": "Yogi Berra"} |
DELETE |
/users/:id |
The following API calls will also be used, but not in the programs
| Method | Route | Example in | Example out |
|---|---|---|---|
GET |
/users |
[0,3,7] |
|
GET |
/users/:id |
{"userId": 42, "userName": "Yogi Berra"} |
|
POST |
/reset |
Representing API calls
Given the information about the API above it seems the following is enough to represent the two calls of interest together with a constructor representing the end of a program
data ApiCall = AddUser Text | DeleteUser Int | EndProgram deriving (Show)
and a program is just a sequence of calls, so list of ApiCall will do.
However, since I want to generate sequences of calls, i.e. implement
Arbitrary, I'll wrap it in a newtype
newtype Program = Prog [ApiCall]
Running against a model (simulation)
First of all I need to decide what model to use. Based on the part of the API
I'm using I'll use an ordinary dictionary of Int and Text
type Model = M.Map Int Text
Simulating execution of a program is simulating each call against a model that's updated with each step. I expect the final model to correspond to the state of the real service after the program is run for real. The simulation begins with an empty dictionary.
simulateProgram :: Program -> Model simulateProgram (Prog cs) = foldl simulateCall M.empty cs
The simulation of the API calls must then be a function taking a model and a call, returning an updated model
simulateCall :: Model -> ApiCall -> Model simulateCall m (AddUser t) = M.insert k t m where k = succ $ foldl max 0 (M.keys m) simulateCall m (DeleteUser k) = M.delete k m simulateCall m EndProgram = m
Here I have to make a few assumptions. First, I assume the indeces for the users
start on 1. Second, that the next index used always is the successor of
highest currently used index. We'll see how well this holds up to reality later
on.
Running against the web service
Running the program against the actual web service follows the same pattern, but
here I'm dealing with the real world, so it's a little more messy, i.e. IO is
involved. First the running of a single call
runCall :: Manager -> ApiCall -> IO () runCall mgr (AddUser t) = do ireq <- parseRequest "POST http://localhost:3000/users" let req = ireq { requestBody = RequestBodyLBS (encode $ User 0 t)} resp <- httpLbs req mgr guard (status201 == responseStatus resp) runCall mgr (DeleteUser k) = do req <- parseRequest $ "DELETE http://localhost:3000/users/" ++ show k resp <- httpNoBody req mgr guard (status200 == responseStatus resp) runCall _ EndProgram = return ()
The running of a program is slightly more involved. Of course I have to set up
the Manager needed for the HTTP calls, but I also need to
- ensure that the web service is in a well-known state before starting, and
- extract the state of the web service after running the program, so I can compare it to the model
runProgram :: Program -> IO Model runProgram (Prog cs) = do mgr <- newManager defaultManagerSettings resetReq <- parseRequest "POST http://localhost:3000/reset" httpNoBody resetReq mgr mapM_ (runCall mgr) cs model <- extractModel mgr return model
The call to POST /reset resets the web service. I would have liked to simply
restart the service completely, but I failed in automating it. I think I'll have
to take a closer look at the implementation of scotty to find a way.
Extracting the web service state and packaging it in a Model is a matter of
calling GET /users and then repeatedly calling GET /users/:id with each id
gotten from the first call
extractModel :: Manager -> IO Model extractModel mgr = do req <- parseRequest "http://localhost:3000/users" resp <- httpLbs req mgr let (Just ids) = decode (responseBody resp) :: Maybe [Int] users <- forM ids $ \ id -> do req <- parseRequest $ "http://localhost:3000/users/" ++ show id resp <- httpLbs req mgr let (Just (user:_)) = decode (responseBody resp) :: Maybe [User] return user return $ foldl (\ map (User id name) -> M.insert id name map) M.empty users
Generating programs
My approach to generating a program is based on the idea that given a certain
state there is only a limited number of possible calls that make sense. Given a
model m it makes sense to make one of the following calls:
- add a new user
- delete an existing user
- end the program
Based on this writing genProgram is rather straight forward
genProgram :: Gen Program genProgram = Prog <$> go M.empty where possibleAddUser _ = [AddUser <$> arbitrary] possibleDeleteUser m = map (return . DeleteUser) (M.keys m) possibleEndProgram _ = [return EndProgram] go m = do let possibles = possibleDeleteUser m ++ possibleAddUser m ++ possibleEndProgram m s <- oneof possibles let m' = simulateCall m s case s of EndProgram -> return [] _ -> (s:) <$> go m'
Armed with that the Arbitrary instance for Program can be implemented as1
instance Arbitrary Program where arbitrary = genProgram shrink p = []
The property of an API
The steps in the first section can be used as a recipe for writing the property
prop_progCorrectness :: Program -> Property prop_progCorrectness program = monadicIO $ do let simulatedModel = simulateProgram program runModel <- run $ runProgram program assert $ simulatedModel == runModel
What next?
There are some improvements that I'd like to make:
- Make the generation of
Programbetter in the sense that the programs become longer. I think this is important as I start tackling larger APIs. - Write an implementation of
shrinkforProgram. With longer programs it's of course more important to actually implementshrink.
I'd love to hear if others are using QuickCheck to test REST APIs in some way,
if anyone has suggestions for improvements, and of course ideas for how to
implement shrink in a nice way.
Footnotes:
Yes, I completely skip the issue of shrinking programs at this point. This is OK at this point though, because the generated =Programs=s do end up to be very short indeed.