Hacking on a Ledger

Ledger (ledger-cli.org) is a gem of a program for managing your finances. Dating from 2003 it embodies the UNIX philosophy of being highly minimalist and modular. In the form of a command-line tool it lets you run queries on your financial data, held in ledger files that have a simple, textual format. The format is so simple in fact you can easily write them by hand.

The elegant design of Ledger birthed a whole ecosystem and a new approach to personal accounting. For an excellent guide visit Plain Text Accounting (plaintextaccounting.org).

I’ve been using Ledger for a long time but one thing has always caused me a problem. In the UK our tax year starts on the 6th April. It’s useful to organise your financial records around this date, starting and ending each year on that day. Ledger however always considers months, weeks and years to start on the first day of the calendar period. You can change which day Ledger considers a week to start on, but this is not enough to cope with the UK’s tax year.

Why does the UK use such a strange date? Like many things in the UK we should probably blame/thank Henry VIII. For a deeper dive into that see Why the UK tax year begins on April 6 (theconversation.com).

I found some discussion of this problem in Ledger’s forums and repository but it looked like no one had ever gone as far as adding a feature. For a project that had made my life a lot easier I was happy to make a contribution and see if I could fix this myself.

Getting set

Ledger has some good help to get started on development for-the-impatient (github.com). Most usefully it has a big python script acprep that attempts to configure your machine for development. The only drawback was the uncertainty about exactly what it would do to my system. Would the system changes live happily alongside other projects I’m working on? I didn’t want to risk breaking anything so using a development container seemed the obvious choice to isolate any changes.

I am a Fedora user and toolbx (containertoolbx.org) is integrated so well that it’s always what I try first. First I create a new container just for ledger:

> toolbox create -c ledger
> toolbox enter ledger

Next I check out the ledger code and try the big script:

> git clone git@github.com:ledger/ledger.git
> cd ledger
> ./acprep dependencies

It completes without error so I try to build the project:

> ./acprep update
acprep: INFO: Invoking primary phase: update
acprep: INFO: Executing phase: update
acprep: INFO: Executing phase: pull
Already up to date.
acprep: INFO: Executing phase: make
acprep: INFO: Executing phase: config
acprep: INFO: Executing phase: configure
acprep: INFO: System type is => Linux
acprep: INFO: Setting up build flavor => debug
acprep: ERROR: Cannot find CMake, please check your PATH

CMake is missing. I guess this is probably the build tool for the project, but I’m a bit surprised it wasn’t installed by acprep dependencies. I install it via dnf and try the build again. This time I’m missing a C++ compiler. Taking a step back I realise I probably need to start by installing the basic tools. Perhaps the acprep script assumes the compiler etc are already present. So to try and save myself some time I use a dnf groupinstall to try and get a fully working development setup:

> sudo dnf groupinstall "Development Tools" "Development Libraries"

I try the build again - no difference! I am surprised again but perhaps “Development Tools” does not cover C++ tools. Let’s address that:

> sudo dnf install gcc-c++

Later I discover the dnf group “Development Tools” in fact contains things like git, subversion, patching, diffing tools etc but no actual compilers. You can see this with command dnf group info "Development Tools". My mistake. I thought it might contain such things to actually work on the Linux kernel and distro itself.

Anyway back to the build. We get further now but still no success:

> ./acprep update
...
CMake Error at CMakeLists.txt:233 (message):
  Ledger requires GMP version or greater, but only GMP version ..  was found

This looks odd. Some kind of complaint about GMP but the message seems to be missing a few details. First I try install to GMP: it takes a bit of hunting around for the right package name but it’s already installed anyway so it can’t be that.

Jumping into CMakeLists.txt at line 233 I can see its doing some kind of work to pull a version number from wherever it finds gmp.h. I find this file in my container and take a look at the contents. It doesn’t have any version information in it but it does have a very useful comment at the top:

 * This gmp.h is a wrapper include file for the original gmp.h, which has been
 * renamed to gmp-<arch>.h...

Eventually I find that I have a file gmp-x86_64.h and I think this is what ledger actually wants to use. A quick hack in CMakeLists.txt and we finally have a successful build!

This problem is very specific to developing with Linux I think. Small changes between how things are packaged on different distros can cause these little road bumps where you might have to hunt around for the right package name or file. It’s probably best to try and use the distro that the authors of Ledger use themselves. In the README.md Ubuntu, Debian and Fedora are all mentioned though so it’s not exactly clear what is the best choice.

As I said earlier I also prefer toolbx for development containers but toolbx defaults to using your host distro. More support for other distros is coming: Fedora Workstation 39 and beyond (gnome.org). I might also explore Distrobox (distrobox.it) in future which I think is more flexible, but you have to do a little more configuration.

Anyway we’ve got Ledger building and the acprep script they provide worked fine once we got the right dependencies installed. After a bit of digging I also find the command to run the tests: ctest . - they all pass. So we’re in a nice place to start poking at the code.

Poking the bear

I have an idea of where I might need to make my changes (times.cc) but initially I just want to run Ledger and get some debug output to see what’s going on.

For compiled languages you of course need to ensure you have run a build before any changes can take effect. Coming recently from a python project the extra delay that compilation causes feels like a real drag on progress, adding minutes to each investigative change I make.

I discover that setting the environment variable LEDGER_DEBUG enables debug output from Ledger which is very flexible. In fact I later find that the debug options are very nicely documented in the Ledger manual: debug options (ledger-cli.org). In addition Ledger already has many useful debug messages included so you can learn a lot just by turning them on.

I add a lot of my own debug print messages but I’m struggling to really get a handle on what’s happening by the time execution reaches the point of calculating time periods. There’s some funky chained function calls/handlers/filters that get setup early and then executed in order to produce reports and it’s not yet clear to me how it all hangs together. See for example the code in chain.cc (github.com).

I realise I need to see the call stack whilst ledger is running to understand what’s happening but have been wary of messing with a debugger. The last time I used one was from within an IDE. It was very easy but I’m using vim and have never tried debugging with it. However there’s no other option if I’m going to make progress.

Before we can debug we need to build the binary with debug symbols. This is as simple as adding a flag to the compiler’s options but I end up hacking this into the build process as I cannot see how to enable it easily through the acprep script I’m using to build. There must be a simpler way I’ve missed. I later discover this is documented in CONTRIBUTING.md (github.com). If only I had the patience to reads the docs first!

So now I’ve got a debug build with symbols (which adds a bit more time to the build unfortunately). With some trepidation I install gdb and open its manual expecting to wrestle with it for some time. Happily I find gdb is actually very approachable and has an excellent manual. I simply prepend my desired ledger command with gdb and I get a command line interface from which I can set breakpoints, step, continue etc with very obvious commands. In no time at all I find I can print the backtrace I wish to see and walk up and down it examining the variables in each frame.

Ledger makes use of the boost library which I understand is pretty standard in C++. I see a lot of boost in the backtraces but largely I can just ignore it and step into the source code before and after it.

The only confusion I suffer with gdb is around some mysterious SIGTTOU signals which keep stopping execution. It initially misleads me into thinking some Ledger code is being called twice (it isn’t) as gdb stops before and after the signals are received on my breakpoints it seems.

It turns out this is a common issue when debugging a process that writes to the terminal. Gdb also writes to the same terminal causing the process under observation to be signalled. The way around this is to remotely debug the process from another terminal. It sounds complicated but in fact you just need to use gdbserver instead of gdb and open another terminal window to connect from. There’s a good summary of the process in the article Debugging binaries invoked from scripts with GDB (redhat.com).

It all feels like hard work. Did I mention that I haven’t touched any C++ code in over two decades?

Making the change

I was finally in a position to be able to make changes, debug and run tests - the full dev cycle. Throughout this I made repeated use of git commit and git stash to make small steps forward/backward as I stumbled towards a better understanding of what was going on.

Most of the time was spent understanding how Ledger sets up report handlers in one context and then runs them later in another with populated data. I repeatedly had to jump into the debugger to check my understanding and print out backtraces. A visual debugger which let me jump forward and backward in execution time would have been useful. I believe gdb can let you move forward and backward in time but when I tried it said it was not supported with the Ledger binary.

I made heavy use of ripgrep to find code references and definitions but ideally my editor (vim) should be setup to do this. I’ve used language servers with other codebases before via the Language Server Protocol (github.io). I need to look into this for C++ though.

Another point of difficulty for me was the use of macros in the source code. Their presence makes debugging just a bit more tricky as the debugger could not map from macro code back to the source files. There are ways around this e.g. expanded C macros (stackoverflow.com) but I could not get anything to work. Probably my inexperience here. I did read a stack overflow answer suggesting macros are best avoided these days but I don’t know how popular that opinion is in the C++ world.

Eventually I worked out the right changes to make and where to put them. I added some tests for my changes and got them passing. Running the full test suite I discovered I’d broken the forecasting feature somehow. A little exception was needed in the code for that and to save some time I learned how to run individual tests with ctest. It seems a little fiddly compared to other test runners - having to specify a start test number, a stop test number, a step, etc: ctest -I (cmake.org).

By this point my source tree is polluted with a lot of build artifacts, test files, temporary files and others bits. Git status gives me two pages of untracked files which makes seeing my real changes difficult. Maybe I’m doing this wrong as I can’t believe people would put up with that. I prefer to have my build outside of the source tree for exactly this reason and to keep a clean separation. There’s probably some way to tell cmake to do that but I don’t know it yet.

Just one failing test remains - a test that runs all the code in Ledger’s documentation to make sure all the examples are correct (which is great). However I haven’t changed anything in there. I do a quick git stash, jump back to master to check the tests were ok before all my changes. They are. I unstash again and run the tests again. They all pass now, weird. Never mind then!

Flagging it up

To finish I add documentation on my changes into the Ledger manual. As I’m writing I realise how weird my change is - it only works on periods of months and years, not weeks. I avoided changing weeks as it made a bunch of existing tests break. Hold on, that doesn’t feel right.

The next day while doing something totally different, the idea to put my changes behind a command line flag pops into my head. If I do that then I can have my change work for all periods and not break existing users and tests. Seems obvious now!

I add a flag to the command line and now I can also put all my tests into their own file, much cleaner. But ctest doesn’t pick up my new test file. Turns out that requires a full clean according to a chat from 2014(!): new tests not being run (groups.google.com). Weird.

All that’s left is just some final tidy-ups to the code to make sure variables have sensible names. I realise I should probably rename the whole feature to ‘align intervals’ as I am starting to understand the jargon that’s used in the manual around time periods. It makes sense. I do a bit more casual testing with various weird intervals but it all seems to work fine. Ok we’re nearly there.

Reaching out

How to approach a project when you are ready to put forward a change? Not all projects make it clear how they like to receive suggestions. The docs suggest raising a bug. I feel this is the best approach, particularly as the issue has been discussed previously and code changes were welcomed at the time. I could have gone to the forums to discuss it first but feel that as I am ready to suggest a code change the repository is best.

I’ve seen projects cheerfully welcome PRs in chat but when you do submit something it gets overlooked. There’s a lot written about maintainer burnout and a PR adds to the workload of a busy team, so I don’t think you should ever expect a reply.

Anyway I get my branch into shape with some git cherry-picking, pulling out the bits that are personal to my setup and push it all up. I create a bug (github.com) and then a pull request (github.com) to resolve it.

Final thoughts

The delays in working with a compiled language are significant. Having just come from a Python project where the change/test/investigate cycle was very fast the difference was stark. I had to be strategic about when to kick off a compile as it took several minutes. The lack of a REPL also stopped me trying things out quickly.

It makes me wonder if there’s a JIT for C++, even just for use during the dev/test cycle to speed things up. I had a quick look and it seems there is a thing called CLing: JIT compiler for C, C++, and the likes (stackexchange.com). I might try that next time.

I had apprehensions at the start, I’ve not touched C++ in decades. I thought it would be a relatively simple change - in the end it was - but the process to get there was not smooth. However I learned a lot and grew some confidence. Particularly when I got gdb working and could really see what was going on inside Ledger.

I hope my change is useful to the project, even if it’s not accepted in its current form but here’s hoping.