Before I really get started on what bothered me about flakes, I would like to very clearly point out that this is my opinion, and also that I am a big dumb-ass.
I use my computer for lots of different creative and professional (sometimes both) endeavour, and while I like Nix in a lot of ways (I have even really come to love the syntax of the language), having "managing my computer" as a hobby is not one of them.
I deliberated whether I should write and publish this blog post, but decided that I still wanted to share my experience and frustrations. Maybe the way that I have come to use Nix on my computers is not in line with "how you're supposed to do it", but maybe that's exactly the reason why I should write something about this topic.
Anyway...
What are flakes
Back in 2019 RFC 0049 was proposed, which introduced the Nix community to the new concept of "flakes", a way to manage Nix build inputs and outputs in a pure and reproducible manner. This made a lot of people very angry and after months of reasonable online discourse, the RFC was withdrawn and implementation work began in the new Nix CLIs behind an "experimental" feature flag. This was done in order to work on the design of flakes, and being able to explore problems that came up more organically. I can see however how people might feel that the design was rejected, and was then snuck in the backdoor by the RFC authors. In the meantime a number of smaller RFCs have come up by community members as a way to voice feature requests for flakes.
Responses to flakes are still varied to this day. Some are really
into the features they enable your builds to have, and I have to admit
that being able to have a single flake.nix file in a project
repository, which pins a version of nixpkgs and produces a build of
the given application is very nice.
Disdain for even the idea of flakes can still be found in some corners of the Internet however, and I think everyone involved in the development of flakes is rightfully on-edge, given the amount of hostility and abuse that they have been getting for checks notes implementing a new feature in a package manager.
I am aware of this history and I don't want this blog post to become a weapon wielded by jackasses to harass a bunch of people online. Don't be a dick and go look at some ducks instead.
What I did before
When I got started using NixOS I began by maintaining a nixpkgs subtree. That's what all my friends were doing, and it seemed like a neat idea at the time. I was occasionally working on upstream nixpkgs, so having all the code right there seemed like a sensible idea. In hindsight, it really wasn't, and it turned into a frequent source of frustration (see the disclaimer above about how I don't want "managing my computer" to be a hobby of mine).
I stuck with this system for close to 4 years anyway (see the disclaimer about being a dumb-ass), because the cost of migrating always seemed to high. I knew I wanted to try out flakes, but the barrier to entry also seemed too high. The fact that the flakes RFC was withdrawn and development continued behind an experimental feature flag signalled, at least to me, a state of flux and uncertainty about the design of flakes. The fact that the only documentation available was one short page on the (generally excellent, please contribute more to it!) NixOS wiki, and a blog post series by tweag didn't help this.
The two big problems with the nixpkgs-subtree approach are: nixpkgs is huge and is going to make your computer unbearably slow, and unmanageable merge conflicts in case I ever updated a computer and didn't immediately push the update to the repository remote.
And while it would seem like something that is easy to avoid, I ran into this problem again and again and again, to the point where by the end my servers and desktops were running completely disjointed configuration repositories, that only occasionally were updated by sending patches around. Fucking yikes. Yes, this is my fault, but the system I had designed myself into encouraged these sorts of human errors to accumulate.
Finally, late last year (2022) I started a job where I came into much closer contact with flakes, and so, I pulled off the several-year-old band-aid and dove right into it.
Migrating to flakes
This was the state that I inherited and moved over to
flakes. It took me a few weeks to get things building again on my
NixOS desktop and Pop_OS (with home-manager) laptop. Right off the
bat I didn't like a few things. I was using the NIX_PATH as a path
lookup for modules in my configuration, so I didn't have to rely on
relative paths between modules (I would frequently expand
<configuration/foo> and similar lookups). This isn't allowed with
flakes, unless you opt into impure evaluation mode, which is what I
did. In hindsight, the way that my flakes configuration ended up
looking was much more linear than the kludge of horror I had built
before, and so this limitation was less of an issue, although it
still bothered me in some places.
I also really wasn't, and still am not a fan of the build syntax:
- Before:
nix-build '<nixpkgs/nixos>' system -I nixos-config="$HOST" - After:
nix build .#nixosConfiguration."$HOST".config.system.build.toplevel
This is the commit that ported my desktop over to
flakes, if you wanna have a look. In my build harness I removed a
bunch of features like being able to quickly build vm and iso
targets. This might still be possible with flakes, but I didn't find
an obvious way to do things in the moment, and so I opted for removing
the features in the meantime.
The NixOS documentation (and SEO!) situation is bad enough, and restricting yourself to results that talk about a subset of the language and ecosystem made it unbearable in some cases to understand what I was supposed to do. I ended up reading several people's configurations to copy things from them.
I ended up with this custom harness function to create a NixOS system
because I hate code duplication (and a similar one for the
homeManagerConfiguration function):
nixosSystem = root: nixpkgs.lib.nixosSystem (root // {
inherit system;
modules = root.modules ++ [
({ ... }: {
nix.nixPath = [
"nixpkgs=${nixpkgs.outPath}"
"home-manager=${home-manager.outPath}"
];
nixpkgs.overlays = defaultOverlays;
})
## Include the home-manager NixOS module
home-manager.nixosModules.home-manager
];
});
It was important to me to have the nixpkgs and home-manager keys
set correctly in my NIX_PATH for the "legacy" CLIs to be able to
pick them up and use. Nonetheless, this caused countless issues that
I still don't fully understand. But more on that in a little bit.
Another rift I saw in the community of flakes users was whether or not
to use flake-utils. The two philosophies seem to be:
- "use flake-utils because it makes your life easier"
- "don't use flake-utils because it makes flakes seem too complex"
Which one is it? Are they both true? I ended up not including another dependency for my configuration, instead writing the wrapper functions myself (as I said, I really quite like writing Nix code!).
Especially because all my computers are x86_64-linux, and
flake-utils is often used to work around some issues with flakes
regarding cross compilation. Depending on what systems you use with
Nix, flake-utils may give you a better bang for your buck because you
need to write less obscure loop code that includes your configuration
root for multiple system targets.
I imagine this will confuse potential adopters of flakes similarly to how it confused me, and it certainly didn't leave me feeling rosy about this technology I had just built my sand castle atop of. If you're interested in the custom wrapper code I wrote: here are my main utilities and user utilities!
Using flakes
And so, that was my life for about 6 months. I would occasionally run
nix flake update, which was easy to complete, almost never caused
any merge conflicts due to my dumbassery (and even the one or two
times it did, was easy enough to fix by just deleting the lockfile),
and life was good.
I also started using flakes for some one-off projects, where I was collaborating with other people (for example jackctl, which you should check out if you make music on Linux), where using flakes made it easy to depend on multiple sources of nix code, and build a single package with a particular postFixup phase or whatever. I will still use flakes for these use-cases, because I think they're the ideal situation to map a simple set of inputs to a simple set of outputs.
Importantly, this is not my development environment, but rather something that would be invoked in CI.
But, I slowly noticed that the development environments littered
around my system stopped getting current rustc packages. The way I
build those is with shell.nix files, which include <nixpkgs> from
my path, which are built by the lorri service. The idea of having
different environments, that all pull from the same local package set
is just to reduce the space requirements on my computer. I have a big
SSD, but might as well not pull in 50 slightly different versions of
rustc, cargo, etc.
At first I didn't really care, but with a lot of Rust projects
recently dropping down their supported Rust compiler period, I started
running into compilation issues. I even briefly installed
rustup on my laptop because I was blocked from working on a
project and it was the easiest way to get out of my
pickle. shudders.
I knew something had to change. I couldn't go back to the way things
were before, and the way flakes made me re-evaluate my configuration
was good. At the same time, my computer started accumulating
strange behaviours I didn't really understand. And maybe it was a
matter of not using the new CLIs exclusively, or wanting to rely on
lorri to asynchronously build environments for me and injecting them
with direnv (something that works very well with emacs too).
The last straw was a build failure as a result of an incompatibility between home-manager and nixpkgs, that had allegedly been fixed, but updating both my nixpkgs and home-manager inputs over the span of 2 weeks didn't allow me to update my computer. Maybe I got extremely unlucky with the unstable channel progressing, but in the end it didn't really matter. I wanted to make a change.
What now
I had previously evaluated different strategies for managing my
nixpkgs and home-manager dependencies. I briefly thought about
subtrees, where I squash committed everything but decided that I didn't
really get any benefit out of that arrangement. I tried a
submodule, which was somehow even worse, and then I tried [niv] and
flakes. In the end, flakes briefly won out because I wanted to try
them. So moving to niv was the obvious choice.
niv was born out of some of the ideas of the initial flakes RFC,
taking basically only its input management, encoded in a json
configuration, which gets parsed by some magic nix code (which I have
to admit, I do not fully understand, nor have I had to look at it
because something was broken or unclear as to how it worked).
My build harness script looks like this now (the relevant bit):
NIXPKGS_ALLOW_UNFREE=1 \
home-manager build -f "$ROOT" \
-I "nixpkgs=$DIR" \
-I "klib=$DIR/lib" \
-I "nixpkgs-overlays=$DIR/overlays" \
-I "home-manager=$($DIR/lib/find-component.sh home-manager)" \
"$@"
and the way I import nixpkgs looks like this:
{ overlays ? [], ... } @ args:
let
sources = import nix/sources.nix;
in
import sources.nixpkgs (args // {
overlays = (import lib/overlays.nix) ++ overlays;
})
I had to write a small utility for setting the appropriate NIX_PATH for home-manager:
# De-reference current directory and grab the component parameter
# This will break if none was provided so... don't do that :)
DIR=$(realpath "$(dirname "$0")/..")
COMPONENT=$1
# Run nix eval and just return the result
nix eval --expr "(import $DIR/nix/sources.nix).${COMPONENT}.outPath" --impure | tr -d '"'
The nice thing about grabbing the component paths from the niv sources
is that the NIX_PATH environment variable gets populated with
absolute /nix/store paths. This means that changing something in my
configuration won't apply to things on my system until I next build
and switch to a new generation:
❤ (tempest) ~/sys> echo $NIX_PATH
nixpkgs=/nix/store/7znqbsc1vqyjhcmdvfjrfp73hhvazpwq-nixfiles
home-manager=/nix/store/0haa6si3qmp8r80irj4a692qvxp51rfh-zb3bip08c8w6jxbhbmb4i852lp4a3d95-home-manager-src
nixpkgs-overlays=/nix/store/q015fg370v3qjwrz840pl5f38km4j5sh-overlays
In conclusion
In the end... is this better? I really don't know.
Flakes felt like magic in a lot of places, even though their
functionality should be able to be encoded into a simple input ->
output relationship. While using flakes I didn't feel like I
understood what my computer was doing. And while my current solution
technically allowed me to reduce the amount of code needed for my
system to build, my solution is bespoke, and I can understand that
someone who hasn't thought about their Nix setup as long and hard as I
have, might feel overwhelmed by it. This is especially a problem if
everyone comes up with slightly different variations on how they would
like to initialise their nixpkgs builds. And maybe I should just have
globally installed rustc and cargo and called it a day.
Shit... maybe managing my computer is my hobby.
I can certainly see how flakes can work for many, many people, and I am probably in the minority of users who will bump into problems.
At the same time, I'm not a fan of the way flakes have been developed.
There seem to be a sentiment that flakes are "done", even though they've technically never left the experimental stage. I do not condone the abuse directed towards the creators of this technology! But at the same time, I can see why some people might feel annoyed at the idea that an RFC was de facto rejected (withdrawing an RFC is not generally done because its initial design was perfect), and then having its feature set snuck into the Nix codebase without going through the same RFC process again. At this point, enough users depend on the experimental flakes feature as it exists, it could be argued that any breaking change would be too disruptive, and so the current state must be stabilised as is.
I have also seen and felt the pressure of coming up with perfectly measured criticism against flakes (one of the reasons, why I was unsure about publishing this blog post), because of the way conversations about them often divolve into abusive tirades. No it's not OK for people to be behaving the way they have been. But it's also not OK for the flake developers to dismiss people's criticism, concearns, problems on the basis of some people being dicks.
Since 2019 the documentation situation for flakes has only improved
marginally. There are a few more resources out there now, but none
are "official" or built by the community, and also none that go beyond
the basics of what a flake is and does (yes nix shell is cool, but
nix-shell existed before too). There is a kind of chicken & egg
problem where documentation must refer to things that are stable, and
flakes need documentation to get more people into the headspace to try
to use them.
Ultimately, I'm open to the idea of using flakes again. Updating my horrible hacks from before flakes to flakes was hard, changing from flakes to niv was two afternoons. So maybe I will try them again at some point. But for the time being, I decided it's more trouble to me than it's worth.
Thank you for reading this rambly post until the end. Maybe you learned something, or were able to contextualise some of your own experiences or problems. Please don't be a dick. In the end, it doesn't really matter. Computers don't matter, and you shouldn't get too upset about them.
Cheers.