You can write nicer nix code

Don’t get me wrong. You can’t write nice nix code, because Nix language is ugly. In my opinion, of course. But sure you can do better.

Here are some suggestions:

• Make nesting match the scope.

• Name things and name them properly.

• Let things in let ... in ... follow order.

• Only use anonymous function when it fits a single line.

• Either fit a function call with all the arguments at a single line, or put each argument at new line with the same indent as called function.

We use Nix at work and so I need to understand it. When I am trying to find out what happens in our flake.nix, I do refactor the code. Some of the source code of a template we use is published so I believe it’s ok to share my findings about that part of code.

For the purpose of this post, the goal of our flake.nix is to buildPythonPackage and provide devShells with the dependencies. We use data from pyproject.toml, stored in the pyproject variable:

pyproject = nixpkgs.lib.trivial.importTOML ./pyproject.toml;

Now, the main problem our flake solves – we take Python package names from the pyproject.toml, but these names correspond to the Python package names in PyPI and it may happen that the same Python package has different name in Nix. Like SQLAlchemy.

I will provide only interesting parts of our flake.nix.

pypi2nix = list: pypkgs:
  attrValues (getAttrs (map (n: let
      pyname = head (match "([^ =<>;~]*).*" n);
      pymap = {
        "SQLAlchemy" = "sqlalchemy";
      };
    in
      pymap."${pyname}" or pyname)
    list)
    pypkgs);

requires = pypi2nix pyproject.project.dependencies;

buildPythonPackage {
  dependencies = requires pythonPackages;
  # ...

devShells.default = pkgs.mkShell {
  packages = with pkgs; [
    ruff
    (python3.withPackages (p:
      [p.build p.mypy]
      ++ foldl (prev: f: prev ++ f p) [] [
        requires
      ]))
    # ...

I like it so much. Let’s understand what is going on here, starting with

pypi2nix = list: pypkgs:
  attrValues (getAttrs (map (n: let
      pyname = head (match "([^ =<>;~]*).*" n);
      pymap = {
        "SQLAlchemy" = "sqlalchemy";
      };
    in
      pymap."${pyname}" or pyname)
    list)
    pypkgs);

The first line is

pypi2nix = list: pypkgs:

saying that pypi2nix is a function of two arguments, list and pypkgs.

  attrValues (getAttrs (map (n: let

attrValues takes attribute set and returns the list of values – it drops the attribute names. Its argument is attribute set (getAttrs ...).

getAttrs takes a list of strings of attribute names as the first argument (map ...) and some attribute set as the second argument pypkgs. It returns the attribute set containing attributes (names and corresponding values) from the second argument pypkgs with the attribute names from the first argument (map ...).

map is higher-order function. It takes a function (n: ...) as the first argument and a list list as the second argument. The map generates new list by applying function in the first argument to each element of the list in the second argument.

n: let ... in ... is a function. In let ... it prepares variables it uses and in in ... there is the implementation. let ... contains:

•       pyname = head (match "([^ =<>;~]*).*" n);

  This piece of code takes first characters of n (the function’s only argument) that are not =<>;~ and stores them in pyname variable. It converts SQLAlchemy>=2.0 to SQLAlchemy.

•       pymap = {
          "SQLAlchemy" = "sqlalchemy";
        };

  pymap is mapping of Python package names used in PyPI to names used in Nix.

The function implementation, i.e., what function returns, is in in ...:

•       pymap."${pyname}" or pyname

  If pyname is in pymap, return the corresponding value. If not, return pyname.

So far so good.

requires = pypi2nix pyproject.project.dependencies;

The interesting part about requires is that pypi2nix is partially applied here: pyproject.project.dependencies is a list of dependencies from the pyproject.toml and pypi2nix takes a list list as the first argument. However, pypi2nix takes two arguments in total, but there is no second argument!

The result of the partial application, requires, is therefore function that expects a single argument. That single argument corresponds to the second argument of the pypi2nix function, pypkgs.

buildPythonPackage {
  dependencies = requires pythonPackages;
  # ...

Here, we build the python package. pythonPackages are Python packages from Nix. In reality, this code looks slightly different, but that’s not interesting now.

Interesting is that dependencies are now the result of requires applied to the pythonPackages. This is the same as:

buildPythonPackage {
  dependencies = pypi2nix pyproject.project.dependencies pythonPackages;
  # ...

Finally, the development shell.

devShells.default = pkgs.mkShell {
  packages = with pkgs; [
    ruff
    (python3.withPackages (p:
      [p.build p.mypy]
      ++ foldl (prev: f: prev ++ f p) [] [
        requires
      ]))
    # ...

From the first line:

devShells.default = pkgs.mkShell {

Create (pkgs.mkShell) new default development shell. The default development shell is run by nix develop. pkgs here is a variable with nixpkgs.legacyPackages for some system.

  packages = with pkgs; [
    ruff

In the development shell, the ruff package will be available. That with pkgs; is just a shortcut. The following has the same meaning:

  packages = [
    pkgs.ruff

We use with pkgs; because we usually need more than a single package.

The last part lets us include Python dependencies in the development shell.

    (python3.withPackages (p:

python3.withPackages is a higher-order function. It takes a function to which it provides the attribute set with Python packages. It returns the list of packages, here to be included in the development shell.

(p: ...) is the function. p here is the attribute set with Python packages provided by the python3.withPackages.

      [p.build p.mypy]

This is the first part of the list of packages to be returned by the (p: ...) and python3.withPackages function, respectively.

      ++ foldl (prev: f: prev ++ f p) [] [
        requires
      ]))
    # ...

This piece of code was the most challenging, to be honest.

• ++ is the concatenation of the lists, so what follows ++ needs also be a list.

• foldl is a higher-order function that takes three arguments, here foldl () [] [].

• The ending )) are to close function (p: ...) and function (python3.withPackages ...).

The first argument to foldl is a function, the second argument is the initial value of an aggregator, and the third argument is a list.

Generally, foldl goes over each element of the list, applying the function to the aggregator and the list’s element, storing the result of the function application to the aggregator. When there is no element left in the list, foldl returns the aggregator.

Our function is

(prev: f: prev ++ f p)

the initial value of the aggregator, here called prev, is

[]

and the list is

[ requires ]

We have only the single element in the list (f in our function), so there is just a single step and therefore foldl returns

[] ++ (requires p)

• Recall that p is the attribute set with Python packages passed by the python3.withPackages.

• ( ) are here due to the precedence – functions can be list’s elements, too! However, we need the result of the function call.

• Yes, our list contains functions we apply to the attribute set with Python packages to generate the list of Python packages to be included in the development shell.

Hypothetically, we could have requires-doc, the result of partial application of the pypi2nix to the list of documentation dependencies, and our list could look like

[ requires requires-doc ]

In such a case, the result of the foldl would be

[] ++ (requires p) ++ (requires-doc p)

It’s nice to find out how things work. Here is the refactored version:

# Return a list of derivations
# - wanted-pkgs is a list of strings of wanted Python package names
# - available-pkgs is attribute set of all Python packages available in Nix
pypi2nix = wanted-pkgs: available-pkgs: let
  pypi-name-to-nix-name = {
    # "pypi-name" = "nix-name";
    "SQLAlchemy" = "sqlalchemy";
  };
  to-nix-name = pypi-name: let
    pkg-name = head (match "([^ =<>;~]*).*" pypi-name);
  in
    pypi-name-to-nix-name.${pkg-name} or pkg-name;
  nix-names = map to-nix-name wanted-pkgs;
in
  attrValues (getAttrs nix-names available-pkgs);

requires = pypi2nix pyproject.project.dependencies;

buildPythonPackage {
  dependencies = requires pythonPackages;
  # ...

let
  wanted-pkgs = wanted-list: wanted-fns: available-pkgs:
    foldl
    (wanted-pkgs: wanted-from: wanted-pkgs ++ wanted-from available-pkgs)
    (attrValues (getAttrs wanted-list available-pkgs))
    wanted-fns;
in
  devShells.default =
    pkgs.mkShell
    {
      packages = with pkgs; [
        ruff
        (python3.withPackages
          (wanted-pkgs
            [ "build" "mypy" ]
            [ requires ]))
        # ...

Feel free to compare to the original code:

pypi2nix = list: pypkgs:
  attrValues (getAttrs (map (n: let
      pyname = head (match "([^ =<>;~]*).*" n);
      pymap = {
        "SQLAlchemy" = "sqlalchemy";
      };
    in
      pymap."${pyname}" or pyname)
    list)
    pypkgs);

requires = pypi2nix pyproject.project.dependencies;

buildPythonPackage {
  dependencies = requires pythonPackages;
  # ...

devShells.default = pkgs.mkShell {
  packages = with pkgs; [
    ruff
    (python3.withPackages (p:
      [p.build p.mypy]
      ++ foldl (prev: f: prev ++ f p) [] [
        requires
      ]))
    # ...