Skip to main content

Automatically Deployed ABI Migrations

· 2 min read
Christopher J. 'CJ' Wright
Christopher J. 'CJ' Wright
Member of conda-forge/core

Handling application binary interface (ABI) migrations has always been a hassle for Conda-Forge. Maintaining ABI consistency helps enable the "just use conda-forge" experience for many of our users, making certain that numpy's blas is the same as scipy's. As libraries update their code, the new versions may be ABI incompatible, as function signatures and other symbols may have changed, leading to the dreaded SegmentationFault and other errors.

Conda-Forge handles this by having a pinning file that tracks all the currently supported ABIs. These pinned ABIs are then used to build the downstream packages, making certain that all are consistent. As new versions of pinned software are released the pins are updated, causing a migration of the pin, and the rebuilding of all packages which rely on the pinned package. In the past, this was handled by a change to the global pinnings and a subsequent migration via the auto-tick bot. While this worked, there were issues that this created. Firstly, this approach could cause unsatisfiable build dependencies for new packages, as some of the new package's dependencies had been compiled with the new pins, but not all. Secondly, migrations happened in series, if a second pin was moved while the first was being migrated then the migration could go wrong as packages which were being rebuilt for the first pin got the second pin before they were ready.

Conda-Forge Core has recently approved CFEP-9, a migration policy to fix these issues. With CFEP-9 pinnings are proposed as small yaml snippets which contain the new pins. The auto-tick bot then starts migrating the packages in order, applying the yaml snippet to each package in turn. If a second pinning change is issued then the bot starts the migration for that package too, enabling the two migrations to work independently. If a package needs a change in both pins then the maintainers can choose the order in which they apply the pins by merging one pin before the other.

This approach will yield much greater stability in migrations and will enable more maintainers to issue migrations. Migrations can be issued by putting a PR into conda-forge/conda-forge-pinning-feedstock, adding a file to the migrations folder, PRs into the auto-tick bot are not needed anymore.