In a previous post we outlined the latest workflow for creating Conan packages for third-party open-source libraries based on some recent changes at that time. Unfortunately, there were some problems with that workflow (and post), which are now rectified with today’s release of Conan 1.1. Here’s a primer on the latest process.
Biggest Change - Two Separate Workflows
After further discussion with the Conan team we realized there were two distinct packaging scenarios the Conan client was trying to accomodate: “in-source” and “out-of-source”. The Bincrafters packages are all built “out-of-source”, meaning the
conanfile.py is in a separate github repository from the library sources, but one of the long-term goals of Conan is to foster the growing number of packages where the library maintainers manage the
conanfile.py in their project (“in-source”).
The previous updates were focused on “in-source” packaging workflows but it seemed it should be suitable for both scenarios. It did work, but it wasn’t optimized, and came with a lot of unfortunate side-effects. We brought this feedback to the Conan team and they aggressively sought various ways to accomodate our scenario and optimize our workflow. This brought about the recognition that the workflows are just fundamentally different, and some insight about possible solutions. The good news is that they were able to dramatically improve the original “out-of-source” workflow with very little modification to Conan.
New Out-Of-Source Packaing Workflow
So, the new workflow brings back the intuitive linear workflow that we had prior to the November 2017 update. After users have a
conanfile.py that looks like it’s ready to go, they can just jump right back to
conan create as in:
$ conan create . bincrafters/testing
Most often, the
source() method is pretty easy to debug and get right, and the first significant problems for debugging start occurring in the
build() method. So once we see that the
source() method is getting the sources, we can focus on the
build() method. Doing so is as simple as adding
-k to our command (short for
$ conan create . bincrafters/testing -k
-k option existed before, was very useful then, and continues to be useful now. With this, we no longer need to use the
conan build command for our workflow (although again, it’s still great for the in-source packaging workflow).
So, after some amount of trial-and-error with the
build() method, the package should eventually build the library properly. The next phase then becomes ensuring that the
package() method is gathering all the proper files from the
build directory, and copying them to the
package directory. This is often harder that it seems, and often requires trial and error. For this trial and error, we just want to re-run the
package() step multiple times, without re-running the
build() step. And thus, this is exactly what the latest feature from Conan 1.1 enables. We can now simply add
-kb to our command (short for
$ conan create . bincrafters/testing -kb
-kb flag also implies
-k, effectively causing conan to fast-forward to
package() method and re-run it with whatever
build directory has already been generated. This was really the missing link for our “out-of-source” workflows before.
The final step of the
conan create command which often needs to be run many times until successful is the
test_package step. In this case, the best approach is to take advantage of the separate and dedicated
conan test command:
$ conan test test_package bincrafters/testing
We discussed another
--keep flag like
-kp, but this was decided to be a bit too much trouble when the
conan test method was perfectly reasonable.
Test Package Build Folder Cleanup
Also regarding test_package, we got some great new features in Conan 1.1. One of the lingering annoyances for our team was that building packages always left behind the
test_package/build directory in each package folder. This directory has been almost completely useless the vast majority of the time (after the test_package is run), and so Conan has now given us a few flexible ways to clean this directory up automatically.
CONAN_TEMP_TEST_FOLDER- This is an environment variable that can be set to
Trueto tell Conan to automatically cleanup the
test_package/builddirectory automatically. You can pass this variable to Conan via the CLI when you call
conan testby adding
-e CONAN_TEMP_TEST_FOLDER=True, but for most use cases, you probably want to set it in your default Conan Profile.
temp_test_folder- This is a value in
conan.confthat you can set to have the same effect as the environment variable above. The difference is that setting it in
conan.confis global, and will affect all recipes and profiles.
--test-build-folder- This is a flag to the
conan createcommands which allows you to specify a different
builddirectory for the
test_package. This doesn’t exactly help “automatically clean up” the directory, but it does allow you to choose to always put the
builddirectory into some well-known temp directory rather than a subdirectory of your project folder. This is a handy piece of flexibility.
Again, it’s important to point out that the recommendations here are for optimizing “out-of-source” builds, and that the previous post is still largely relevant for “in-source” builds. However, for “out-of-source” builds which is still the majority of packages we see, this workflow brings a number of advantages.
- It’s dramatically simpler, effectively only requiring knowledge of two Conan commands and two flags.
- It’s inherently idempotent, users don’t have to manually create, specify, or delete any directories or files manually.
We really hope other OSS packagers test this workflow for themselves and let us know their thoughts. The Conan team is really amazing when it comes to listening to feedback, and we try to help encourage the feedback loop from the community for that reason.