Updating Espressif Development Framework (esp-idf) and Switching to CMake
A How-To Guide
First a bit of a background, but if you want to jump straight to the steps, click here.
Introduction
In the middle of 2022, for a client project, we’ve been working with esp32-wroom-32d and esp-idf v3.3.4. The project was firmware for a communication module with FOTA update capability, as well as WiFi and BLE communication and control options. We had already planned to use the new version of esp-idf for similar projects that were in the development pipeline, but since the project in question was finally reaching the completion stage and the release date was coming closer, changing the framework was too risky for the client.
The plans changed when we encountered a critical issue which was preventing the use of some of the important features, and degraded user experience.
Days of debugging and searching for hints online showed that many people faced the same or similar issues, but none of the few solutions that were mentioned helped with the problem we’ve been facing. What we could see is that esp-idf had a lot of new changes and fixes, and esp-idf v3 was already EOL, so we figured updating the framework was the only sensible step we could take at that point.
Since the esp-idf v4.4 still has support for legacy code, updating the esp-idf itself wasn’t too complicated; remove the existing esp-idf directory, paste the newly downloaded one, fix a few compilation bugs caused by -Werror flag, et voilà! But we wanted to update our project to be in line with the development guidelines of esp-idf projects and make it ready for future versions. Two steps that were needed to reach this goal are updating the build system to use CMake and ninja instead of GNU make and removing the legacy code and changing to the new. Changing the legacy code was too big of a change for our situation, so we started with the build system overhaul.
Following are the steps we needed to take to make everything work.
How-To Guide
This guide focuses on steps to update the esp-idf to v4.4.1 (latest stable as of time of writing) and CMake+ninja build system.
The official ESP-IDF Programming Guide was immensely helpful, but it still had some shortcomings, especially for a project like ours that has a bit unusual directory structure (compared to what esp-idf expects). Reference it when you need more information.
This is what our project roughly looked like before the changes:
The unusual thing here is that we have one component, and its files are not divided into src/inc files but are split in sub-directories depending on the functionality those files cover inside the project. The directories repo_root/commod_project/build and repo_root/toolchain are not actually tracked in the repository, they are created and updated with each build (build) or with the first build (toolchain) and stored in the local repo directory in order to set all the tools locally and not pollute the environment.
⚠ If you’re following the steps from the official esp-idf documentation, the tools will be set globally by default.
Step 1: toolchain and building environment
Our project had a build.sh script which was a wrapper for different use cases — from setting up the environment to building and flashing. Its arguments are the same that could be passed to idf.py ( clean , flash , monitor , ….), but it would internally use GNU make directly. We wanted to keep the script, but it needed updating so this is how we set it up (only the important excerpts are shown here):
When the script is started, we check if the toolchain is there. If it’s not, it means it needs to be set up; otherwise, only the tools’ paths should be set up:
Once that’s done, our script calls BuildFromConfig() , which sets up the configuration and then calls esp-idf’s idf.py tool directly with appropriate arguments so it can do its job. Please note that we have several sdkconfig.defaults files ([release, debug]-[prod, preprod]) which is reflected in the code below. This is not always necessary.
When the build is done, build artifacts will be created inside the repo_root/commod_project/build directory.
Now we were ready for the changes in the project itself.
Step 2: setting up components; CMakeLists.txt
Our project had a mix of CMakeLists and component.mk files, but only CMakeLists are now needed. If you’re following the basic structure of the esp-idf based project, inside the project directory, you will have one component called main (in a directory of the same name) and one or more components in a directory called components.
You should then have one top-level CMakeLists.txt inside the root of your project directory, and one CMakeLists.txt file in each component directory (main and the additional components). Looking at stripped down version of our project, it should look like this:
Let’s get into the details of each CMakeLists.txt, starting with the top-level one.
Step 2.1: top-level project CMakeLists
At the start, we had CMakeLists.txt and Makefile in the root of our project which looked like this: CmakeLists.txt
Makefile
Since we only need the CMakeLists.txt file, we needed to make some adjustments. Our resulting CMakeFile.txt looks like this and Makefile is completely removed.
As you see, the top-level CMakeLists.txt hasn’t changed, but since Makefile was removed, additional changes needed to be made elsewhere — namely in the components’ CMakeLists files.
Step 2.2: “main” component CMakeLists
At the start, our main component had a component.mk file that looked like this:
Very basic, but when switching to CMake, we need to make a CMakeLists.txt for this component, and list all the source/header files.
Since our main component has only one .c file, that’s the only one that should be listed for sources, but the main component files also include some header files from our commod component, so they also need to be listed in INCLUDE_DIRS variable. This is what the resulting CMakeLists.txt looks like:
Step 2.3: additional components’ CMakeLists
As our project has only one component besides the main one, we only had to do it once. But if you have more, you need to do analogous changes for each of them. Our component commod had a CMakeLists.txt and a component.mk files when we started: component.mk
CMakeLists.txt
This CMakeLists.txt wasn’t really written according to CMake recommendations which state that all the source files should be listed manually so after the intervention, component.mk file was removed and CMakeLists.txt was changed to this:
⚠ Note that we also state all the esp-idf components our component requires. This is not needed for the “main” component, but it is for all the other components.
Step 3: additional configuration files
Our project used Kconfig.projbuild and Makefile.projbuild files to override certain aspects of the project build settings. For details, take a look at the esp-idf guide.
There was no need to change the Kconfig.projbuild as it is still used to insert additional options into the menuconfig tool, but Makefile.projbuild isn’t used anymore, so we needed to find another place for those settings.
Our Makefile.projbuild was used to enable additional BLE settings with a few defines:
which was now moved to project’s top-level CMakeLists.txt:
Step 4: build and test
That is a wrap-up for our adaptations. Upon cleaning up, our repo looks like this:
After all the changes are made, what’s left is a thorough test of the project in order to check that nothing is broken.
We already noticed that esp-idf v4.4.1 has changed some APIs (e.g. esp_event_loop ) and we need to think about changing the project to reflect that. Currently, the legacy code is still available, but it will surely be removed in the future.
That could be a topic for another How-To Guide.
