The Lost Art of the MakefileWhat a Makefile is and why it’s importantOlio AppsBlockedUnblockFollowFollowingMar 14By Jesse Hallett — originally posted February 28th, 2018An arrow sign on a snowy mountain.
I want to talk about the merits of Make (specifically GNU Make).
Make is a general-purpose build tool that has been improved upon and refined continuously since its introduction over forty years ago.
It is very good at incremental builds, which can save a lot of time when you rebuild after changing one or two files in a large project.
Make has been around long enough to have solved problems that newer build tools are only now discovering for themselves.
Despite the title of this post, Make is still widely used.
You are more likely to see a Makefile in a C or C++ project, for example.
You can find the complete file here.
When to stick with WebpackThe job that Webpack does is quite specialized.
If you are writing a frontend app and you need code bundling you should absolutely use Webpack (or a similar tool like Parcel).
On the other hand if your needs are more general Make is a good go-to tool.
I use Make when I am writing a client- or server-side library, or a Node app.
Those are cases where I do not benefit from the specialized features in Webpack.
I want to be able to write Stage 4 ECMAScript while targeting browsers or recent stable versions of Node.
So I use Make to transpile code using Babel.
Introducing the MakefileMake looks for a file called Makefile in the current directory.
A Makefile is a list of tasks that generally look like this:Unless you specify otherwise, Make assumes that the target (target_file in this example) and prerequisites (prerequisite_file1 and prerequisite_file2) are files or directories.
You can ask Make to build a target from the command line like this:If the target_file does not exist, or if prerequisite_file1 or prerequisite_file2 have been modified since target_file was last built, Make will run the given shell commands.
But first Make will check to see if there are recipes in the Makefile for prerequisite_file1 and prerequisite_file2 and build or rebuild those if necessary.
A practical example of a Makefile ruleA minimal project might have a file called src/index.
We want a rule that tells Make to transpile that file and write the result to lib/index.
But Make looks at things the other way around: Make expects to be told the desired result, and it uses rules to work out how to produce that result.
So we write a Makefile with a rule where the target is lib/index.
js and src/index.
js is a prerequisite:The recipe uses babel to produce lib/index.
js using src/index.
js as input.
The shell commands in a Makefile recipe are almost exactly what you would type in bash – but note that Make substitutes variables and expressions prefixed with $ before commands are executed.
You can escape a $ in a recipe command by doubling it (e.
In the recipe above there are two special variables: $< is a shorthand for the list of prerequisites (src/index.
js in this case) and $@ is the target (lib/index.
We will see why those variables are indispensable in a moment.
The mkdir -p line creates the lib/ directory in case it does not already exist.
The function dir extracts the directory portion from a file path.
So $(dir $@) is read as “the path to the directory that contains the file referenced by $@”.
A target and it’s prerequisites can include wildcards to create a pattern:This tells Make that any file path that begins with lib/ can be built using the given steps, and that the target depends on a matching path under src/.
Whatever string Make substitutes in the position of the % in the target, it substitutes the same string for % on the prerequisite’s side.
Now it becomes clear why the variables $< and $@ are necessary: we won’t know what the values of those variables will be until the rule is invoked.
Why invoke Babel separately for each source file?Babel can transpile all files in a directory tree with one invocation.
But the rule above will run babel separately for every file under src/.
There is some startup time overhead every time babel runs; so invoking babel many times is slower when building from a fresh checkout.
But thanks to Make’s talent for incremental builds separate invocations make incremental builds much faster.
When we ask Make to transpile all files under src/ it will skip files that already have up-to-date results under lib/.
I run incremental builds far more often than full builds so I appreciate the speedup!Edit: several commenters on Hacker News (falcolas, Jtsummers, jlg23, nzoschke) point out that Make can run tasks in parallel.
Because Make rules explicitly list dependencies for each target Make knows which tasks can be run in parallel safely.
Using the command make –jobs=4 will run up to four instances of Babel at once, which can offset some of the performance loss of running a separate instance of Babel for each source file.
Locating BabelI have the above rule in my Makefile with one small change:The babel executable is provided by the babel-cli NPM package.
I prefer to install babel-cli as a project dev dependency, which causes babel executable to be installed at the path node_modules/.
That way anyone who wants to build my project does not have to take a special step to install babel-cli globally.
But then babel will not be in the executable $PATH on most machines.
To avoid typing out the path to the executable I assign the location of babel to a variable in the Makefile (babel := node_modules/.
bin/babel), and use Make’s variable substitution to splice that path into recipe commands.
(Pro tip: you can add node_modules/.
bin to your shell $PATH like this: PATH="node_modules/.
That makes it easy to run executables installed by dependencies of the project in your current directory.
Executables installed with the project will take precedence over executables installed globally.
NPM automatically makes this $PATH adjustment when you run NPM scripts.
I type out the path to babel in my Makefile because I do not want to assume that other people have made the same $PATH modification, and I do not always run make from an NPM script.
But you probably do not want to run make manually for every source file.
What you want is to be able to just type make and have it transpile all source files.
Remember that Make needs to be told the results that you want.
To do that, first compute a list of all source files and assign it to a variable:The expression on the right side of that assignment uses Make’s built-in shell function to run an external shell command.
In this case we use the find command to recursively list all files under src/ that have the extension .
You could use another command like [fd] – but find is more likely to be installed on your colleagues’ workstations and on your CI server.
That gives us a list of files that we have.
But we need to tell Make which files we want.
For every file under src/ we want a transpiled file with a matching path under lib/.
We can compute that list by applying Make’s patsubst function to the path of every source file:The substitution expression uses % as a wildcard in the same way as the rule that we wrote earlier.
Now we can define a target that lists the files that we want as prerequisites.
When we request that target, Make will automatically build a transpiled result for every source file:The target name all is special: When you run make with no target specified it will evaluate the all target by default.
This is a case where the target is not a file or directory – all is just a label.
You should declare non-file targets in your Makefile like this so that Make does not waste time or confuse itself trying to find matching files in your project:Oh yeah, you probably want a way to remove build artifacts so that you can build cleanly.
With this target you can run make clean to do that:Automatically install node modules when package.
json changesMake is powerful enough to accomplish pretty much any task that you can imagine.
Do you ever pull updates to a project, and find out after some debugging that you forgot to run yarn install to update your dependencies?.You can catch that with Make!.When you run yarn install the result is that the node_modules directory is created or updated.
You can add a rule for the node_modules target to represent that fact to Make.
The state of node_modules depends on the content of package.
json and yarn.
lock, so those files should be listed as prerequisites:This change to the all target adds node_modules as a prerequisite:Now Make will run yarn install if and only if package.
json or yarn.
lock has changed since the last build.
I put node_modules before $(transpiled_files) just in case new dependencies include items such as updates to Babel modules that might affect the way that project files should be built.
Watch files and rebuild on changesEvery build tool should have a watch-files-for-changes option for rapid development.
You can get that effect by pairing Make with a general purpose file-watching tool:Just make sure that you do not watch lib/ or you will get into an infinite build loop.
Using Make to distribute Flow type definitionsI mentioned that I often use Flow to type-check my projects.
Flow supports that by looking for a files with the .
js; Flow will additionally look for a file called User.
flow in the same directory, which should be the original source file with type annotations.
My Makefile copies every file under src/ to the corresponding path under lib/ and adds a .
flow extension according to this rule:To make sure that Flow runs this step for all source files I compute the list of .
flow files that I expect the same way that we computed the list of transpiled files that we expect:And I include flow_files in the prerequisites of the all task:Going furtherMake has many capabilities that I have not touched on here.
For example Make supports macros that can compute rules on-the-fly for especially complex use cases.
And a Makefile can delegate to targets in other Makefiles, which is useful when distributing Make libraries, or for multi-tiered projects where a build process involves combining artifacts from building multiple subprojects.
There is much information to be found in the Gnu Make Manual.
Olio Apps creates software for the web, mobile, and other platforms.
To read more articles like this, subscribe to our newsletter.
Originally published at www.