Architectural Model

High-level Overview

Buck2 is a build system whose core is written in Rust. Starlark, which is a deterministic, immutable version of Python, is used to extend the Buck2 build system, enabling Buck2 to be language-agnostic.

The high-level flow starts with a user creating a build file (a BUCK file) containing one or more targets, which is specified by the target label, its inputs (sources, attributes, configurations, and dependencies), and the type of macro or rule to use.

Briefly, a macro is a wrapper around a rule, which runs necessary commands to generate what’s needed for a target (for example, for a cxx_binary target, generate the header map and run necessary clang commands). Macros can be used to reduce boilerplate code for users (such as to supply the same set of attributes for a rule for all targets). Macros and rules are both written in Starlark and are specified by input sources, attributes, and the implementation function.

If the target type is a macro, then the macro will fill in some details (for example, for a cxx_binary target, these are the compilation, debug flags to use, this is the clang to use). If the target type is a rule, then the macro layer is skipped altogether.

This is all orchestrated by the core, which performs operations such as executing Buck2 CLI args, generating/updating the dependency graph (which contains the configured target nodes, unconfigured target nodes, action nodes, among other types of nodes that all allow for incrementality and execution), and materializing the artifacts. The core is written in Rust.

The following diagram shows the high-level overview.

The Buck2 CLI runs in a client process, which sends commands to the Buck2 daemon via gRPC. The daemon goes through several phases after receiving a request from the client: evaluation, configuration, analysis, execution, and materialization (see Execution Model, below). When using buck2 test, there is a final stage for testing. Note that these are the phases that a build goes through, but they are not always sequential.

After finishing all phases, the daemon will send the response back to the client via gRPC.

Execution Model

The following diagram shows the Execution Model, which consists of 5 phases and states.

Each of the phases and states shown in the Execution Model, are detailed in the following sub-sections.

State 0 - Build Files

Build files (commonly referred to as BUCK files, their default name) are the main input to Buck2 and are syntactically Python.

Each build file is uniquely identified by the directory in which it's located. Since all build files have the same name, there cannot be two build files in the same directory. This is usually represented as the relative path from the root of the project (the directory where the .buckconfig file is).

Each build file has a set of targets. These describe the things the user wants Buck2 to know about. Each target has a type and a set of named attributes, including at least a name (also known as the label) identifying it. Additional attributes depend on the type of the target.

Phase A: Evaluation

First, Buck2 evaluates a build file, and then constructs an unconfigured target graph.

Buck2 performs directory listings to discover packages, then evaluates the build files that were found, expands any macros detected into their underlying rules, and then will take rule attributes and convert them from Starlark to Rust types to construct a target node, and insert it into the unconfigured target graph, which is a smaller portion of Buck2’s larger dependency graph. The target node consists of a reference to rule implementation, and the set of attributes and sources.

The result of evaluation is a list of targets read from the build file mapped to a target node in Buck2 unconfigured target graph.

State 1 - Unconfigured Target Graph is generated

At this point, the unconfigured target graph is available for the next stage of transformation, which is to configure the target nodes within the graph.

Phase B: Configuration

At the end of evaluation, the target nodes are not yet configured. Configuration means applying a list of constraints (such as resolving selects to specify the right CPU) to make sure the target can be run where it needs to. This is also known as target platform resolution, and can be configured within the target, the buckconfig, propagated from dependencies, or passed into the CLI. After applying configurations, the target nodes are transformed into configured target nodes within the Buck2 configured target graph, which is a smaller portion of Buck2’s larger dependency graph.

State 2 - Configured Target Graph is generated

At this point, the configured target graph is available for the analysis stage to generate the action graph.

Phase C: Analysis

In the analysis phase, Buck2 constructs a context object (ctx) which contains relevant information (such as attributes pulled from the configuration stage), all converted into Starlark types and made available to the rule. For example, the target’s dependencies are turned into a ProviderCollection, source files are converted into StarlarkArtifacts, and String attributes are turned into a StarlarkString. This ctx object is backed by Buck2’s dependency graph for computation and rules use it to tell Buck2 to run actions, create dynamic actions, or create new files.

The rule will return a list of providers, which is data that the rule wants to expose to its dependents (that is, can flow through the dependency graph), such as output artifact information (such as file paths and file hashes). Providers could be actions, source files, or attributes. Within the returned list, DefaultInfo always needs to be returned, which indicates what the default outputs are. Some other common built-in providers include RunInfo, TestInfo, and InstallInfo.

The end result is a list of providers and actions (inserted into the action graph) that Buck2 needs to execute to produce the desired outputs, known as 'bound artifacts'.

State 3 - Action Graph and Providers are generated

At this point, the action graph and providers are available to be processed by the execution stage.

Phase D: Execute

Execution is where Buck2 takes all the providers (input files from the targets, args from the command line), runs the actions, and then outputs the computed results. The critical path is the theoretical lower bound for the duration of a build, which are the slowest set of actions.

Buck2 can be run locally or on remote execution, or in a hybrid manner.

For each action, an input action digest is created from the action (hash of command line and all of the action’s inputs), uploaded, and cached within RE. This is known as the RE action cache.

If there is a cache hit, then Buck2 does not need to run the command for the action, and RE returns the output action digest. This is known as remote execution.

If there is not a cache hit, then local execution has to be done, where all the action’s input files are retrieved from the filesystem (most likely from EdenFS), computation is run on these source files, and then outputted to buck-out using I/O operations in the filesystem.

Hybrid execution allows Buck2 to race local and remote execution and return the returns of whichever finishes first for a performance speedup.

These action digests are how Buck2 communicates with RE. The action outputs, including final/build artifacts, intermediaries, file, directories, and symlinks related to the build, are then materialized (downloaded to disk), and can be found in the buck-out path. There are different configurations that a user can set to control how materialization is handled.

State 4 - Build outputs are generated

At this point, the build is complete.

If a user ran buck2 test, then there is a final transformation for Buck2 to construct a command for TPX to execute the actual test.

Phase E: Execute tests

For more detail on testing, review Test Execution.