worker_tool

Some external tools have high startup costs. To amortize those costs over the whole build rather than paying them for each rule invocation, use the worker_tool() rule in conjunction with genrule(). Buck then starts the external tool once and reuses it by communicating with it over stdin and stdout using a simple JSON protocol.

Details

A worker_tool rule can be referenced in the cmd parameter of a genrule by using the macro:

$(exe //path/to:target)

Function Signature

def worker_tool(
    *,
    name: str,
    default_target_platform: None | str = None,
    target_compatible_with: list[str] = [],
    compatible_with: list[str] = [],
    exec_compatible_with: list[str] = [],
    visibility: list[str] = [],
    within_view: list[str] = ["PUBLIC"],
    metadata: OpaqueMetadata = {},
    tests: list[str] = [],
    modifiers: OpaqueMetadata = [],
    _apple_platforms: dict[str, str] = {},
    _worker_tool_runner: str = "prelude//js/worker_runner:worker_tool_runner",
    args: str | list[str] = [],
    contacts: list[str] = [],
    default_host_platform: None | str = None,
    env: dict[str, str] = {},
    exe: None | str = None,
    labels: list[str] = [],
    licenses: list[str] = [],
    max_workers: None | int = None,
    max_workers_per_thread_percent: None | int = None,
    persistent: None | bool = None,
) -> None

Parameters

name: (required)

name of the target
default_target_platform: (defaults to: None)

specifies the default target platform, used when no platforms are specified on the command line
target_compatible_with: (defaults to: [])

a list of constraints that are required to be satisfied for this target to be compatible with a configuration
compatible_with: (defaults to: [])

a list of constraints that are required to be satisfied for this target to be compatible with a configuration
exec_compatible_with: (defaults to: [])

a list of constraints that are required to be satisfied for this target to be compatible with an execution platform
visibility: (defaults to: [])

a list of visibility patterns restricting what targets can depend on this one
within_view: (defaults to: ["PUBLIC"])

a list of visibility patterns restricting what this target can depend on
metadata: (defaults to: {})

a key-value map of metadata associated with this target
tests: (defaults to: [])

a list of targets that provide tests for this one
modifiers: (defaults to: [])

an array of modifiers associated with this target
args: (defaults to: [])

A string of args that is passed to the executable represented by exe on initial startup.
contacts: (defaults to: [])

A list of organizational contacts for this rule. These could be individuals who you would contact in the event of a failure or other issue with the rule.
```
contacts = [ 'Joe Sixpack', 'Erika Mustermann' ]
```
env: (defaults to: {})

A map of environment variables that is passed to the executable represented by exe on initial startup.
exe: (defaults to: None)

A build target for a rule that outputs an executable, such as an sh_binary(). Buck runs this executable only once per build.
labels: (defaults to: [])

Set of arbitrary strings which allow you to annotate a build rule with tags that can be searched for over an entire dependency tree using buck query().
licenses: (defaults to: [])

Set of license files for this library. To get the list of license files for a given build rule and all of its dependencies, you can use buck query
max_workers: (defaults to: None)

The maximum number of workers of this type that Buck starts. Use -1 to allow the creation of as many workers as necessary.
max_workers_per_thread_percent: (defaults to: None)

The maximum ratio of workers of this type that Buck starts per thread, specified as a positive integer percentage (1-100). Must be greater than or equal to 1 and less than or equal to 100. Only one of max_workers and max_workers_per_thread_percent may be specified.
persistent: (defaults to: None)

If set to true, Buck does not restart the tool unless the tool itself changes. This means the tool persists across multiple Buck commands without being shut down and may see the same rule being built more than once. Be careful not to use this setting with tools that don't expect to process the same input—with different contents—twice!

Examples

Consider the following build rules:

#
# Buck
#
worker_tool(
  name = 'ExternalToolWorker',
  exe = ':ExternalTool',
  args = '--arg1 --arg2'
)

sh_binary(
  name = 'ExternalTool',
  main = 'external_tool.sh',
)

genrule(
  name = 'TransformA',
  out = 'OutputA.txt',
  cmd = '$(exe :ExternalToolWorker) argA',
)

genrule(
  name = 'TransformB',
  out = 'OutputB.txt',
  cmd = '$(exe :ExternalToolWorker) argB',
)

genrule(
  name = 'TransformC',
  out = 'OutputC.txt',
  cmd = '$(exe :ExternalToolWorker) argC',
)

When doing a buck build on all three of the above genrules, Buck first creates the worker process by invoking:

./external_tool.sh --arg1 --arg2

Buck then communicates with this process using JSON over stdin, starting with a handshake:

[
  {
    "id": 0,
    "type": "handshake",
    "protocol_version": "0",
    "capabilities": []
  }

Buck then waits for the tool to reply on stdout:

[
  {
    "id": 0,
    "type": "handshake",
    "protocol_version": "0",
    "capabilities": []
  }

Then, when building the first genrule, Buck writes to stdin:

  ,{
    "id": 1,
    "type": "command",
    "args_path": "/tmp/1.args",
    "stdout_path": "/tmp/1.out",
    "stderr_path": "/tmp/1.err"
  }

The file /tmp/1.args contains argA. The tool should perform the necessary work for this job and then write the job's output to the files supplied by Buck—in this case, /tmp/1.out and /tmp/1.err. Once the job is done, the tool should reply to Buck on stdout with:

  ,{
    "id": 1,
    "type": "result",
    "exit_code": 0
  }

Once Buck hears back from the first genrule's job, it submits the second genrule's job in the same fashion and awaits the response. When the build is all finished, Buck closes the JSON by writing to stdin:

]

which signals the tool that it should exit after replying on stdout with:

]

In this example, Buck is guaranteed to invoke

./external_tool.sh --arg1 --arg2

only once during the build. The three jobs corresponding to the three genrules are submitted synchronously to the single worker process.

Note that the id values in the messages are not necessarily increasing or sequential, but they do have to match between the request message and the response message of a given job as well as in the initial handshake.

If the tool receives a message type it cannot interpret it should answer with:

{
  "id": &ltn>,
  "type": "error",
  "exit_code": 1
}

If the tool receives a message type it can interpret, but the other attributes of the message are in an inconsistent state, it should answer with:

{
  "id": &ltn>,
  "type": "error",
  "exit_code": 2
}

Details​

Function Signature​

Parameters​

Examples​

Details

Function Signature

Parameters

Examples