vystem/docs/vybuild/actions.md

Actions

Introduction

Actions are atomic blocks of the pipeline. They are used for all sorts of things such as compiling, filesystem manipulation, control flow, cache management and command execution. Please note that things related to VFTM aren't documented in this documentation.

Syntax

Actions are listed in the actions field of each module. They are executed in the order they are specified, without any automatic reordering. Actions all follow at their cores the same syntax:

    {
      "action":"...",
      "args":{
        ...
      }
    }

The action field specify the name of the action to execute and the args field specify all sorts of arguments, specific to each actions.

List of all available actions

Commands related actions

Note: There is, for the moment, no action capable of combining the saving of output of a command with a working directory for example

run_command_wait

The run_command_wait action allows you to run a command and wait for it to finish. If the command fail, the action and therefore the entire module fail, interrupting the build process. It can be used in multithreaded build pipelines without restriction.

Syntax:

    {
      "action":"run_command_wait",
      "args":{
        "command_variables":...,
        "command":[
          ...
        ],
        "show_output":...,
        "success_status":[
          ...
        ],
        "ignore_success_status":...
      }
    }

Arguments:

Name	Type	Possible values	Role
command_variables	boolean	true or false	Indicate if variables replacement should be done in the command argument.
command	list of strings	anything	The list of arguments that will be passed to the execvp function. The first element is considered as the name of the executable, and should be either accessible in PATH or from the indicated root_folder of the module.
show_output	string	live, on_failure or silent	Indicate how the output of the command should be shown. Uses live if your command require user input in the console. Failure detection is based on output codes of the processus.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.

run_command_wait_dir

The run_command_wait_dir action behave exactly as the run_command_wait action, except it allows you to provide a path in which VyBuild will chdir before executing the command. It can be used in multithreaded build pipelines without restriction.

Syntax:

    {
      "action":"run_command_wait_dir",
      "args":{
        "command_variables":...,
        "command":[
          ...
        ],
        "success_status":[
          ...
        ],
        "ignore_success_status":...,
        "show_output":...,
        "working_dir_variables":...,
        "working_dir":...
      }
    }

Arguments:

Name	Type	Possible values	Role
command_variables	boolean	true or false	Indicate if variables replacement should be done in the command argument.
command	list of strings	anything	The list of arguments that will be passed to the execvp function. The first element is considered as the name of the executable, and should be either accessible in PATH or from the indicated working_dir argument.
show_output	string	live, on_failure or silent	Indicate how the output of the command should be shown. Uses live if your command require user input in the console. Failure detection is based on output codes of the processus.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.
working_dir_variables	boolean	true or false	Indicate if variables replacement should be done in the working_dir argument.
working_dir	string	any valid path	Indicate the working directory in which VyBuild should chdir before executing the command.

save_command_content

The save_command_content behave exactly like the run_command_wait action, except instead of showing the output, it save it into a file. It can be used in multithreaded build pipelines without restriction.

Syntax:

    {
      "action":"save_command_content",
      "args":{
        "command_variables":...,
        "command":[
          ...
        ],
        "success_status":[
          ...
        ],
        "ignore_success_status":...,
        "command_output_file_variables":...,
        "command_output_file":...
      }
    }

Arguments:

Name	Type	Possible values	Role
command_variables	boolean	true or false	Indicate if variables replacement should be done in the command argument.
command	list of strings	anything	The list of arguments that will be passed to the execvp function. The first element is considered as the name of the executable, and should be either accessible in PATH or from the indicated working_dir argument.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.
command_output_file_variables	boolean	true or false	Indicate if variables replacement should be done in the command_output_file argument.
command_output_file	string	any valid path	Indicate the file in which the output of the command should be saved. The previous content of the file is erased. The parent directory of the file must exists.

run_command_str_wait

The run_command_str_wait behave exactly like the run_command_wait action, except that instead of providing the command in a list of arguments, they are provided as a single string that is then processed with wordexp, which parse them nearly exactly as a shell would do. The variables replacement is ran before the wordexp call. It can be used in multithreaded build pipelines without restriction.

    {
      "action":"run_command_str_wait",
      "args":{
        "command_variables":...,
        "command":...,
        "success_status":[
          ...
        ],
        "ignore_success_status":...,
        "show_output":...
      }
    }

Arguments:

Name	Type	Possible values	Role
command_variables	boolean	true or false	Indicate if variables replacement should be done in the command argument.
command	string	anything	The string that will be parsed by wordexp and then fed into execvp. The first element of the parsing result is considered as the name of the executable, and should be either accessible in PATH or from the indicated root_folder of the module.
show_output	string	live, on_failure or silent	Indicate how the output of the command should be shown. Uses live if your command require user input in the console. Failure detection is based on output codes of the processus.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.

Compilation related actions

The compile_one_* action are safe and authorized in multithreaded pipelines, but compile_multiple_cppand vyld_compilation can only be used in parts of pipelines that aren't in a multithreaded phase.

compile_one_asm

The compile_one_asm action, as its name imply, compile a single assembly file. It is intended for simple and single compilation operation. It can be used in multithreaded build pipelines without restriction.

Syntax:

    {
      "action":"compile_one_asm",
      "args":{
        "source_variables":...,
        "source":...,
        "assembler_variables":...,
        "assembler":...,
        "pre_arguments_variables":...,
        "pre_arguments":[
          ...
        ],
        "post_arguments_variables":...,
        "post_arguments":[
          ...
        ],
        "output_file_variables":...,
        "output_file":...,
        "success_status":[
          ...
        ],
        "ignore_success_status":...,
        "cache_authorized":...
      }
    }

Arguments:

Name	Type	Possible values	Role
source_variables	boolean	true or false	Indicate if variables replacement should be done in the source argument.
source	string	any valid path	Provide the path (path relative to the root_folder field of the module or absolute path) to the file to compile.
assembler_variables	boolean	true or false	Indicate if variables replacement should be done in the assembler argument.
assembler	string	any valid executable name	Provide the executable name of the assembler to use for this action. It must be an executable accessible in PATH.
pre_arguments_variables	boolean	true or false	Indicate if variables replacement should be done in the pre_arguments argument.
pre_arguments	list of strings	anything	Indicate the pre arguments to fed into execvp
post_arguments_variables	boolean	true or false	Indicate if variables replacement should be done in the post_arguments argument.
post_arguments	list of strings	anything	Indicate the post arguments to fed into execvp
output_file_variables	boolean	true or false	Indicate if variables replacement should be done in the output_file argument.
output_file	string	any valid path	A path relative to root_folder that indicate to VyBuild which file to put into cache or to extract from cache. The parent folder must already exists.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.
cache_authorized	boolean	true or false	Allows VyBuild to extract output file from cache if the compilation context matchs, and to store the output file into the cache

The compilation context is used to generate a key for the cache. For this action, the compilation context contain: source file content, assembler binary hash, assembler path, assembler pre and post arguments.

The command is executed using execvp. The arguments are concatenated as follows:

1. Resolved compiler path.
2. Pre-arguments (in order).
3. Source file path.
4. Post-arguments (in order).

The user is responsible to provide any -o assembler argument where the assembler require it. This is this limitation that indicate which arguments to put into pre and post arguments.

compile_one_cpp

The compile_one_cpp action compiles a single C or C++ source file. It is designed to handle dependency tracking through a dedicated header discovery command, ensuring that the cache is invalidated if any included header file is modified. It can be used in multithreaded build pipelines without restriction.

Syntax:

    {
      "action":"compile_one_cpp",
      "args":{
        "source_variables":...,
        "source":...,
        "compiler_variables":...,
        "compiler":...,
        "pre_arguments_variables":...,
        "pre_arguments":[
          ...
        ],
        "post_arguments_variables":...,
        "post_arguments":[
          ...
        ],
        "output_file_variables":...,
        "output_file":...,
        "success_status":[
          ...
        ],
        "ignore_success_status":...,
        "cache_authorized":...,
        "headers_command_variables":...,
        "headers_command":...
      }
    }

Arguments:

Name	Type	Possible values	Role
source_variables	boolean	true or false	Indicates if variables replacement should be performed in the source argument.
source	string	any valid path	Provides the path (relative to root_folder or absolute) to the C/C++ file to compile.
compiler_variables	boolean	true or false	Indicates if variables replacement should be performed in the compiler argument.
compiler	string	any valid executable name	Provides the executable name of the compiler (e.g., gcc, g++, clang). Must be accessible in PATH.
pre_arguments_variables	boolean	true or false	Indicate if variables replacement should be done in the pre_arguments argument.
pre_arguments	list of strings	anything	Arguments to be passed to the compiler before the source file.
post_arguments_variables	boolean	true or false	Indicate if variables replacement should be done in the post_arguments argument.
post_arguments	list of strings	anything	Arguments to be passed to the compiler after the source file (for example -o and the output path).
output_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the output_file argument.
output_file	string	any valid path	A path relative to root_folder that indicate to VyBuild which file to put into cache or to extract from cache. The parent folder must already exists.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.
cache_authorized	boolean	true or false	Allows VyBuild to use the cache for this action.
headers_command_variables	boolean	true or false	Indicates if variables replacement should be performed in the headers_command argument.
headers_command	string	shell command	A command string used to extract header dependencies.

To ensure perfect cache integrity, VyBuild must know which headers are included by the source file. The headers_command field is executed in a shell. VyBuild automatically provides two internal variables for this command:

• %sourcefile%: The path to the source file being compiled
• %dfile%: A temporary path where VyBuild expects the dependency information to be written (usually in Makefile format)

Example for GCC: "headers_command": "gcc -MMD -MF %dfile% -E %sourcefile% -o /dev/null"

The compilation context used to generate the cache key for this action includes:

• The content of the source file
• The content of all discovered header files (using headers_command)
• The compiler binary hash and its system path
• All pre and post arguments

The command is executed using execvp. The arguments are concatenated as follows:

1. Resolved compiler path.
2. Pre-arguments (in order).
3. Source file path.
4. Post-arguments (in order).

compile_multiple_cpp

The compile_multiple_cpp action allows for the mass compilation of C and C++ source files. It is the multithreaded engine of VyBuild, capable of handling large sets of files either by explicit listing or via a powerful path completion system. This action automatically manages dependency tracking and caching for each individual file.

Syntax:

    {
      "action":"compile_multiple_cpp",
      "args":{
        "source_files_path_completion":...,
        "path_completion_source_ext":...,
        "path_completion_output_ext":...,
        "source_files_variables":...,
        "source_files":[
          ...
        ],
        "compiler_variables":...,
        "compiler":...,
        "pre_arguments_variables":...,
        "pre_arguments":[
          ...
        ],
        "post_arguments_variables":...,
        "post_arguments":[
          ...
        ],
        "output_files_variables":...,
        "output_files":[
          ...
        ],
        "success_status":[
          ...
        ],
        "ignore_success_status":...,
        "cache_authorized":...,
        "headers_command_variables":...,
        "headers_command":...
      }
    }

Arguments:

Name	Type	Possible values	Role
source_files_path_completion	boolean	true or false	Enables the automatic discovery of source files using wildcards.
path_completion_source_ext	string	e.g., .cpp	The extension of source files to look for during path completion.
path_completion_output_ext	string	e.g., .o	The extension to apply to the generated output files.
source_files_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_files list
source_files	list of strings	paths	List of source files or directory patterns (if path completion is enabled).
compiler_variables	boolean	true or false	Indicates if variables replacement should be performed in the compiler argument.
compiler	string	any valid executable name	Provides the executable name of the compiler (e.g., gcc, g++, clang). Must be accessible in PATH.
pre_arguments_variables	boolean	true or false	Indicate if variables replacement should be done in the pre_arguments argument.
pre_arguments	list of strings	anything	Arguments to be passed to the compiler before the source file.
post_arguments_variables	boolean	true or false	Indicate if variables replacement should be done in the post_arguments argument.
post_arguments	list of strings	anything	Arguments to be passed to the compiler after the source file (for example -o and the output path).
output_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the output_file argument.
output_file	string	any valid path	A path relative to root_folder that indicate to VyBuild which file to put into cache or to extract from cache. The parent folder must already exists. Must be empty if using path completion.
success_status	list of integers	any integers	Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
ignore_success_status	boolean	true or false	If this field is true, no output code will fail the action.
cache_authorized	boolean	true or false	Allows VyBuild to use the cache for this action.
headers_command_variables	boolean	true or false	Indicates if variables replacement should be performed in the headers_command argument.
headers_command	string	shell command	A command string used to extract header dependencies.

VyBuild features an advanced path completion system to avoid manual file listing. It is triggered when source_files_path_completion is true.

Syntax:

• path/to/dir/* : VyBuild will look for all files with path_completion_source_ext in that specific directory.
• path/to/dir/** : VyBuild will recursively search for all matching files in that directory and all its subdirectories.

When path completion is used, the output_files field must be empty. VyBuild will automatically generate the output paths by taking the source file path and replacing the source extension with the path_completion_output_ext. Variables replacement is performed on the source_files strings before the path completion engine scans the filesystem.

To ensure perfect cache integrity, VyBuild must know which headers are included by the source file. The headers_command field is executed in a shell. VyBuild automatically provides two internal variables for this command:

• %sourcefile%: The path to the source file being compiled
• %dfile%: A temporary path where VyBuild expects the dependency information to be written

Example for GCC: "headers_command": "gcc -MMD -MF %dfile% -E %sourcefile% -o /dev/null"

For pre_arguments and post_arguments, VyBuild provides the following variable:

• %outputfile%: The path of the current output file (either explicitly provided in output_files or auto-generated via path completion). This is typically used with the -o flag.

The compilation context used to generate the cache key for each file individually for this action includes:

• The content of the source file
• The content of all discovered header files (using headers_command) for this file
• The compiler binary hash and its system path
• The specific pre and post arguments (including the resolved %outputfile%)

This action is natively multithreaded. VyBuild will distribute the compilation of the file set across the available CPU threads. The command of compilation for each file is executed using execvp. The order of execution within the set is not guaranteed, but the arguments are strictly concatenated as follows:

1. Resolved compiler path.
2. Pre-arguments (in order, local to each file).
3. Source file path (local to each file).
4. Post-arguments (in order, local to each file).

vyld_compilation

The vyld_compilation action is a high-level orchestration tool designed to produce a final VYX executable. It automates the entire pipeline: compiling C/C++ and Assembly files into temporary objects, managing a per-object cache, generating appropriate linker scripts, and performing the final link and VYX header injection.

Syntax:

    {
      "action":"vyld_compilation",
      "args":{
        "source_files_path_completion":...,
        "source_files_variables":...,
        "source_files":[
          ...
        ],
        "compiler_variables":...,
        "compiler":...,
        "source_asm_path_completion":...,
        "source_asm_variables":...,
        "source_asm":[
          ...
        ],
        "assembler_variables":...,
        "assembler":...,
        "linker_variables":...,
        "linker":...,
        "compiler_flags_variables":...,
        "compiler_flags":...,
        "output_file_variables":...,
        "output_file":...,
        "block_if_fail":...,
        "object_files_cache_authorized":...,
        "headers_command_variables":...,
        "headers_command":...,
        "payloads":[
          {
            "source_file_variables":...,
            "source_file":...,
            "is_executable":...,
            "is_writable":...,
            "expected_va":...
          }
        ]
      }
    }

Arguments:

Name	Type	Possible values	Role
source_files_path_completion	boolean	true or false	Enables the automatic discovery of .c source files using wildcards.
source_files_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_files list
source_files	list of strings	paths	List of source files or directory patterns (if path completion is enabled).
compiler_variables	boolean	true or false	Indicates if variables replacement should be performed in the compiler argument.
compiler	string	any valid executable name	Provides the executable name of the compiler (e.g., gcc, g++, clang). Must be accessible in PATH.
source_asm_path_completion	boolean	true or false	Enables the automatic discovery of .asm source files using wildcards.
source_asm_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_asm list
source_asm	list of strings	paths	List of source assembly files or directory patterns (if path completion is enabled).
assembler_variables	boolean	true or false	Indicates if variables replacement should be performed in the assembler argument.
assembler	string	any valid executable name	Provides the executable name of the assembler (e.g., nasm). Must be accessible in PATH and have nasm-like argument syntax.
linker_variables	boolean	true or false	Indicates if variables replacement should be performed in the linker argument.
linker	string	any valid executable name	Provides the executable name of the linker (e.g., ld). Must be accessible in PATH and be compatible with ld linker script.
compiler_flags_variables	boolean	true or false	Indicates if variables replacement should be performed in the compiler_flags argument.
compiler_flags	string	any valid compiler arguments	Custom flags passed to the C compiler during the object generation phase.
output_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the output_file argument.
output_file	string	any valid path	A path relative to root_folder that indicate where to put the produced VYX executable. The parent folder must already exists.
block_if_fail	boolean	true or false	If true, any error in the pipeline stops the entire action
object_files_cache_authorized	boolean	true or false	Enables caching at the individual object file level (not the final binary).
headers_command_variables	boolean	true or false	Indicates if variables replacement should be performed in the headers_command argument.
headers_command	string	shell command	A command string used to extract header dependencies.

Each entry in the payloads list defines an external resource to be embedded in the VYX file:

Name	Type	Possible values	Role
source_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_file string.
source_file	string	any valid path	Path to the raw data file to be integrated.
is_executable	boolean	true or false	Sets the executable flag in the VYX payload header.
is_writable	boolean	true or false	Sets the writable flag in the VYX payload header.
expected_va	string	string formatted as 0xNNNNNNNNNNNNNNNN where each N must be a valid hexadecimal number	The specific virtual address where this payload must be mapped.

When vyld_compilation is triggered, VyBuild call to its Vyld integration, which follows a rigorous multi-stage process to ensure the resulting binary strictly adheres to the VYX specifications:

1. Object Compilation Phase

VyBuild first processes all C/C++ source files with multithreading. Each file is compiled into a temporary object file. If caching is enabled, Vyld uses the VyBuild cache to checks for each individual object. The cache key includes the source content, discovered headers, compiler hash, and flags. Object files are stored in a managed temporary directory and are not exposed to the user after the build.

2. Assembly Phase

All assembly files are then processed. Unlike C compilation, this phase is currently single-threaded. The assembler is invoked with NASM-compatible arguments to produce the remaining object files.

3. Automated Linker Script Generation

Instead of requiring a manual linker script, vyld_compilation dynamically generates a script tailored for the VYX format. It enforces:

• The specific section order: .text -> .data -> .rodata -> .bss
• Memory alignment on 4 kilobytes boundaries (0x1000)
• Correct mapping of the base addresses generated in the build pipeline

4. Final Linking & VYX Wrapping

The linker is called using the generated script to produce a temporary ELF binary. VyBuild then:

• Extracts the raw binary data for each section
• Calculates the required padding to ensure each section starts on a new page
• Header Injection: Constructs the VYX header
• Payload Integration: Appends each specified payload at the end of the file, preceded by the payload header

To ensure perfect cache integrity, VyBuild must know which headers are included by the source file. The headers_command field is executed in a shell. VyBuild automatically provides two internal variables for this command:

• %sourcefile%: The path to the source file being compiled
• %dfile%: A temporary path where VyBuild expects the dependency information to be written

Example for GCC: "headers_command": "gcc -MMD -MF %dfile% -E %sourcefile% -o /dev/null"

Filesystem related actions

All filesystem related actions are allowed in multithreaded parts of the build pipeline. However, its the responsability of the user to ensure that the effect of a filesystem action in a multithreaded part of the pipeline doesn't collide with another action. Please keep in mind that not all filesystems manipulations are possible yet.

ensure_folder_existence

The ensure_folder_existence action will check if a folder exists. If the folder doesn't exist, it will be created. This action can create multiple folders at once.

Syntax:

    {
      "action":"ensure_folder_existence",
      "args":{
        "path_variables":...,
        "path":...
      }
    }

Arguments:

Name	Type	Possible values	Role
path_variables	boolean	true or false	Indicates if variables replacement should be performed in the path string.
path	string	any valid path	Indicate the path of the folder to ensure the existence of.

move_file

The move_file action move a file or a folder to another folder. The destination path must be a folder and exists before the action. The original file/folder keeps its name.

Syntax:

    {
      "action":"move_file",
      "args":{
        "source_file_variables":...,
        "source_file":"...",
        "destination_folder_variables":...,
        "destination_folder":...
      }
    }

Arguments:

Name	Type	Possible values	Role
source_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_file string.
source_file	string	any valid path	Indicate the path of the source file/folder.
destination_folder_variables	boolean	true or false	Indicates if variables replacement should be performed in the destination_folder string.
destination_folder	string	any valid path	Indicate the path of the destination folder.

Warning: while this action is theoretically capable of moving folders, it is not intended or designed for that.

copy_folder

The copy_folder action copy a folder into another folder. Both source_folder and destination_folder must be path to existing folders. It is a recursive copy that overwrite existing files/folders.

Syntax:

    {
      "action":"copy_folder",
      "args":{
        "source_folder_variables":...,
        "source_folder":...,
        "destination_folder_variables":...,
        "destination_folder":...
      }
    }

Arguments:

Name	Type	Possible values	Role
source_folder_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_folder string.
source_folder	string	any valid path	Indicate the path of the source folder to copy.
destination_folder_variables	boolean	true or false	Indicates if variables replacement should be performed in the destination_folder string.
destination_folder	string	any valid path	Indicate the path of the destination folder.

ensure_folder_reset

The ensure_folder_reset action will check if a folder exists. If the folder doesn't exist, it will be created. If the folder exists, it will get removed (with all its content) and recreated. This action can create multiple folders at once.

Syntax:

    {
      "action":"ensure_folder_reset",
      "args":{
        "path_variables":...,
        "path":...
      }
    }

Arguments:

Name	Type	Possible values	Role
path_variables	boolean	true or false	Indicates if variables replacement should be performed in the path string.
path	string	any valid path	Indicate the path of the folder to ensure the reset of.

copy_file

The copy_file action copy a file into a folder. source_file must be a path to an existing file, and destination_folder must be an existing folder. It will overwrite the file if a file with the same name in the destination folder already exists.

Syntax:

    {
      "action":"copy_file",
      "args":{
        "source_file_variables":...,
        "source_file":...,
        "destination_folder_variables":...,
        "destination_folder":...
      }
    }

Arguments:

Name	Type	Possible values	Role
source_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_file string.
source_file	string	any valid path	Indicate the path of the source file to copy.
destination_folder_variables	boolean	true or false	Indicates if variables replacement should be performed in the destination_folder string.
destination_folder	string	any valid path	Indicate the path of the destination folder.

Cache related actions

The cache being thread-safe, these two actions can be used safely in multithreaded part of any build pipelines.

extract_from_cache

The extract_from_cache action will extract a file from the cache based on the provided key. If a file already exists at the path indicated by output_file, it will overwrite it. If the cache doesn't contain anything associated to that key, the action fails silently without stopping the build pipeline.

Syntax:

    {
      "action":"extract_from_cache",
      "args":{
        "target_variables":...,
        "target":...,
        "output_file_variables":...,
        "output_file":...
      }
    }

Arguments:

Name	Type	Possible values	Role
target_variables	boolean	true or false	Indicates if variables replacement should be performed in the target string.
target	string	anything	Indicate the key to use to retrieve the cached object.
output_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the output_file string.
output_file	string	any valid path	Indicate the path of the destination file.

store_in_cache

The store_in_cache action store a file in cache. If the cache already contains the provided key, the overwrite_allowed argument will specify if the cache object is overwritten or not. If overwrite_allowed is false and the cache already contains the key, the action will fail silently.

Syntax:

    {
      "action":"store_in_cache",
      "args":{
        "key_variables":...,
        "key":...,
        "file_variables":...,
        "file":...,
        "overwrite_allowed":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything	Indicate the key to use to store the file.
file_variables	boolean	true or false	Indicates if variables replacement should be performed in the file string.
file	string	any valid path	Indicate the path of the file to store in cache.
overwrite_allowed	boolean	true or false	Indicates if the overwrite of already-existing cache object is allowed.

Control flow related actions

VyBuild provides a way to build dynamic build pipelines using control flow actions. These actions are the following:

• run_actions_if: contains two arguments, condition and actions. actions is a list of actions to execute if the evaluated condition is satisfied.
• run_actions_if_else: contains three arguments, condition, actions_if and actions_else. actions_if is a list of actions executed if the condition is satisfied. Otherwise, actions_else is a list of actions executed if the condition isn't satisfied.
• run_submodule_if: this action is the same as run_submodule, except it contains a condition object that will determine if the submodule is executed or not, based on the satisfaction of the condition.

These two actions enforce the rules on multithreading usage depending on where they are executed.

Conditions

Conditions are a special kind of JSON objects in VyBuild. They have the following syntax:

    "condition":{
      "condition_variables":...,
      "condition":...,
      "condition_args_variables":...,
      "condition_args":[
        ...
      ]
    }

condition_variables and condition_args_variables are booleans specifying if the variables replacement process should be performed on their respective fields. condition is a string and condition_args is a list of strings.

Here are described the available conditions with their arguments and behaviour:

Name	Arguments	Behaviour
folder_exists	Expect one argument: the path of the folder	Returns true if the folder exists
folder_not_exists	Expect one argument: the path of the folder	Returns true if the folder doesn't exist
folder_empty	Expect one argument: the path of the folder	Returns true if the folder is empty
folder_not_empty	Expect one argument: the path of the folder	Returns true if the folder isn't empty
cache_contains	Expect one argument: the key of the cache object	Returns true if the cache contains an object associated with that key
cache_not_contains	Expect one argument: the key of the cache object	Returns true if the cache doesn't contain an object associated with that key
var_exists	Expect one argument: the name of the variable	Returns true if the variable exists
var_not_exists	Expect one argument: the name of the variable	Returns true if the variable doesn't exist
var_equals	Expect two arguments in order: the name of the variable and the value	Returns true if the variable is equal to the value. Stop the build pipeline if variable doesn't exist
var_not_equals	Expect two arguments in order: the name of the variable and the value	Returns true if the variable isn't equal to the value. Stop the build pipeline if variable doesn't exist.

exit_current_module

The exit_current_module action is a special action that can only be executed in a list of actions from a run_actions_if or run_actions_if_else action. It exits the current module, and stop the build pipeline if the current module is the root module.

Syntax:

    {
      "action":"exit_current_module",
      "args":{
        "is_fail":...
      }
    }

The is_fail argument allows to completely stop the build pipeline even if the current module isn't the root module.

Multithreading related actions

VyBuild provides a clear way to track how the build pipeline is parallelized: you control exactly how you want your pipeline to be parallelized.

run_submodule

The run_submodule action allows you to load and run another module as a submodule. The module can be executed on the main thread of VyBuild (blocking the build pipeline until it completes) or as another process (using fork). If block_if_fail is true and the submodule is executed in another process, VyBuild will only stop the build pipeline if the fork failed, not if the execution of the submodule fail. The pipeline continue on the main process and the submodule process continue until the submodule execution end, without any synchronization between the two processes once the fork succeded. If block_if_fail is true and the submodule is executed in the main process, VyBuild will stop the build pipeline immediatly. If block_if_fail is false, the build pipeline will continue no matter the returned status of the submodule execution.

Syntax:

    {
      "action":"run_submodule",
      "args":{
        "file_variables":...,
        "file":...,
        "wait":...,
        "block_if_fail":...
      }
    }

Arguments:

Name	Type	Possible values	Role
file_variables	boolean	true or false	Indicates if variables replacement should be performed in the file string.
file	string	any valid path	Indicate the path of the submodule to execute. If the path is relative, tt must be accessible from the root folder of the current module.
wait	boolean	true or false	If false, the submodule is executed in another process. If true, the submodule is executed on the main process.
block_if_fail	boolean	true or false	If true, any error in the submodule (except others errors than fork failure if wait is false) will stop the build pipeline.

run_actions_parallel

The run_actions_parallel action allow for multithreaded actions execution without loading another module. All actions are placed in a queue and a certain amount of threads determined by this formula min(number_of_actions,number_of_physical_threads) is created. This action doesn't offer any synchronization or order of execution guarantee. It just block the main pipeline until all parallel actions are finish. However, if any action executed in a thread return anything other than a success, the entire build pipeline is stopped, including others threads.

The following actions can't be run in parallel by run_actions_parallel: run_submodule, run_actions_parallel, vyld_compilation, run_submodule_if, run_submodule_parallel, compile_multiple_cpp.

Syntax:

    {
      "action":"run_actions_parallel",
      "args":{
        "actions":[
          ...
        ]
      }
    }

This action only has one argument: the list of actions to run in parallel.

run_submodule_parallel

The run_submodule_parallel action allows for multithreaded submodules execution. All submodules are placed in a queue and a certain amount of threads determined by this formula min(number_of_submodules,number_of_physical_threads) is created. This action doesn't offer any synchronization or order of execution guarantee. It just blocks the main pipeline until all parallel submodules are finish. However, if any submodule executed in a thread return anything other than a success, the entire build pipeline is stopped, including others threads.

The run_submodule_parallel enforces a ban on actions that could start an uncontrolled thread creation chain. That mean that any submodule ran using this action will fail (and stop the build pipeline) as soon as it encounters one of the following action: run_submodule, run_actions_parallel, vyld_compilation, run_submodule_if, run_submodule_parallel, compile_multiple_cpp.

Each thread has its own working directory changed to the root folder of the submodule it executes.

Syntax:

    {
      "action":"run_submodule_parallel",
      "args":{
        "submodules_variables":...,
        "submodules":[
          ...
        ]
      }
    }

Arguments:

Name	Type	Possible values	Role
submodules_variables	boolean	true or false	Indicates if variables replacement should be performed in the submodules list of strings.
submodules	list of strings	any valid path	Indicate the path of the submodule to execute. If the path is relative, it must be accessible from the root folder of the current module.

FAT32 related actions

VyBuild integrates its own FAT32 generator, that can be extremely helpful for projects that needs to generate FAT32 images, especially if you are building your own disk image. Because VyBuild is still in early development, it can only generate FAT32 images with these limitations:

• No boot code compatibility (it will be set to 0)
• Custom enforced OEM name and volume label
• No FAT12 or FAT16 generation
• 512 bytes per sector, 1 sector per cluster, 32 reserved sectors count, 2 FATs
• Limited to short files names, lowercases letter are automatically transformed to uppercases letters
• Directory are only allocated one cluster and can't be extended/expanded later, that mean that a directory can't contain more than 14 files or subfolder. Technically, it's 16 but we automatically add the . and .. entries. This is a limitation we planned to raise in the future
• Empty files in the image need to be created from an existing empty file on the disk
• Once a folder has been created or a file has been flashed, it can no longer be edited, copied, moved, expanded or deleted. But that also implies that files are always contiguous on the FAT32 images
• Each folder in a path must be created one by one
• FSInfo structure aren't completed with actuals useful information

This implementation doesn't have the objective of being a complete FAT32 driver but rather, a simpler alternative to easily generate FAT32 image directly in the build pipeline, avoiding the complex Linux tricks and need for root permissions usages. But, it's still sufficient for generating simple ESP partition.

Warning: while there is no global lock on the FAT32 image generator/on each FAT32 image, there is also no ban of FAT32 related actions in multithreaded part of the build pipeline. This mean that the user is fully responsible that no more than one part of the build pipeline use the FAT32 image generator at the same time.

create_fat32

The create_fat32 action will create an empty FAT32 image in RAM and will initialize all FAT32 structures. The value in key is used to identify the image for later actions on this image. The size must be provided in bytes and must be a multiple of 512. The key must not be empty.

Syntax:

    {
      "action":"create_fat32",
      "args":{
        "key_variables":...,
        "key":...,
        "size":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the FAT32 image.
size	integer	any integer multiple of 512 except 0	Indicate the size in bytes of the FAT32 image. It should be at least 32MB to be recognized as a valid FAT32.

create_folder_fat32

The create_folder_fat32 action will create a folder at the indicated path inside the FAT32 image referenced by the provided key. The path must start with /, indicating the root folder of the FAT32 image. All of the folder names in the path should be in uppercases letters and not longer than 8 characters. The last folder name is the name of the folder to create. The rest of the path must already exist inside the FAT32 image.

Example: /EFI/BOOT: Here BOOT is the name of the folder to create and EFI must already exist as a folder inside the root folder.

Syntax:

    {
      "action":"create_folder_fat32",
      "args":{
        "key_variables":...,
        "key":...,
        "full_path_variables":...,
        "full_path":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the FAT32 image.
full_path_variables	boolean	true or false	Indicates if variables replacement should be performed in the full_path string.
full_path	string	any valid path in the FAT32 image	Indicate the full path of the folder to create.

create_file_fat32

The create_file_fat32 action will create a file at the indicated path inside the FAT32 image referenced by the provided key. All the folders in the full path must already exist. The last name in the full path is interpreted as the file name and must satisfies the 8.3 file name format (if file name isn't formated as expected, VyBuild will fail the action and therefore the build pipeline). The content of the file is read from a file on disk and the file size is automatically determined by the file size of the source file on disk. Source file must be an existing regular file.

Syntax:

    {
      "action":"create_file_fat32",
      "args":{
        "key_variables":...,
        "key":...,
        "source_path_variables":...,
        "source_path":...,
        "full_path_variables":...,
        "full_path":...,
        "is_vftm_target":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the FAT32 image.
source_path_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_path string.
source_path	string	any valid path	Indicate the path of the source file on disk.
full_path_variables	boolean	true or false	Indicates if variables replacement should be performed in the full_path string.
full_path	string	any valid path in the FAT32 image	Indicate the full path of the file to create.
is_vftm_target	boolean	true or false	Indicate if this file is the VFTM target. If you don't know what VFTM is, set this flag to false and ignore it.

export_fat32

The export_fat32 action exports the FAT32 image referenced by the provided key to a specified file on the disk. disk_guid_var and efi_part_unique_guid_var are two optionals arguments and must be provided only if generate_vftm is true. If you don't know what VFTM is, set the generate_vftm flag to false and ignore it.

Syntax:

    {
      "action":"export_fat32",
      "args":{
        "key_variables":...,
        "key":...,
        "export_path_variables":...,
        "export_path":...,
        "generate_vftm":...,
        "disk_guid_var":...,
        "efi_part_unique_guid_var":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the FAT32 image.
export_path_variables	boolean	true or false	Indicates if variables replacement should be performed in the export_path string.
export_path	string	any valid path	Indicate the path of the file in which the FAT32 image should be exported.
generate_vftm	boolean	true or false	Indicates if the VFTM should be generated and integrated to the exported image. If you don't know what VFTM is, set this flag to false and ignore it.
disk_guid_var	string	any variable name without the $ symbols	Indicate the variable that contain the disk GUID. Only mandatory if generate_vftm is true. If you don't know what VFTM is, just don't set this field.
efi_part_unique_guid_var	string	any variable name without the $ symbols	Indicate the variable that contain the ESP unique GUID. Only mandatory if generate_vftm is true. If you don't know what VFTM is, just don't set this field.

Disk related actions

VyBuild integrates its own disk images generator. Because VyBuild is still in early development, it can only generate disk images with these limitations:

• No MBR support at all
• 128 partition entries maximum
• 128 bytes per partition entries
• GUID must be provided as UUID v4 syntax. All 16 bytes of data are considered given as Big Endian, and so the first 8 bytes of data will be converted to Little Endian to comply with the GPT specification
• Fixed LBA size of 512 bytes
• No partition alignment other than LBA size

This implementation doesn't have the objective of being a complete disk driver but rather, a simpler alternative to easily generate disk image directly in the build pipeline, avoiding the complex Linux tricks and need for root permissions usages. But, it's still sufficient for generating simple disk images, even with custom partitions.

Warning: while there is no global lock on the disk image generator/on each disk image, there is also no ban of disk related actions in multithreaded part of the build pipeline. This mean that the user is fully responsible that no more than one part of the build pipeline use the disk image generator at the same time.

create_disk

The create_disk action will create an empty disk image in RAM and will initialize all GPT structures. The value in key is used to identify the disk image for later actions on this image. The size must be provided in bytes and must be a multiple of 512. The key and disk GUID must not be empty.

Syntax:

    {
      "action":"create_disk",
      "args":{
        "key_variables":...,
        "key":...,
        "size":...,
        "disk_guid_variables":...,
        "disk_guid":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the disk image.
size	integer	any integer multiple of 512 except 0	Indicate the size in bytes of the disk image.
disk_guid_variables	boolean	true or false	Indicates if variables replacement should be performed in the disk_guid string.
disk_guid	string	any valid UUID v4	Indicate a GUID for the disk under the form of a UUID v4 Big Endian.

add_partition_disk

The add_partition_disk action will create a new partition on the disk image referenced by the provided key. The key must be a valid disk image key. The type GUID, unique GUID and name must not be empty. The source file path must lead to a valid partition image stored in an existing regular file on disk.

Syntax:

    {
      "action":"add_partition_disk",
      "args":{
        "key_variables":...,
        "key":...,
        "type_guid_variables":...,
        "type_guid":...,
        "unique_guid_variables":...,
        "unique_guid":...,
        "name_variables":...,
        "name":...,
        "source_file_variables":...,
        "source_file":...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the disk image.
type_guid_variables	boolean	true or false	Indicates if variables replacement should be performed in the type_guid string.
type_guid	string	any valid UUID v4	Indicate a type GUID for the partition under the form of a UUID v4 Big Endian.
unique_guid_variables	boolean	true or false	Indicates if variables replacement should be performed in the unique_guid string.
unique_guid	string	any valid UUID v4	Indicate a unique GUID for the partition under the form of a UUID v4 Big Endian.
name_variables	boolean	true or false	Indicates if variables replacement should be performed in the name string.
name	string	anything except empty string	Indicate a name for the partition. It's recommanded only putting ASCII characters in this field, as VyBuild will convert in a very simple manner into UTF-16LE: simply casting each char into uint16_t
source_file_variables	boolean	true or false	Indicates if variables replacement should be performed in the source_file string.
source_file	string	any valid path	Indicate the path of the partition image on disk.

export_disk

The export_disk action exports the disk image referenced by the provided key to a specified file on the disk.

Syntax:

    {
      "action":"export_disk",
      "args":{
        "key_variables":...,
        "key":...,
        "export_path_variables":...,
        "export_path":"...
      }
    }

Arguments:

Name	Type	Possible values	Role
key_variables	boolean	true or false	Indicates if variables replacement should be performed in the key string.
key	string	anything except empty string	Indicate an identifier for the disk image.
export_path_variables	boolean	true or false	Indicates if variables replacement should be performed in the export_path string.
export_path	string	any valid path	Indicate the path of the file in which the disk image should be exported.