PastaBoy12345/vystem-b
2
0
Fork 0
forked from lolo859/vystem
Code Pull Requests Activity
Files
d238606b755549cfa1bb713cd68ca05d6f7a8395
vystem-b/docs/vybuild/actions.md
lolo859 d238606b75 Vystem 0.2
2026-05-27 19:34:54 +02:00

1071 lines
56 KiB
Markdown
Raw Blame History

# Actions
## Introduction
Actions are atomic blocks of the pipeline. They are used for all sorts of things such as compiling, filesystem manipulation, control flow, cache management and command execution. Please note that things related to VFTM aren't documented in this documentation.
## Syntax
Actions are listed in the `actions` field of each module. They are executed in the order they are specified, without any automatic reordering.
Actions all follow at their cores the same syntax:
``` json
{
"action":"...",
"args":{
...
}
}
```
The `action` field specify the name of the action to execute and the `args` field specify all sorts of arguments, specific to each actions.
## List of all available actions
### Commands related actions
**Note:** There is, for the moment, no action capable of combining the saving of output of a command with a working directory for example
#### `run_command_wait`
The `run_command_wait` action allows you to run a command and wait for it to finish. If the command fail, the action and therefore the entire module fail, interrupting the build process. It can be used in multithreaded build pipelines without restriction.
Syntax:
``` json
{
"action":"run_command_wait",
"args":{
"command_variables":...,
"command":[
...
],
"show_output":...,
"success_status":[
...
],
"ignore_success_status":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`command_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `command` argument.
`command` | list of strings | anything | The list of arguments that will be passed to the `execvp` function. The first element is considered as the name of the executable, and should be either accessible in PATH or from the indicated `root_folder` of the module.
`show_output` | string | `live`, `on_failure` or `silent` | Indicate how the output of the command should be shown. Uses `live` if your command require user input in the console. Failure detection is based on output codes of the processus.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
#### `run_command_wait_dir`
The `run_command_wait_dir` action behave exactly as the `run_command_wait` action, except it allows you to provide a path in which VyBuild will `chdir` before executing the command. It can be used in multithreaded build pipelines without restriction.
Syntax:
``` json
{
"action":"run_command_wait_dir",
"args":{
"command_variables":...,
"command":[
...
],
"success_status":[
...
],
"ignore_success_status":...,
"show_output":...,
"working_dir_variables":...,
"working_dir":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`command_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `command` argument.
`command` | list of strings | anything | The list of arguments that will be passed to the `execvp` function. The first element is considered as the name of the executable, and should be either accessible in PATH or from the indicated `working_dir` argument.
`show_output` | string | `live`, `on_failure` or `silent` | Indicate how the output of the command should be shown. Uses `live` if your command require user input in the console. Failure detection is based on output codes of the processus.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
`working_dir_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `working_dir` argument.
`working_dir` | string | any valid path | Indicate the working directory in which VyBuild should `chdir` before executing the command.
#### `save_command_content`
The `save_command_content` behave exactly like the `run_command_wait` action, except instead of showing the output, it save it into a file. It can be used in multithreaded build pipelines without restriction.
Syntax:
``` json
{
"action":"save_command_content",
"args":{
"command_variables":...,
"command":[
...
],
"success_status":[
...
],
"ignore_success_status":...,
"command_output_file_variables":...,
"command_output_file":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`command_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `command` argument.
`command` | list of strings | anything | The list of arguments that will be passed to the `execvp` function. The first element is considered as the name of the executable, and should be either accessible in PATH or from the indicated `working_dir` argument.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
`command_output_file_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `command_output_file` argument.
`command_output_file` | string | any valid path | Indicate the file in which the output of the command should be saved. The previous content of the file is erased. The parent directory of the file must exists.
#### `run_command_str_wait`
The `run_command_str_wait` behave exactly like the `run_command_wait` action, except that instead of providing the command in a list of arguments, they are provided as a single string that is then processed with `wordexp`, which parse them nearly exactly as a shell would do. The variables replacement is ran before the `wordexp` call. It can be used in multithreaded build pipelines without restriction.
``` json
{
"action":"run_command_str_wait",
"args":{
"command_variables":...,
"command":...,
"success_status":[
...
],
"ignore_success_status":...,
"show_output":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`command_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `command` argument.
`command` | string | anything | The string that will be parsed by `wordexp` and then fed into `execvp`. The first element of the parsing result is considered as the name of the executable, and should be either accessible in PATH or from the indicated `root_folder` of the module.
`show_output` | string | `live`, `on_failure` or `silent` | Indicate how the output of the command should be shown. Uses `live` if your command require user input in the console. Failure detection is based on output codes of the processus.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
### Compilation related actions
The `compile_one_*` action are safe and authorized in multithreaded pipelines, but `compile_multiple_cpp`and `vyld_compilation` can only be used in parts of pipelines that aren't in a multithreaded phase.
#### `compile_one_asm`
The `compile_one_asm` action, as its name imply, compile a single assembly file. It is intended for simple and single compilation operation. It can be used in multithreaded build pipelines without restriction.
Syntax:
``` json
{
"action":"compile_one_asm",
"args":{
"source_variables":...,
"source":...,
"assembler_variables":...,
"assembler":...,
"pre_arguments_variables":...,
"pre_arguments":[
...
],
"post_arguments_variables":...,
"post_arguments":[
...
],
"output_file_variables":...,
"output_file":...,
"success_status":[
...
],
"ignore_success_status":...,
"cache_authorized":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`source_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `source` argument.
`source` | string | any valid path | Provide the path (path relative to the `root_folder` field of the module or absolute path) to the file to compile.
`assembler_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `assembler` argument.
`assembler` | string | any valid executable name | Provide the executable name of the assembler to use for this action. It must be an executable accessible in PATH.
`pre_arguments_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `pre_arguments` argument.
`pre_arguments` | list of strings | anything | Indicate the pre arguments to fed into `execvp`
`post_arguments_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `post_arguments` argument.
`post_arguments` | list of strings | anything | Indicate the post arguments to fed into `execvp`
`output_file_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `output_file` argument.
`output_file` | string | any valid path | A path relative to `root_folder` that indicate to VyBuild which file to put into cache or to extract from cache. The parent folder must already exists.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
`cache_authorized` | boolean | `true` or `false` | Allows VyBuild to extract output file from cache if the compilation context matchs, and to store the output file into the cache
The compilation context is used to generate a key for the cache. For this action, the compilation context contain: source file content, assembler binary hash, assembler path, assembler pre and post arguments.
The command is executed using `execvp`. The arguments are concatenated as follows:
1. Resolved compiler path.
2. Pre-arguments (in order).
3. Source file path.
4. Post-arguments (in order).
The user is responsible to provide any `-o` assembler argument where the assembler require it. This is this limitation that indicate which arguments to put into pre and post arguments.
#### `compile_one_cpp`
The `compile_one_cpp` action compiles a single C or C++ source file. It is designed to handle dependency tracking through a dedicated header discovery command, ensuring that the cache is invalidated if any included header file is modified. It can be used in multithreaded build pipelines without restriction.
Syntax:
```json
{
"action":"compile_one_cpp",
"args":{
"source_variables":...,
"source":...,
"compiler_variables":...,
"compiler":...,
"pre_arguments_variables":...,
"pre_arguments":[
...
],
"post_arguments_variables":...,
"post_arguments":[
...
],
"output_file_variables":...,
"output_file":...,
"success_status":[
...
],
"ignore_success_status":...,
"cache_authorized":...,
"headers_command_variables":...,
"headers_command":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`source_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source` argument.
`source` | string | any valid path | Provides the path (relative to `root_folder` or absolute) to the C/C++ file to compile.
`compiler_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `compiler` argument.
`compiler` | string | any valid executable name | Provides the executable name of the compiler (e.g., `gcc`, `g++`, `clang`). Must be accessible in PATH.
`pre_arguments_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `pre_arguments` argument.
`pre_arguments` | list of strings | anything | Arguments to be passed to the compiler before the source file.
`post_arguments_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `post_arguments` argument.
`post_arguments` | list of strings | anything | Arguments to be passed to the compiler after the source file (for example `-o` and the output path).
`output_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `output_file` argument.
`output_file` | string | any valid path | A path relative to `root_folder` that indicate to VyBuild which file to put into cache or to extract from cache. The parent folder must already exists.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
`cache_authorized` | boolean | `true` or `false` | Allows VyBuild to use the cache for this action.
`headers_command_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `headers_command` argument.
`headers_command` | string | shell command | A command string used to extract header dependencies.
To ensure perfect cache integrity, VyBuild must know which headers are included by the source file. The `headers_command` field is executed in a shell. VyBuild automatically provides two internal variables for this command:
- `%sourcefile%`: The path to the source file being compiled
- `%dfile%`: A temporary path where VyBuild expects the dependency information to be written (usually in Makefile format)
Example for GCC:
`"headers_command": "gcc -MMD -MF %dfile% -E %sourcefile% -o /dev/null"`
The compilation context used to generate the cache key for this action includes:
- The content of the source file
- The content of all discovered header files (using `headers_command`)
- The compiler binary hash and its system path
- All pre and post arguments
The command is executed using `execvp`. The arguments are concatenated as follows:
1. Resolved compiler path.
2. Pre-arguments (in order).
3. Source file path.
4. Post-arguments (in order).
#### `compile_multiple_cpp`
The `compile_multiple_cpp` action allows for the mass compilation of C and C++ source files. It is the multithreaded engine of VyBuild, capable of handling large sets of files either by explicit listing or via a powerful path completion system. This action automatically manages dependency tracking and caching for each individual file.
Syntax:
``` json
{
"action":"compile_multiple_cpp",
"args":{
"source_files_path_completion":...,
"path_completion_source_ext":...,
"path_completion_output_ext":...,
"source_files_variables":...,
"source_files":[
...
],
"compiler_variables":...,
"compiler":...,
"pre_arguments_variables":...,
"pre_arguments":[
...
],
"post_arguments_variables":...,
"post_arguments":[
...
],
"output_files_variables":...,
"output_files":[
...
],
"success_status":[
...
],
"ignore_success_status":...,
"cache_authorized":...,
"headers_command_variables":...,
"headers_command":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`source_files_path_completion` | boolean | `true` or `false` | Enables the automatic discovery of source files using wildcards.
`path_completion_source_ext` | string | e.g., `.cpp` | The extension of source files to look for during path completion.
`path_completion_output_ext` | string | e.g., `.o` | The extension to apply to the generated output files.
`source_files_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_files` list
`source_files` | list of strings | paths | List of source files or directory patterns (if path completion is enabled).
`compiler_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `compiler` argument.
`compiler` | string | any valid executable name | Provides the executable name of the compiler (e.g., `gcc`, `g++`, `clang`). Must be accessible in PATH.
`pre_arguments_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `pre_arguments` argument.
`pre_arguments` | list of strings | anything | Arguments to be passed to the compiler before the source file.
`post_arguments_variables` | boolean | `true` or `false` | Indicate if variables replacement should be done in the `post_arguments` argument.
`post_arguments` | list of strings | anything | Arguments to be passed to the compiler after the source file (for example `-o` and the output path).
`output_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `output_file` argument.
`output_file` | string | any valid path | A path relative to `root_folder` that indicate to VyBuild which file to put into cache or to extract from cache. The parent folder must already exists. **Must be empty** if using path completion.
`success_status` | list of integers | any integers | Indicate the integers that should be considered as a success of the command. If the output code of the command isn't in this list, the command is considered as failed.
`ignore_success_status` | boolean | `true` or `false` | If this field is `true`, no output code will fail the action.
`cache_authorized` | boolean | `true` or `false` | Allows VyBuild to use the cache for this action.
`headers_command_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `headers_command` argument.
`headers_command` | string | shell command | A command string used to extract header dependencies.
VyBuild features an advanced path completion system to avoid manual file listing. It is triggered when `source_files_path_completion` is `true`.
Syntax:
- `path/to/dir/*` : VyBuild will look for all files with `path_completion_source_ext` in that specific directory.
- `path/to/dir/**` : VyBuild will recursively search for all matching files in that directory and all its subdirectories.
When path completion is used, the `output_files` field **must be empty**. VyBuild will automatically generate the output paths by taking the source file path and replacing the source extension with the `path_completion_output_ext`.
Variables replacement is performed on the `source_files` strings before the path completion engine scans the filesystem.
To ensure perfect cache integrity, VyBuild must know which headers are included by the source file. The `headers_command` field is executed in a shell. VyBuild automatically provides two internal variables for this command:
- `%sourcefile%`: The path to the source file being compiled
- `%dfile%`: A temporary path where VyBuild expects the dependency information to be written
Example for GCC:
`"headers_command": "gcc -MMD -MF %dfile% -E %sourcefile% -o /dev/null"`
For `pre_arguments` and `post_arguments`, VyBuild provides the following variable:
- `%outputfile%`: The path of the current output file (either explicitly provided in `output_files` or auto-generated via path completion). This is typically used with the `-o` flag.
The compilation context used to generate the cache key for **each file individually** for this action includes:
- The content of the source file
- The content of all discovered header files (using `headers_command`) for this file
- The compiler binary hash and its system path
- The specific pre and post arguments (including the resolved `%outputfile%`)
This action is natively multithreaded. VyBuild will distribute the compilation of the file set across the available CPU threads. The command of compilation for each file is executed using `execvp`. The order of execution within the set is not guaranteed, but the arguments are strictly concatenated as follows:
1. Resolved compiler path.
2. Pre-arguments (in order, local to each file).
3. Source file path (local to each file).
4. Post-arguments (in order, local to each file).
#### `vyld_compilation`
The `vyld_compilation` action is a high-level orchestration tool designed to produce a final VYX executable. It automates the entire pipeline: compiling C/C++ and Assembly files into temporary objects, managing a per-object cache, generating appropriate linker scripts, and performing the final link and VYX header injection.
Syntax:
``` json
{
"action":"vyld_compilation",
"args":{
"source_files_path_completion":...,
"source_files_variables":...,
"source_files":[
...
],
"compiler_variables":...,
"compiler":...,
"source_asm_path_completion":...,
"source_asm_variables":...,
"source_asm":[
...
],
"assembler_variables":...,
"assembler":...,
"linker_variables":...,
"linker":...,
"compiler_flags_variables":...,
"compiler_flags":...,
"output_file_variables":...,
"output_file":...,
"block_if_fail":...,
"object_files_cache_authorized":...,
"headers_command_variables":...,
"headers_command":...,
"payloads":[
{
"source_file_variables":...,
"source_file":...,
"is_executable":...,
"is_writable":...,
"expected_va":...
}
]
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|-----------------|-----------------------
`source_files_path_completion` | boolean | `true` or `false` | Enables the automatic discovery of `.c` source files using wildcards.
`source_files_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_files` list
`source_files` | list of strings | paths | List of source files or directory patterns (if path completion is enabled).
`compiler_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `compiler` argument.
`compiler` | string | any valid executable name | Provides the executable name of the compiler (e.g., `gcc`, `g++`, `clang`). Must be accessible in PATH.
`source_asm_path_completion` | boolean | `true` or `false` | Enables the automatic discovery of `.asm` source files using wildcards.
`source_asm_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_asm` list
`source_asm` | list of strings | paths | List of source assembly files or directory patterns (if path completion is enabled).
`assembler_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `assembler` argument.
`assembler` | string | any valid executable name | Provides the executable name of the assembler (e.g., `nasm`). Must be accessible in PATH and have nasm-like argument syntax.
`linker_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `linker` argument.
`linker` | string | any valid executable name | Provides the executable name of the linker (e.g., `ld`). Must be accessible in PATH and be compatible with `ld` linker script.
`compiler_flags_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `compiler_flags` argument.
`compiler_flags` | string | any valid compiler arguments | Custom flags passed to the C compiler during the object generation phase.
`output_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `output_file` argument.
`output_file` | string | any valid path | A path relative to `root_folder` that indicate where to put the produced VYX executable. The parent folder must already exists.
`block_if_fail` | boolean | `true` or `false` |If `true`, any error in the pipeline stops the entire action
`object_files_cache_authorized` | boolean | `true` or `false` | Enables caching at the individual object file level (not the final binary).
`headers_command_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `headers_command` argument.
`headers_command` | string | shell command | A command string used to extract header dependencies.
Each entry in the `payloads` list defines an external resource to be embedded in the VYX file:
Name | Type | Possible values | Role
-----|------|------------------|-----
`source_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_file` string.
`source_file` | string | any valid path | Path to the raw data file to be integrated.
`is_executable` | boolean | `true` or `false` | Sets the executable flag in the VYX payload header.
`is_writable` | boolean | `true` or `false` | Sets the writable flag in the VYX payload header.
`expected_va` | string | string formatted as `0xNNNNNNNNNNNNNNNN` where each `N` must be a valid hexadecimal number | The specific virtual address where this payload must be mapped.
When `vyld_compilation` is triggered, VyBuild call to its Vyld integration, which follows a rigorous multi-stage process to ensure the resulting binary strictly adheres to the VYX specifications:
**1. Object Compilation Phase**
VyBuild first processes all C/C++ source files with multithreading. Each file is compiled into a temporary object file. If caching is enabled, Vyld uses the VyBuild cache to checks for each individual object. The cache key includes the source content, discovered headers, compiler hash, and flags. Object files are stored in a managed temporary directory and are not exposed to the user after the build.
**2. Assembly Phase**
All assembly files are then processed. Unlike C compilation, this phase is currently single-threaded. The assembler is invoked with NASM-compatible arguments to produce the remaining object files.
**3. Automated Linker Script Generation**
Instead of requiring a manual linker script, `vyld_compilation` dynamically generates a script tailored for the VYX format. It enforces:
- The specific section order: `.text` -> `.data` -> `.rodata` -> `.bss`
- Memory alignment on 4 kilobytes boundaries (0x1000)
- Correct mapping of the base addresses generated in the build pipeline
**4. Final Linking & VYX Wrapping**
The linker is called using the generated script to produce a temporary ELF binary. VyBuild then:
- Extracts the raw binary data for each section
- Calculates the required padding to ensure each section starts on a new page
- Header Injection: Constructs the VYX header
- Payload Integration: Appends each specified payload at the end of the file, preceded by the payload header
To ensure perfect cache integrity, VyBuild must know which headers are included by the source file. The `headers_command` field is executed in a shell. VyBuild automatically provides two internal variables for this command:
- `%sourcefile%`: The path to the source file being compiled
- `%dfile%`: A temporary path where VyBuild expects the dependency information to be written
Example for GCC:
`"headers_command": "gcc -MMD -MF %dfile% -E %sourcefile% -o /dev/null"`
### Filesystem related actions
All filesystem related actions are allowed in multithreaded parts of the build pipeline. However, its the responsability of the user to ensure that the effect of a filesystem action in a multithreaded part of the pipeline doesn't collide with another action.
Please keep in mind that not all filesystems manipulations are possible yet.
#### `ensure_folder_existence`
The `ensure_folder_existence` action will check if a folder exists. If the folder doesn't exist, it will be created. This action can create multiple folders at once.
Syntax:
``` json
{
"action":"ensure_folder_existence",
"args":{
"path_variables":...,
"path":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `path` string.
`path` | string | any valid path | Indicate the path of the folder to ensure the existence of.
#### `move_file`
The `move_file` action move a file or a folder to another folder. The destination path must be a folder and exists before the action. The original file/folder keeps its name.
Syntax:
``` json
{
"action":"move_file",
"args":{
"source_file_variables":...,
"source_file":"...",
"destination_folder_variables":...,
"destination_folder":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`source_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_file` string.
`source_file` | string | any valid path | Indicate the path of the source file/folder.
`destination_folder_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `destination_folder` string.
`destination_folder` | string | any valid path | Indicate the path of the destination folder.
**Warning:** while this action is theoretically capable of moving folders, it is not intended or designed for that.
#### `copy_folder`
The `copy_folder` action copy a folder into another folder. Both `source_folder` and `destination_folder` must be path to existing folders. It is a recursive copy that overwrite existing files/folders.
Syntax:
``` json
{
"action":"copy_folder",
"args":{
"source_folder_variables":...,
"source_folder":...,
"destination_folder_variables":...,
"destination_folder":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`source_folder_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_folder` string.
`source_folder` | string | any valid path | Indicate the path of the source folder to copy.
`destination_folder_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `destination_folder` string.
`destination_folder` | string | any valid path | Indicate the path of the destination folder.
#### `ensure_folder_reset`
The `ensure_folder_reset` action will check if a folder exists. If the folder doesn't exist, it will be created. If the folder exists, it will get removed (with all its content) and recreated. This action can create multiple folders at once.
Syntax:
``` json
{
"action":"ensure_folder_reset",
"args":{
"path_variables":...,
"path":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `path` string.
`path` | string | any valid path | Indicate the path of the folder to ensure the reset of.
#### `copy_file`
The `copy_file` action copy a file into a folder. `source_file` must be a path to an existing file, and `destination_folder` must be an existing folder. It will overwrite the file if a file with the same name in the destination folder already exists.
Syntax:
``` json
{
"action":"copy_file",
"args":{
"source_file_variables":...,
"source_file":...,
"destination_folder_variables":...,
"destination_folder":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`source_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_file` string.
`source_file` | string | any valid path | Indicate the path of the source file to copy.
`destination_folder_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `destination_folder` string.
`destination_folder` | string | any valid path | Indicate the path of the destination folder.
### Cache related actions
The cache being thread-safe, these two actions can be used safely in multithreaded part of any build pipelines.
#### `extract_from_cache`
The `extract_from_cache` action will extract a file from the cache based on the provided key. If a file already exists at the path indicated by `output_file`, it will overwrite it. If the cache doesn't contain anything associated to that key, the action fails silently without stopping the build pipeline.
Syntax:
``` json
{
"action":"extract_from_cache",
"args":{
"target_variables":...,
"target":...,
"output_file_variables":...,
"output_file":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`target_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `target` string.
`target` | string | anything | Indicate the key to use to retrieve the cached object.
`output_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `output_file` string.
`output_file` | string | any valid path | Indicate the path of the destination file.
#### `store_in_cache`
The `store_in_cache` action store a file in cache. If the cache already contains the provided key, the `overwrite_allowed` argument will specify if the cache object is overwritten or not. If `overwrite_allowed` is `false` and the cache already contains the key, the action will fail silently.
Syntax:
``` json
{
"action":"store_in_cache",
"args":{
"key_variables":...,
"key":...,
"file_variables":...,
"file":...,
"overwrite_allowed":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything | Indicate the key to use to store the file.
`file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `file` string.
`file` | string | any valid path | Indicate the path of the file to store in cache.
`overwrite_allowed` | boolean | `true` or `false` | Indicates if the overwrite of already-existing cache object is allowed.
### Control flow related actions
VyBuild provides a way to build dynamic build pipelines using control flow actions. These actions are the following:
- `run_actions_if`: contains two arguments, `condition` and `actions`. `actions` is a list of actions to execute if the evaluated condition is satisfied.
- `run_actions_if_else`: contains three arguments, `condition`, `actions_if` and `actions_else`. `actions_if` is a list of actions executed if the condition is satisfied. Otherwise, `actions_else` is a list of actions executed if the condition isn't satisfied.
- `run_submodule_if`: this action is the same as `run_submodule`, except it contains a condition object that will determine if the submodule is executed or not, based on the satisfaction of the condition.
These two actions enforce the rules on multithreading usage depending on where they are executed.
#### Conditions
Conditions are a special kind of JSON objects in VyBuild. They have the following syntax:
``` json
"condition":{
"condition_variables":...,
"condition":...,
"condition_args_variables":...,
"condition_args":[
...
]
}
```
`condition_variables` and `condition_args_variables` are booleans specifying if the variables replacement process should be performed on their respective fields. `condition` is a string and `condition_args` is a list of strings.
Here are described the available conditions with their arguments and behaviour:
Name | Arguments | Behaviour
-----|-----------|----------
`folder_exists` | Expect one argument: the path of the folder | Returns `true` if the folder exists
`folder_not_exists` | Expect one argument: the path of the folder | Returns `true` if the folder doesn't exist
`folder_empty` | Expect one argument: the path of the folder | Returns `true` if the folder is empty
`folder_not_empty` | Expect one argument: the path of the folder | Returns `true` if the folder isn't empty
`cache_contains` | Expect one argument: the key of the cache object | Returns `true` if the cache contains an object associated with that key
`cache_not_contains` | Expect one argument: the key of the cache object | Returns `true` if the cache doesn't contain an object associated with that key
`var_exists` | Expect one argument: the name of the variable | Returns `true` if the variable exists
`var_not_exists` | Expect one argument: the name of the variable | Returns `true` if the variable doesn't exist
`var_equals` | Expect two arguments in order: the name of the variable and the value | Returns `true` if the variable is equal to the value. Stop the build pipeline if variable doesn't exist
`var_not_equals` | Expect two arguments in order: the name of the variable and the value | Returns `true` if the variable isn't equal to the value. Stop the build pipeline if variable doesn't exist.
#### `exit_current_module`
The `exit_current_module` action is a special action that can only be executed in a list of actions from a `run_actions_if` or `run_actions_if_else` action. It exits the current module, and stop the build pipeline if the current module is the root module.
Syntax:
``` json
{
"action":"exit_current_module",
"args":{
"is_fail":...
}
}
```
The `is_fail` argument allows to completely stop the build pipeline even if the current module isn't the root module.
### Multithreading related actions
VyBuild provides a clear way to track how the build pipeline is parallelized: you control exactly how you want your pipeline to be parallelized.
#### `run_submodule`
The `run_submodule` action allows you to load and run another module as a submodule. The module can be executed on the main thread of VyBuild (blocking the build pipeline until it completes) or as another process (using `fork`). If `block_if_fail` is `true` and the submodule is executed in another process, VyBuild will only stop the build pipeline if the fork failed, not if the execution of the submodule fail. The pipeline continue on the main process and the submodule process continue until the submodule execution end, without any synchronization between the two processes once the fork succeded. If `block_if_fail` is `true` and the submodule is executed in the main process, VyBuild will stop the build pipeline immediatly. If `block_if_fail` is `false`, the build pipeline will continue no matter the returned status of the submodule execution.
Syntax:
``` json
{
"action":"run_submodule",
"args":{
"file_variables":...,
"file":...,
"wait":...,
"block_if_fail":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `file` string.
`file` | string | any valid path | Indicate the path of the submodule to execute. If the path is relative, tt must be accessible from the root folder of the current module.
`wait` | boolean | `true` or `false` | If `false`, the submodule is executed in another process. If `true`, the submodule is executed on the main process.
`block_if_fail` | boolean | `true` or `false` | If `true`, any error in the submodule (except others errors than fork failure if `wait` is `false`) will stop the build pipeline.
#### `run_actions_parallel`
The `run_actions_parallel` action allow for multithreaded actions execution without loading another module. All actions are placed in a queue and a certain amount of threads determined by this formula `min(number_of_actions,number_of_physical_threads)` is created. This action doesn't offer any synchronization or order of execution guarantee. It just block the main pipeline until all parallel actions are finish. However, if any action executed in a thread return anything other than a success, the entire build pipeline is stopped, including others threads.
The following actions can't be run in parallel by `run_actions_parallel`: `run_submodule`, `run_actions_parallel`, `vyld_compilation`, `run_submodule_if`, `run_submodule_parallel`, `compile_multiple_cpp`.
Syntax:
``` json
{
"action":"run_actions_parallel",
"args":{
"actions":[
...
]
}
}
```
This action only has one argument: the list of actions to run in parallel.
#### `run_submodule_parallel`
The `run_submodule_parallel` action allows for multithreaded submodules execution. All submodules are placed in a queue and a certain amount of threads determined by this formula `min(number_of_submodules,number_of_physical_threads)` is created. This action doesn't offer any synchronization or order of execution guarantee. It just blocks the main pipeline until all parallel submodules are finish. However, if any submodule executed in a thread return anything other than a success, the entire build pipeline is stopped, including others threads.
The `run_submodule_parallel` enforces a ban on actions that could start an uncontrolled thread creation chain. That mean that any submodule ran using this action will fail (and stop the build pipeline) as soon as it encounters one of the following action: `run_submodule`, `run_actions_parallel`, `vyld_compilation`, `run_submodule_if`, `run_submodule_parallel`, `compile_multiple_cpp`.
Each thread has its own working directory changed to the root folder of the submodule it executes.
Syntax:
``` json
{
"action":"run_submodule_parallel",
"args":{
"submodules_variables":...,
"submodules":[
...
]
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`submodules_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `submodules` list of strings.
`submodules` | list of strings | any valid path | Indicate the path of the submodule to execute. If the path is relative, it must be accessible from the root folder of the current module.
### FAT32 related actions
VyBuild integrates its own FAT32 generator, that can be extremely helpful for projects that needs to generate FAT32 images, especially if you are building your own disk image. Because VyBuild is still in early development, it can only generate FAT32 images with these limitations:
- No boot code compatibility (it will be set to 0)
- Custom enforced OEM name and volume label
- No FAT12 or FAT16 generation
- 512 bytes per sector, 1 sector per cluster, 32 reserved sectors count, 2 FATs
- Limited to short files names, lowercases letter are automatically transformed to uppercases letters
- Directory are only allocated one cluster and can't be extended/expanded later, that mean that a directory can't contain more than 14 files or subfolder. Technically, it's 16 but we automatically add the `.` and `..` entries. This is a limitation we planned to raise in the future
- Empty files in the image need to be created from an existing empty file on the disk
- Once a folder has been created or a file has been flashed, it can no longer be edited, copied, moved, expanded or deleted. But that also implies that files are always contiguous on the FAT32 images
- Each folder in a path must be created one by one
- FSInfo structure aren't completed with actuals useful information
This implementation doesn't have the objective of being a complete FAT32 driver but rather, a simpler alternative to easily generate FAT32 image directly in the build pipeline, avoiding the complex Linux tricks and need for root permissions usages. But, it's still sufficient for generating simple ESP partition.
**Warning:** while there is no global lock on the FAT32 image generator/on each FAT32 image, there is also no ban of FAT32 related actions in multithreaded part of the build pipeline. This mean that the user is fully responsible that no more than one part of the build pipeline use the FAT32 image generator at the same time.
#### `create_fat32`
The `create_fat32` action will create an empty FAT32 image in RAM and will initialize all FAT32 structures. The value in `key` is used to identify the image for later actions on this image. The size must be provided in bytes and must be a multiple of 512. The key must not be empty.
Syntax:
``` json
{
"action":"create_fat32",
"args":{
"key_variables":...,
"key":...,
"size":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the FAT32 image.
`size` | integer | any integer multiple of 512 except 0 | Indicate the size in bytes of the FAT32 image. It should be at least 32MB to be recognized as a valid FAT32.
#### `create_folder_fat32`
The `create_folder_fat32` action will create a folder at the indicated path inside the FAT32 image referenced by the provided key. The path must start with `/`, indicating the root folder of the FAT32 image. All of the folder names in the path should be in uppercases letters and not longer than 8 characters. The last folder name is the name of the folder to create. The rest of the path must already exist inside the FAT32 image.
Example: `/EFI/BOOT`: Here `BOOT` is the name of the folder to create and `EFI` must already exist as a folder inside the root folder.
Syntax:
``` json
{
"action":"create_folder_fat32",
"args":{
"key_variables":...,
"key":...,
"full_path_variables":...,
"full_path":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the FAT32 image.
`full_path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `full_path` string.
`full_path` | string | any valid path in the FAT32 image | Indicate the full path of the folder to create.
#### `create_file_fat32`
The `create_file_fat32` action will create a file at the indicated path inside the FAT32 image referenced by the provided key. All the folders in the full path must already exist. The last name in the full path is interpreted as the file name and must satisfies the 8.3 file name format (if file name isn't formated as expected, VyBuild will fail the action and therefore the build pipeline). The content of the file is read from a file on disk and the file size is automatically determined by the file size of the source file on disk. Source file must be an existing regular file.
Syntax:
``` json
{
"action":"create_file_fat32",
"args":{
"key_variables":...,
"key":...,
"source_path_variables":...,
"source_path":...,
"full_path_variables":...,
"full_path":...,
"is_vftm_target":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the FAT32 image.
`source_path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_path` string.
`source_path` | string | any valid path | Indicate the path of the source file on disk.
`full_path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `full_path` string.
`full_path` | string | any valid path in the FAT32 image | Indicate the full path of the file to create.
`is_vftm_target` | boolean | `true` or `false` | Indicate if this file is the VFTM target. If you don't know what VFTM is, set this flag to `false` and ignore it.
#### `export_fat32`
The `export_fat32` action exports the FAT32 image referenced by the provided key to a specified file on the disk. `disk_guid_var` and `efi_part_unique_guid_var` are two optionals arguments and must be provided only if `generate_vftm` is `true`. If you don't know what VFTM is, set the `generate_vftm` flag to `false` and ignore it.
Syntax:
``` json
{
"action":"export_fat32",
"args":{
"key_variables":...,
"key":...,
"export_path_variables":...,
"export_path":...,
"generate_vftm":...,
"disk_guid_var":...,
"efi_part_unique_guid_var":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the FAT32 image.
`export_path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `export_path` string.
`export_path` | string | any valid path | Indicate the path of the file in which the FAT32 image should be exported.
`generate_vftm` | boolean | `true` or `false` | Indicates if the VFTM should be generated and integrated to the exported image. If you don't know what VFTM is, set this flag to `false` and ignore it.
`disk_guid_var` | string | any variable name without the `$` symbols | Indicate the variable that contain the disk GUID. Only mandatory if `generate_vftm` is `true`. If you don't know what VFTM is, just don't set this field.
`efi_part_unique_guid_var` | string | any variable name without the `$` symbols | Indicate the variable that contain the ESP unique GUID. Only mandatory if `generate_vftm` is `true`. If you don't know what VFTM is, just don't set this field.
### Disk related actions
VyBuild integrates its own disk images generator. Because VyBuild is still in early development, it can only generate disk images with these limitations:
- No MBR support at all
- 128 partition entries maximum
- 128 bytes per partition entries
- GUID must be provided as UUID v4 syntax. All 16 bytes of data are considered given as Big Endian, and so the first 8 bytes of data will be converted to Little Endian to comply with the GPT specification
- Fixed LBA size of 512 bytes
- No partition alignment other than LBA size
This implementation doesn't have the objective of being a complete disk driver but rather, a simpler alternative to easily generate disk image directly in the build pipeline, avoiding the complex Linux tricks and need for root permissions usages. But, it's still sufficient for generating simple disk images, even with custom partitions.
**Warning:** while there is no global lock on the disk image generator/on each disk image, there is also no ban of disk related actions in multithreaded part of the build pipeline. This mean that the user is fully responsible that no more than one part of the build pipeline use the disk image generator at the same time.
#### `create_disk`
The `create_disk` action will create an empty disk image in RAM and will initialize all GPT structures. The value in `key` is used to identify the disk image for later actions on this image. The size must be provided in bytes and must be a multiple of 512. The key and disk GUID must not be empty.
Syntax:
``` json
{
"action":"create_disk",
"args":{
"key_variables":...,
"key":...,
"size":...,
"disk_guid_variables":...,
"disk_guid":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the disk image.
`size` | integer | any integer multiple of 512 except 0 | Indicate the size in bytes of the disk image.
`disk_guid_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `disk_guid` string.
`disk_guid` | string | any valid UUID v4 | Indicate a GUID for the disk under the form of a UUID v4 Big Endian.
#### `add_partition_disk`
The `add_partition_disk` action will create a new partition on the disk image referenced by the provided key. The key must be a valid disk image key. The type GUID, unique GUID and name must not be empty. The source file path must lead to a valid partition image stored in an existing regular file on disk.
Syntax:
``` json
{
"action":"add_partition_disk",
"args":{
"key_variables":...,
"key":...,
"type_guid_variables":...,
"type_guid":...,
"unique_guid_variables":...,
"unique_guid":...,
"name_variables":...,
"name":...,
"source_file_variables":...,
"source_file":...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the disk image.
`type_guid_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `type_guid` string.
`type_guid` | string | any valid UUID v4 | Indicate a type GUID for the partition under the form of a UUID v4 Big Endian.
`unique_guid_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `unique_guid` string.
`unique_guid` | string | any valid UUID v4 | Indicate a unique GUID for the partition under the form of a UUID v4 Big Endian.
`name_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `name` string.
`name` | string | anything except empty string | Indicate a name for the partition. It's recommanded only putting ASCII characters in this field, as VyBuild will convert in a very simple manner into UTF-16LE: simply casting each `char` into `uint16_t`
`source_file_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `source_file` string.
`source_file` | string | any valid path | Indicate the path of the partition image on disk.
#### `export_disk`
The `export_disk` action exports the disk image referenced by the provided key to a specified file on the disk.
Syntax:
``` json
{
"action":"export_disk",
"args":{
"key_variables":...,
"key":...,
"export_path_variables":...,
"export_path":"...
}
}
```
Arguments:
Name | Type | Possible values | Role
-----|------|------------------|-----
`key_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `key` string.
`key` | string | anything except empty string | Indicate an identifier for the disk image.
`export_path_variables` | boolean | `true` or `false` | Indicates if variables replacement should be performed in the `export_path` string.
`export_path` | string | any valid path | Indicate the path of the file in which the disk image should be exported.
View Git Blame Copy Permalink