lolo859/vystem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First commit, Vystem v0.1

Browse Source
This commit is contained in:
lolo859
2026-03-31 22:15:00 +02:00
commit e15daed8c0
462 changed files with 134655 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
docs/blastproof/bootconfig.md Normal file
View File

@@ -0,0 +1,111 @@
# Blastproof boot config
## Introduction
Blastproof store his configuration inside a text file named `bp.conf` and stored inside `\EFI\BlastProofFiles` inside the EFI partition. Due to being protected by SBFIE, the config is immutable after compilation and any change to it will cause the boot to fail, even if provided password was good. A example configuration is provided as `Blastproof/config/bp_template.conf`
## Syntax and behaviour
Blastproof configuration format is pretty simple:
- Any line starting by `#` will be considered a comment and ignored
- Any empty line will be ignored
- All the other lines will be considered as a key-value pair, the first `=` will be considered as a separator.
- All spaces from both sides of the separator will be considered a part of the key or value, including starting and trailing spaces
- All lines are separated by `\n`
Any non-ascii character will cause the boot to fail. All the keys that are listed below are automatically replaced by their default value if non-present. Somes keys are mandatory for the boot to succeed. Any invalid value will cause the boot process to fail.
## List of all keys
### Serial port related keys
**serial_port_enabled:**
- Description: Authorize the output on serial port once the bootloader enable the selected graphic output mode. Override `serial_port_erroring` and `serial_port_debuging`
- Possible value: `true` or `false`
- Default value: `false`
**serial_port_debuging:**
- Description: Authorize the output on serial port for anything else than errors
- Possible value: `true` or `false`
- Default value: `false`
**serial_port_erroring:**
- Description: Authorize the output on serial port for anything related to errors
- Possible value: `true` or `false`
- Default value: `false`
### Graphics related keys
**font:**
- Description: Give the name of the file containing the font the bootloader will atempt to use.
- Possible value: any value representing a file name stored inside `EFI\BlastProofFiles`
- Default value: `bitra-ascii-medium.fbm`
**disable_boot_animation:**
- Description: Disable the loading and playing of the boot animation
- Possible value: `true` or `false`
- Default value: `true`
**default_horizontal_resolution:**
- Description: define the target horizontal resolution for graphic output mode selection. Set this and `default_vertical_resolution` to `0` to select highest resolution available
- Possible value: any null or positive integer
- Default value: `0`
**default_vertical_resolution:**
- Description: define the target vertical resolution for graphic output mode selection. Set this and `default_horizontal_resolution` to `0` to select highest resolution available
- Possible value: any null or positive integer
- Default value: `0`
### InitFS and SignSyst related keys
**initfs_partition_type_guid:**
- Description: define the type GUID of an InitFS partition. Warning: this GUID is common to all installation and shouldn't be modified in any way
- Possible value: any GUID under the form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` where `x` is any hexadecimal character
- Default value: `8362b434-d825-11f0-a68f-10ffe08423a6`
**initfs_partition_guid:**
- Description: define the unique GUID of the InitFS partition. Warning: this GUID is changed at every compilation and auto-completed by build script
- Possible value: any GUID under the form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` where `x` is any hexadecimal character
- Default value: there is no default value for this pair
**signsyst_partition_type_guid:**
- Description: define the type GUID of an SignSyst partition. Warning: this GUID is common to all installation and shouldn't be modified in any way
- Possible value: any GUID under the form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` where `x` is any hexadecimal character
- Default value: `da0048b4-d826-11f0-b877-10ffe08423a6`
**signsyst_partition_guid:**
- Description: define the unique GUID of the SignSyst partition. Warning: this GUID is changed at every compilation and auto-completed by build script
- Possible value: any GUID under the form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` where `x` is any hexadecimal character
- Default value: there is no default value for this pair
### Shelter related keys
**kernel_log_level:**
- Description: define the log level of the kernel
- Possible value: any integer ranging from 0 to 6
- Default value: `1`
**kernel_test_benchmark:**
- Description: define if the kernel should test and benchmark his subsystems
- Possible value: `true` or `false`
- Default value: `false`
**kernel_bench_iterations:**
- Description: define the amount of iterations used for tests and benchmarks
- Possible value: any integer ranging from 0 to 10000
- Default value: `10000`
**kernel_log_disable_serial_port:**
- Description: disable the outputting of logs on the serial port
- Possible value: `true` or `false`
- Default value: `true`
**kernel_disable_serial_port:**
- Description: disable any outputting on the serial port
- Possible value: `true` or `false`
- Default value: `false`
**kernel_log_ring_size:**
- Description: define the amount of pages used for logging ring buffer. Setting this pair to 0 will disable the logging ring.
- Possible value: any integer ranging from 0 to 65535
- Default value: `2048`

35
docs/blastproof/bootprocess.md Normal file
View File

@@ -0,0 +1,35 @@
# Blastproof boot process
Blastproof boot process follow a very rigid sequences of steps that are detailled below:
- Printing informations:
- Bootloader version
- Firmware informations
- CPU informations
- Reading boot password from EFI console
- Verifying integrity of integrated keys (see SBFIE)
- Acquiring access to ESP partition
- Acquiring loaded image handle
- Acquiring BlockIO protocol for ESP partition
- Acquiring filesystem protocol (Blastproof doesn't have an integrated FAT32 driver)
- Opening root folder
- Performing cryptographic checks with internal keys on his own files (see SBFIE)
- Readind, parsing and applying boot configuration
- Locating Simple Graphic Protocol (GOP) handle
- Identify a suitable graphic output format, based on configuration, or highest resolution available in case of fallback
- Loading and rendering bitmap font according to selected graphic output format
- Loading, rendering and playing boot animation if configuration allow it
- Loading InitFS and SignSyst
- Listing all availables BlockIO protocol
- Locating handles for InitFS and SignSyst
- Allocating memory for them
- Loading them into RAM
- Parsing their headers
- Performing cryptographic checks to validate their headers and contents
- Loading the kernel
- Loading Shelter and Keycard binary
- Generating Shelter boot configuration
- Allocating pages for kernel sections
- Creating page table
- Parsing EFI Memory Map and transforming it into a understandable format for Vystem
- Jumping to kernel

44
docs/blastproof/bplib.md Normal file
View File

@@ -0,0 +1,44 @@
# Blastproof library
## Introduction
While Blastproof is based on EDK II C library, it uses a wrapper around what EDK II can provide (mainly BlockIO protocol, filesystem protocol, graphic output protocol and custom Vystem formats support) as well custom-integrated cryptographic libraries. This wrapper is called the Blastproof library.
## External libraries
The source code for external libraries are stored in different folders depending on the library. Their source code has been adapted to run in an EDK II environment. Here is a list of all external libraries inside the Blastproof library, that doesn't include libraries used in boot utilities:
Folder | Original source code | Author | License
--- | --- | --- | ---
Blastproof/src/libs/sha3 | [mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3) | mjosaarinen | MIT
Blastproof/src/libs/sphincsplus | [sphincs/sphincsplus](https://github.com/sphincs/sphincsplus) | SPHINCS+ team | MIT-0
Blastproof/src/libs/argon2 | [P-H-C/phc-winner-argon2](https://github.com/P-H-C/phc-winner-argon2) | Argon2 team | CC0-1.0
Note: some of the projects cited above let the user choose the license at their convenience. The most permissive option has been selected when applicable
## EDK II
The Blastproof bootloader is based on the EDK II framework:
- Source: https://github.com/tianocore/
- License: BSD-2-Clause-Patent
- Copyright:
Copyright (c) Intel Corporation and other contributors.
## Blastproof library components
Blastproof library is divided into several components, some relying on the external libraries cited above, and all relying on the EDK II framework. Here is the list of all components with their headers:
- `conf.h`: manage configuration loading and parsing
- `console.h`: provide a simple way to enter a password inside the EFI shell
- `cpu.h`: provide a simple way to print CPU ID informations
- `crypto.h`: provide the cryptographic abstraction layer to use algorithms such as Argon2, SHA3 or Sphincs+
- `debug.h`: provide outputting primitives to the serial port
- `default.h`: provide macros for default configuration values
- `disk.h`: provide functions to read files from ESP partition
- `font.h`: provide functions to load, parse and render FBM fonts
- `graphic.h`: provide basic primitives to interact with GOP framebuffers
- `initfs.h`: InitFS and SignSyst drivers
- `ui.h`: provide primitives to render text and boot animation in the GOP framebuffer
- `vyx.h`: provide a very basic Vyx-based kernel loader
Additional headers can be generated at compilation time and aren't included in this list, such as `key.h`.

53
docs/blastproof/fbm.md Normal file
View File

@@ -0,0 +1,53 @@
# Font Bitmap
## Introduction
Font Bitmap (FBM) is the custom font format used by the Blastproof bootloader. It has been designed to be simple and support a selected set of features useful and simple to render from a bare metal environnement. This font format is generated by the `fontgen` utility. In this file, we describe how they are structured, not how they are generated, rendered or used. Please take note that this format has been designed for the x86-64 architecture and for little endian usage. More precisely, all multibytes integer are little endian, except codepoint encoding (see below).
## Header
Each FBM font start with the following 10 bytes header:
- Signature: 3 bytes that are set to `FBM` in ASCII
- Encoding: 1 byte. Reserved encoding are 0x01 for ASCII, 0x08 for UTF-8 and 0x10 for UTF-16
- Entries count: 4 bytes, indicate the number of glyph inside the file
- Width: 1 byte, the width in pixel of a glyph
- Height: 1 byte, the heigth in pixel of a glyph
Please note that the provided font in Vystem are only tested for ASCII encoding.
## Glyphs table
Each glyph occupies `4+width*height` bytes.
### Codepoint
The 4 first byte are for the codepoint. The codepoint is encoded as detailled below, **no matter the endianness**:
- ASCII: the first byte of the codepoint is the ASCII code, the next 3 are set to 0
- UTF-8: if `N` is the amount of byte the UTF-8 code take, then the first `N` are to be set to the UTF-8 code bytes, and the next `4-N` bytes are set to 0
- UTF-16: if 2 bytes are used, the two first bytes of the UTF-16 code are set into the first two bytes of the codepoint. If 4 bytes are used, use all codepoint bytes
In all cases, the useful bytes are always left-aligned and the unused bytes are set to 0. The codepoint `0xFFFFFFFF` is reserved for the replacement character in case of missing character in the font.
### Pixel encoding
Each pixel is encoded with two values in mind:
- a boolean indicating if the pixel is on or off
- a 4 bit value indicating the shading of the pixel, ranging from 0 (background color) to 15 (text full color)
Each pixel is encoded on one byte with the following scheme:
- Bit 0 (MSB): shading bit 0 (MSB of the shading value)
- Bit 1: second bit of the first byte of the codepoint
- Bit 2: shading bit 1
- Bit 3: fourth bit of the second byte of the codepoint
- Bit 4: shading bit 2
- Bit 5: sixth bit of the third byte of the codepoint
- Bit 6: shading bit 3 (LSB of the shading value)
- Bit 7: on/off bit
Note: above, when we provide a bit index, it starting from left to rigth. It apply to pixel byte, shading value and bit index for corruption detection.
When rendering the font, the renderer will take two values: background color and text color. It will create a scale ranging from 0 to 15 using linear interpolation and apply the expected color to each pixel using this scale and the shading value. The bit 1, 3 and 5 of each pixel is set to bits found in the codepoint to detect corruption. The pixels are stored in a grid, line by line, from top to bottom.
## Provided font
Vystem provide a basic font named `bitra`, with 12 pixels as width and 20 as height, only supporting ASCII characters. This font is stored in the file `bitra-medium-ascii.fbm`.

91
docs/blastproof/fs.md Normal file
View File

@@ -0,0 +1,91 @@
# Custom filesystems
## Introduction
Vystem define two new filesystems: InitFS and SignSyst. The purpose of these filesystems are to store any sensitive systems files. Both of these filesystems are designed to be integrated together. For each InitFS, there is always a SignSyst and vice versa. These filesystems are generated by the `initfsgen` utility. In this file, we describe how they are structured, not how they are generated, verified or used. Please take note that these filesystems has been designed for the x86-64 architecture and for little endian usage.
## InitFS
InitFS is new, very simple filesystem. It doesn't support folders and is completely immutable after generation. It has been designed to support strong files integrity check and be loader as one big chunk into memory.
### Header
The header of any InitFS filesystem is as follow:
- Signature: 8 bytes that are set to `InitFiSy` in ASCII
- Bootloader version: an uint16_t indicating the bootloader version, currently set to `0x0001`
- InitFS version: an uint16_t indicating the InitFS version, currently set to `0x0001`
- OS version: an uint32_t indicating the OS version, currently set to `0x00000001`
- Installation ID: an array of 48 completely random bytes. His SHA3-512 hash is stored inside the `initfs-footprint.bin` file, protected by SPFIE
- InitFS Size: the amount of bytes in the total filesystem, not forcefully rounded to the disk block size. Stored as an uint64_t
- Files table size: the amount of bytes used for the filesystem files table. Stored as an uint64_t
- Files content size: the amount of bytes for the filesystem files content area. Stored as an uint64_t
- Entry width: the width of a file table entry. In the current version of InitFS, must always be set to 256. Stored as an uint64_t
- Entries count: the count of entries inside files table, stored as an uint64_t
- Files content area offset: the amount to bytes to add to the filesystem base to reach the files content area. Stored as an uint64_t
- Entropy: 8 bytes of random data, stored as an uint64_t
- Entropy check: 8 bytes of a value generated with this formula : `header.entropy_check=(header.entropy*0x9E3779B185EBCA87)^header.entropy`. Stored as an uint64_t
- Files table hash: an array of 64 bytes containing the SHA3-512 hash of the files table
- Files content area hash: an array of 64 bytes containig the SHA3-512 hash of the files content area
- Installation ID double hash: an array of 64 bytes containing the SHA3-512 hash of the content of the `initfs-footprint.bin` file.
- Padding: 128 bytes of padding generated with a specific pattern (see below)
- Header hash: an array of 64 bytes containing the SHA3-512 of the InitFS header, not including the last 64 bytes
This header is exactly 512 bytes. The mecanism decribed above with the installation ID and the `initfs-footprint.bin` is to ensure that an installation of Blastproof find his associated InitFS.
### Files table and files entries
The files table is filled with files entries structured as follow, each being 256 bytes:
- File offset in the files content area, starting at the beginning of the files content area,stored as an uint64_t
- File size, stored in bytes as an uint64_t
- Sphincs+ public key of the file, stored as an array of 64 bytes
- SHA3-512 hash of the file, stored as an array of 64 bytes
- File name, stored as an array of 112 bytes
Files names can only use ASCII and have to be finished by `\0`, unless the file name is exactly 112 characters.
### Files content area
The files content area come just after the files table and contain the files content.
## SignSyst
SignSyst is a special filesystem: it doesn't contain files but signatures. Signatures have all the same fixed size and are stored in the same order as the files entries inside the associated InitFS.
### Header
The header of any SignSyst is as follow:
- Signature: 8 bytes that are set to `SignSyst` in ASCII
- Bootloader version: an uint16_t indicating the bootloader version, need to be set as same as the associated InitFS
- InitFS version: an uint16_t indicating the InitFS version, need to be set as same as the associated InitFS
- OS version: an uint32_t indicating the OS version, need to be set as same as the associated InitFS
- Installation ID: an array of 48 bytes containing the installation ID, need to be set as same as the associated InitFS
- Signature size: the size of a signature, stored as an uint64_t
- Signature count: the count of signatures, stored as an uint64_t
- SignSyst size: the amount of bytes in the total filesystem, not forcefully rounded to the disk block size. Stored as an uint64_t
- Signature area size: the amount of bytes used to actually store the signatures, stored as an uint64_t
- Signature area hash: an array of 64 bytes containing the SHA3-512 hash of the signatures area
- Padding: 288 bytes of padding generated with the same pattern as InitFS
- Header hash: an array of 64 bytes containing the SHA3-512 of the SignSyst header, not including the last 64 bytes
The header is exactly 512 bytes. The file `signsyst-hash.bin`, protected by SPFIE, contain the full hash of the SignSyst header.
### Signatues area
The signatures area come just after the SignSyst header and contain all Sphincs+ signatures.
## Padding generation
InitFS and SignSyst padding use a very specific generation method:
The padding is first initialized with zeroed bytes, after which a marker value `p` is written at positions determined by the sequence defined by (f0,f1)=(0,1) and iteratively updated as (f0←f1,f1←f0+f1). At each step, the byte at index `f0` is set to `p`, resulting in writes at offsets following the Fibonacci sequence until the index exceeds the padding size.
In InitFS, the marquer value `p` is the first byte of the installation ID.
In SignSyst, the marquer value `p` is the last byte of the installation ID.
## Hash generation
The files table hash, files content area hash and signatures area hash are altered in a certain way: their 48 last bytes are xored with the installation ID. The header hash for both filesystems are generated at the final end of the generation process.
## Stored files
To this day, the InitFS contain the following files
- `shelter.vyx`
- `keycard.bin`

21
docs/blastproof/index.md Normal file
View File

@@ -0,0 +1,21 @@
# Blastproof Docs
## Introduction
Blastproof is Vystem's bootloader. It is a UEFI x86-64 only bootloader that rely on EDK II for all hardware interactions. It is able to confirm the integrity of his files, locate the kernel and load it into memory, using the specific boot protocol designed for Shelter.
## Blastproof architectural overview
The Blastproof bootloader rely mainly on the EDK II framework for printing to the EFI console, interacting with disks/partitions and FAT32 filesystem, graphic output protocol, and memory allocation. However, it uses Vystem standard formats for fonts, filesystems, security and integrity checks, graphic renderer, and kernel loading.
## Ressources
1) [Bootprocess](bootprocess.md)
2) [Boot configuration](bootconfig.md)
3) [Security and integrity model at boot level](security.md)
4) [Secure Boot Files Integrity Enforcement (SBFIE)](sbfie.md)
5) [Custom filesystems](fs.md)
6) [Font Bitmap](fbm.md)
7) [Blastproof library](bplib.md)
For detailled informations about kernel loading, please see Vyld docs.

53
docs/blastproof/sbfie.md Normal file
View File

@@ -0,0 +1,53 @@
# SBFIE
## Introduction
Secure Boot Files Integrity Enforcement (SBFIE) is a compoment of Blastproof in charge of verifying the integrity of Blastproof's files. This allow for cryptographic immutability of boot files.
## Limitations
SBFIE has been conceived with the following limitations in mind:
- it doesn't check integrity of the entire bootloader itself
- it doesn't replace UEFI Secure Boot but make it impossible to use without customised certificates generated at each compilation (not supported for the moment)
- due to the will of it being compatible with a large variety of devices, it doesn't integrate TPM utilization (but support in an eventual future is planned)
- any user of a non-QWERTY keyboard will have a hard time typing a password in an UEFI console
- it use the EFI console as input (support of self-made input methods and protocol will be implemented in a near future)
- any missing file that is supposed to be protected by SPFIE will cause the boot process to fail
## How it work
For details on how to setup SBFIE, see the doc on the `keygen` utility.
### Initial parameters and variables
The following explenations assume that the bootloader has been compiled with a specially generated header `key.h`, generated by the `keygen` utility, for protecting `N` number of files.
The `key.h` header contain the following datas:
- `bp_key_mainsig`: an array of bytes containing the main signature generated with Sphincs+
- `bp_key_pkblob`: an array of `64*N` bytes containing `N` Sphincs+ public keys, one for each file
- `bp_key_pwdsalt`: an array of 32 bytes containing the salt to use for hashing the boot password
- `bp_key_files`: an array of `N` strings indicating the order in which boot files should be verified for each public key from `sh_key_pkblob` to correspond with each file
We assume that all files are in the `\EFI\BlastProofFiles` folder with their corresponding names as specified in `bp_key_files` and that their Sphincs+ signatures are stored as `<file name>.sig` inside of the `\EFI\BlastProofSign` folder.
### Detailled steps
The steps in which SPFIE check the files is as follow:
1. The user enter the boot password
2. The boot password is hashed using Argon2 using 262144 of memory cost and 3 of time cost. The salt is stored inside `bp_key_pwdsalt`The output is a 96 digest.
3. The digest is used as seed for generating a Sphincs+ keypair, immediately erasing the private key and keeping only the public key.
4. The generated public key is used to verify the signature inside `bp_key_mainsig` with the message being `bp_key_pkblob`
5. If the verification is successfull, the boot process continue. If not, the boot process is halted.
6. Each file indicated inside `bp_key_files` is opened in order, their signature file is also opened and their public key is extracted from `bp_key_pkblob`. Each file is verified.
7. If any verification fail, the boot process is halted.
**Note:** all the signatures verifications mecanisms use Sphincs+ and Argon2 for password hashing. The boot password hash is never stored on disk. Their sources come directly from their official implementations in C, only adapted to use types and functions provided by EDK II. All non-deterministic/randomness generation happen at compile time.
## Concerned files
To this date, the following files generated by build script are protected by SPFIE:
- `bp.conf`
- `bitra-ascii-medium.fbm`
- `initfs-footprint.bin`
- `signsyst-hash.bin`

22
docs/blastproof/security.md Normal file
View File

@@ -0,0 +1,22 @@
# Security and integrity model at boot level
## Introduction
The Vystem project has been designed for the ground up for security and integrity, and that start directly at the boot level, by focusing on boot file integrity.
## Overview
In order to achieve a strong amount of security without sacrificing compatibility with existing devices, we have chosen to integrate the following elements into the boot sequence:
- Secure Boot Files Integrity Enforcement (SBFIE): protect cores bootloader files assuming bootloader integrity with a user-defined password, without storing it directly but integrating it very deeply in the integrity check chain. See [SPFIE docs](spfie.md) for more informations
- Robust post-quantum cryptography integrated directly into the bootloader: while we rely on EDK II for files and disk IO as well as password input, all the cryptography stack is integrated into the bootloader, not relying on firmware cryptography at any time. We use post-quantum signing algorithm like SPHINCS+ and strong hashing primitives like Argon2 and SHA3, with their sources codes directly taken from their official implementation, only modified to integrate with EDK II types and memory allocation system
- Signing of every system files: every sensitive system file is integrated into the InitFS, signed into a trio of check: each Blastproof binary (and so SPFIE keys), InitFS and SignSyst is uniquely associated (mainly by installation ID), meaning that the whole system can only work if all of them are properly setup on the same machine device. InitFS integrity check are also cryptographically linked to SPFIE verification.
## Limitations
The current Vystem security model assume that we can trust the firmware and that the user boot password is secure. We plan to add support for UEFI secure boot by using custom certificates generation in the future.
## Detailled parameters for cryptographic algorithms
For SPHINCS+, we use the set of parameters named `sphincs-sha2-256f`.
For SHA3, we use the standard implementation with the 512 bits variant.
For Argon2, we disable parallelism (this would be a pain to setup in a UEFI environnement) and use the following parameters: 96 bytes output (to match SPHINCS+ keypair seed size), 32 bytes salt, memory cost set to 262144 and time cost set to 3. We use the Argon2id variant.

66
docs/boottools/bootanim.md Normal file
View File

@@ -0,0 +1,66 @@
# Bootanim docs
## Introduction
The `bootanim` utility is responsible for generating the boot animation. It is used even if the bootloader configuration specify to not play the boot animation, as it contain the Vystem logo blueprint.
## Detailled informations
Folder: `Blastproof/bootanim`
Source code file: `bootanim.cpp`
For building `bootanim`, use the following command:
``` bash
g++ bootanim.cpp -o bootanim -Ofast -march=native
```
The `-Ofast -march=native` is recommended for better performances, as the boot animation generation is a bit heavy on computation.
External library:
- stb_image, made by `Sean Barrett`, sourced from [nothings/stb](https://github.com/nothings/stb), provided in the public domain
## Usage
Prerequisites: having `ffmpeg` accessible in path for the use of folder mode, for generating the video.
The `bootanim` can be used like this:
``` bash
bootanim <png logo file> [file/folder] <width> <height>
```
You can use `bootanim` in two differents mode:
- file mode: will export the boot animation as `bootanim.bin`, into a format readable by the bootloader
- folder mode: will export the boot animation as a folder named `frames` that will contain all the generated frames under the name format `frameXXXX.ppm`, as well as a video named `out.mp4`
Command example:
``` bash
bootanim logo.png file 1280 720
bootanim logo.png folder 2160 1440
```
## Boot animation generation
The first thing to know about Blastproof boot animation is that, due to poor FAT32 read performance, we decided to find a compromise between the amount of data being stored into the generated file and the amount of data computed at each boot. In order to maintain a fluid, 60 fps animation at each boot on most hardware, we landed on this compromise:
- We only store 16 frames per second, under the form of points coordinates
- The 48 others frames per second are computed using linear interpolation at each boot
This result in the `bootanim.bin` file being only 14 megabytes in size, instead of 56 megabytes, for the actual Vystem logo and a resolution of 1920x1080.
The detailled generation procedure is as follow:
1) Locate and open the logo file in monochrom mode (with only one color channel, conversion is made by stb_image). The utility will reject any non-square logo.
2) Center the logo on a black canva the size provided by `width` and `height` arguments.
3) Scan the canva to detect all points that have their value over 127. Save these points with their coordinates. In the Vystem logo, there is over 45k points.
4) For each of these points, generate a random position in the canva, the seed doesn't have any cryptographic implication and is randomised for every new build.
5) Apply two quadratic bezier curves: one for the "explosion" (all points starting from the exact center of the screen and being scattered around the screen to their random position) and one for the logo appearence (each point is assigned a random coordinates in the logo points and will be attrected to this position from their random position).
6) Generate the selected output media and write it into the files.
It is recommanded that the image file provided is already monochrom.
## Output file structure
The outputted file named `bootanim.bin` is the format that is usable by the bootloader. The file header is structured like this (all uint64_t are in little endian):
- Signature: an array of 8 bytes containing `BootAnim` in ASCII
- The width of canva used for generating the boot animation, stored as an uint64_t
- The height of canva used for generating the boot animation, stored as an uint64_t
- Number of frame inside the boot animation, stored as an uint64_t
- Number of points inside each frame, stored as an uint64_t
Then, each frame is written inside the file. Each frame is made of <number of points per frame> points. Each points is made of two uint16_t, one for the x position and one for the y position.

52
docs/boottools/fontgen.md Normal file
View File

@@ -0,0 +1,52 @@
# Fontgen docs
## Introduction
The `fontgen` utility is responsible for generating the font used by Blastproof. It can only generate FBM font.
## Detailled informations
Folder: `Blastproof/fontgen`
Source code file: `fontgen.cpp`
For building `fontgen`, use the following command:
``` bash
g++ fontgen.cpp -o fontgen
```
External library:
- stb_image, made by `Sean Barrett`, sourced from [nothings/stb](https://github.com/nothings/stb), provided in the public domain
## Usage
The `fontgen` utility can be used like this:
``` bash
fontgen <text color> <background color> [ascii/utf8/utf16] <path to folder containing images files for characters>
```
The text and background color arguments must be given under a hexadecimal RGB color code, starting with `#`. They serve as boundary for generating the shading scale.
The encoding argument can be `ascii`, `utf8` or `utf16`, however in Vystem build script, only the `ascii` encoding is used. Others supported encoding modes haven't been tested.
The final argument is the path to the folder that contain all images files for the font characters.
Each character is encoded with the following format:
- they must named with this format `0xXXXXXXXX` where `X` is an hexadecimal character. All `X` aren't mandatory, and the utility interprete them as characters codepoint. For example, file name `0x00` will be interpreted as a codepoint of value `0x00000000`, resulting in the `\0` character for ASCII
- each file inside the provided folder must be a PNG file
- each file must only contain pixel with colors inside the generated shading scale. For instance, the `bitra` font only use color from the two extremities of the shading scale. Transparency isn't checked as the shading scale is in RGB and all images files are opened in 3 channels mode.
- all files must have the same height and width, that will indicate the dimensions of the bitmap grid for each pixel
Any failure to meet the above points will result in font compilation error.
Command example:
``` bash
fontgen "#FFFFFF" "#000000" ascii ./chars
```
## Detailled generation process
1) Generate the shading scale by associating colors from the linear interpolation of the two provided colors to values ranging from 0 to 15. The text color will be occupy the value 15 and the background color the value 0.
2) For each image file in the provided folders: extract the character codepoint from the file name, endode it to FBM codepoint format, extract the raw pixel data
3) For each valid character: extract the codepoint bits that will be inserted into pixel bytes, iterate for each pixel of the character: identify which value in the shading scale the pixel color is, encode it into a byte with codepoint bits and on/off value (off if the shading value is 0, on otherwise)
4) Write the header and all the encoded characters into the final file that will be named `font.fbm`
Any failure to meet the above criteria (except some of the second step that will only trigger a warning) will result in font compilation error.
For a detailled overview of how a `.fbm` file is structured, please see [FBM docs](../blastproof/fbm.md).

11
docs/boottools/index.md Normal file
View File

@@ -0,0 +1,11 @@
# Bootloader tools Docs
In order to generate all necessary components for the build of a Vystem disk image, we have created several C++ tools designed to help generating these components. All of these tools have been intented to be used only in a normal Linux environnement. Here is the list of bootloader tools:
1) [bootanim](bootanim.md)
2) [fontgen](fontgen.md)
3) [initfsgen](initfsgen.md)
4) [keygen](keygen.md)
Here a few precisions about the global mecanisms inside each tools:
- when a tool is iterating on the content of a folder, it rely on the order provided by the filesystem. But tool like `keygen` get there files list through arguments specifying direct files path rather than a folder path, making it a deterministic order if the build script doesn't change
- when we say 'securely erased', it mean we are using the `explicit_bzero` function to prevent compiler optimization

50
docs/boottools/initfsgen.md Normal file
View File

@@ -0,0 +1,50 @@
# Initfsgen documentation
## Introduction
The `initfsgen` utility is responsible for generating the InitFS and SignSyst filesystems.
## Detailled informations
Folder: `Blastproof/initfsgen`
Source code file: `initfsgen.cpp`
For building `initfsgen`, use the provided build script `build.sh` while being in the utility directory.
External library:
- SPHINCS+, made by the SPHINCS+ team, sourced from [sphincs/sphincsplus](https://github.com/sphincs/sphincsplus), under the MIT-0 license
- tiny_sha3, made by mjosaarinen, sourced from [mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3), under the MIT license
## Usage
The `initfsgen` utility can be used like this:
``` bash
initfsgen <path to folder containing InitFS file>
```
The folder must contain only regular files. Any links, block device or folders inside the root folder will result in an error.
Command example:
``` bash
initfsgen ./initfs_dir
```
## Detailled generation process
1) Iterate over the content of the provided folder, checking if all elements are regular files, collecting their file sizes.
2) Collect 48 bytes of entropy for installation ID and 8 bytes of entropy for entropy check, by opening `/dev/urandom`. No others method of entropy generation are or will be supported.
3) Copying installation ID inside InitFS and SignSyst headers, filling all possible informations about files and signatures quantity and sizes inside the headers.
4) Generating entropy check, putting it into InitFS header.
5) Computing installation ID hash and saving it into `initfs-footprint.bin`, saved into the current directory.
6) Computing hash of the content of `initfs-footprint.bin` and saving it into the InitFS header.
7) Iterate for each file: loading the entire file, generating InitFS file name, copying file data into files content area, computing file content hash, generating SPHINCS+ keypair unique to this file, generating signature for file by using private key, securely erasing private key (without compiler optimization), saving file hash and public key into file entry, saving file entry into files table and adding generated signature to SignSyst
8) Generating files table, files content area and signatures area hash, saving them into headers by xoring them with part of the installation ID
9) Generating and saving padding for InitFS and SignSyst headers. Computing headers hash and finalizing filesystems raw data.
10) Saving InitFS under `initfs.bin`, hashing SignSyst header and saving it into `signsyst-hash.bin`, saving SignSyst under `signsyst.bin`
For the full and detailled layout of InitFS and SignSyst, please see [customs filesystems docs](../blastproof/fs.md).
All hashing operations use SHA3-512. All entropy generations use secure entropy from `/dev/urandom`.
This utility generate four files:
- `initfs.bin` and `signsyst.bin` are the filesystems expected to be flashed into their respectives partitions on disk
- `initfs-footprint.bin` and `signsyst-hash.bin` are files expected to be protected by SPFIE and put inside the ESP partition

47
docs/boottools/keygen.md Normal file
View File

@@ -0,0 +1,47 @@
# Keygen docs
## Introduction
The `keygen` utility is responsible for generating SPFIE keys, files and boot password. This file focus on how these ressources are generated and setuped, not how the verification is made at boot time. The explenations in this file are complementary to the informations in [SPFIE docs](../blastproof/spfie.md).
## Detailled informations
Folder: `Blastproof/keygen`
Source code file: `keygen.cpp`
For building `keygen`, first make sure that you have the `libargon2.a` static library file inside the utility directory. This one and the source of Argon2 for the `keygen` isn't included inside the main Vystem repository but downloaded and compiled by the build script. So you may have to compile Argon2 yourself before compiling `keygen`. To compile `keygen`, use the provided build script `build.sh` while being in the utility directory.
External libraries:
- SPHINCS+, made by the SPHINCS+ team, sourced from [sphincs/sphincsplus](https://github.com/sphincs/sphincsplus), under the MIT-0 license
- tiny_sha3, made by mjosaarinen, sourced from [mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3), under the MIT license
- Argon2, made by the Argon2 team, sourced from [P-H-C/phc-winner-argon2](https://github.com/P-H-C/phc-winner-argon2), under the CC0-1.0 license
## Usage
The `keygen` utility can be used like this:
``` bash
sudo keygen <path to file 1> <path to file 2> ... <path to file n>
```
You can provide as much paths as you want, as long as they all lead to regular files.
It's recommanded the `keygen` utility be launched as root.
## Detailled generation processus
1) Firstly, the `keygen` processus disable all possibilty of process memory dumps or memory swapping before doing anything else. Due the very thight security here, `keygen` should be running as root to ensure his own security.
2) Then, it check for the existence of all the provided files and lock private and public keys buffer as well as signature buffer to ensure that all the secrets aren't dumped on disk due to process fault or swapping.
3) Iterating for all the provided files: loading the file into memory, generating SPHINCS+ keypair for it, computing and signing the SHA3-512 hash of the file content if it's above 1 megabyte, or signing directly the file if under 1 megabyte, securely erasing all temporary keys and signature buffer, saving signature and public key into their dedicated buffer.
4) Saving all signatures into a directory named `sign`, created if non existing, under the name format `<original name>.sig`, then securely erasing the signatures buffer
5) Letting the user enter a boot password. If password is empty, it will ask for a number multiple of 2 to generate this many hexadecimal characters that will serve as password, the default amount of hexadecimal characters being 16. If the user enter a password, the user will have to confirm it
6) Checking that the password fill all criterias: should be (not enforced) at least 12 characters, can (enforced) only include ASCII characters due to EFI shell limitations, shouldn't be longer that 512 characters (enforced), and shoudn't use the numerical pad at all (not well supported in the EFI shell, but the user can still use numbers)
7) Offering the user the possibility to translate their password into a QWERTY representation of it, if they doesn't use a QWERTY keyboard. That way, the user can enter his password like he would do on his keyboard but the password that will be hashed will be the password translated in QWERTY.
**Warning:** This feature is still very experimental, the only translation supported is AZERTY to QWERTY and the user experience is still very unsettling. For testing purposes, it's recommanded using a password with no characters differences between keyboard layout such as `test`, for example compatibility between the AZERTY and QWERTY keyboard. A better password inputing experience with support for various keyboard layout is planned for future release.
8) Converting the final password to UTF16LE for EFI environnement easier compatibilty. Obtaining 32 bytes of entropy from `/dev/urandom`. Hashing the password with the Argon2id algorithm using no parralelism, time cost of 3 and memory cost of 262144, with the salt, resulting in a 96 bytes output.
9) Using the password as seed for generating a master SPHINCS+ keypair. Generating a signature of the public keys of each files concatenated into a single block. Securely erasing both keys of the master keypair.
10) Saving the final datas the bootloader will need into a file named `key.h`, containing the master signature, the block of files public keys, the password salt and the list of all files to verify, in order.
This utility generate the following files:
- `key.h`: header file containing important datas, made for single use only, should be placed into `Blastproof/src/libs/include` in order for the bootloader to succesfully compile
- all the `.sig` files: binary files containg the signature for each file, should be placed inside the ESP partition.

54
docs/build.md Normal file
View File

@@ -0,0 +1,54 @@
# Build process
## Introduction
In this file, we will see how you can build and boot successfully into a VM containing a Vystem disk image. It's recommanded to read the whole documentation for better understanding of why certains parts are necessary.
## Prerequisites
You should have cloned the full repository on your device. Please keep in mind that Vystem hasn't been tested on real hardware yet, so the only way to test it is through a virtual machine.
Vystem has been designed to be build and run from a standard Linux system. Any Linux distro should work. A WSL environnement can also work but the VM might be very slow, because of KVM unavailability. Make sure you have KVM enable and accessible before starting, or else remove the `--enable-kvm` argument at the end of the build script.
Before starting, please make sure that you have the following binary reachable in your terminal path:
```
python3 python nasm g++ gcc iasl git make sed wget unzip fallocate parted mkfs.fat sgdisk partprobe losesetup qemu-system-x86_64
```
Finally, please make sure that you have an internet connexion (required for downloading additionnal assets like EDK II and Argon2) as well as root access/sudo permissions, as it is recommanded for running `keygen` and required for creating the disk image. Root permissions is only obtained as soon as it's needed. If you don't want to give root permissions, you can also run the commands inside the build script one by one just to be sure.
## Build and run
Once you are ready, run the following command while being in the root folder:
``` bash
./build.sh
```
This will download, compile, and assemble every asset together. This can take a while depending on your internet connexion, since EDK II is quite heavy and will be downloaded from it's source repository and locally compiled, with OVMF. For the moment, using packaged versions of EDK II and/or OVMF provided by packages managers isn't supported.
The VM will automatically be launched with 4 gigabytes of RAM (you can obviously raise or lower that amount depending on your device), KVM enabled and serial port output redirected inside the terminal.
## Detailled build process
The folder used for InitFS base is named `initfs_dir` and is placed in the root folder of the repository.
The build process is as follow:
1) Checking virtual memory layout
2) Generating payloads for TAB subsystem
3) Compiling Keycard and putting it into `initfs_dir`
4) Compiling Vyld
5) Compiling Shelter and putting it into `initfs_dir`
6) Cloning EDK II, building required tools and OVMF
7) Compiling `bootanim` and generating boot animation
8) Compiling `fontgen` and generating font
9) Compiling `initfsgen` and generating InitFS and SignSyst
10) Patching bootloader configuration template to insert random partition GUID and fixed partition type GUID
11) Cloning and building Argon2, compiling `keygen` and generating required files for SPFIE
12) Compiling bootloader
13) Generating disk image
14) Launching virtual machine
## Troubleshooting
If anything goes wrong, the script will stop and show you the error. Sometimes it's just a dependency that isn't reachable. Carefully analyse the error message and try to install the missing dependencies, also check that the required dependencies listed above are all installed.
If it's related to EDK II (which occur the majority of the time), it could be that a new update in EDK II has broken something. Don't hesitate to open an issue, and this should be fix fairly quickly.

22
docs/index.md Normal file
View File

@@ -0,0 +1,22 @@
# Vystem Docs
**Warning:** this documentation only include currently implemented elements and doesn't include plannned future elements, except some confirmed planned updates scattered within the documentation
## Introduction
Welcome to Vystem documentation. This documentation is divided into severals parts for easier understanding.
## Components
Vystem is an OS made of severals components which can be divided into severals categories:
1) [Blastproof](blastproof/index.md): the bootloader
2) [Bootloader tools](boottools/index.md): all the tools needed to generate components for the boot process
3) [Vyld](vyld/index.md): the VYX executable format linker and VYX executable format specification
4) [Shelter](shelter/index.md): the kernel
5) [Kernel tools](kerneltools/index.md): all the `.py` scripts used for various usage around the kernel
## Ressources
- [Build process](build.md)
- [Contributing and licensing](licenses.md)
- [Roadmap](roadmap.md)

10
docs/kerneltools/index.md Normal file
View File

@@ -0,0 +1,10 @@
# Kernel tools
## Introduction
For somes usages, like generating payloads datas or checking certains files, a few python scripts are included with the kernel.
## Summary
1) [Virtual memory checker](vmemcheck.md)
2) [Payloads generator](payloads.md)

28
docs/kerneltools/payloads.md Normal file
View File

@@ -0,0 +1,28 @@
# Payloads generator
## Introduction
Payloads for the TAB subsystem are generated by special python scripts. Each script prints the generated file inside the terminal. For details about how the contents of the payloads are used, see [TAB documentation](../shelter/tab.md)
## Pez subsystem payload
File: `shelter/tools/generator/pez_alloc_free_payload_gen.py`
This script generates two arrays of `sh_uint64`:
- `test_pez_physical_size`: 2000 values that should be interpreted as pages count. Half of them are 1. The other half ranges from 2 to 1000. In the other half, higher values are less likely to appear as their value rises.
- `test_pez_physical_alloc`: 4000 values ranging 0 to 1999, each one appearing twice.
## Radix tree subsystem
File: `shelter/tools/generator/radix_tree_payload_gen.py`
This script generates three arrays of `sh_uint64`, containing 10000 random 8 bytes unsigned integers each: `test_keys`, `test_values` and `test_search`.
## Malloc subsystem
File: `shelter/tools/generator/malloc_payload_gen.py`
This script generate three arrays of `sh_uint64`:
- `test_malloc_small_size`: 10000 values that should be interpreted as size in bytes. They range from 1 to 1024, higher values are less likely to appear as their value rises.
- `test_malloc_big_size`: 1000 values ranging from 1 to 25, higher values are less likely to appear as their value rises.
- `test_malloc_big_alloc`: 2000 values ranging 0 to 999, each one appearing twice.

24
docs/kerneltools/vmemcheck.md Normal file
View File

@@ -0,0 +1,24 @@
# Virtual memory checker
## Introduction
This file is responsible for checking that there is no overlapp between virtual pages range defined in `shelter/lib/include/memory/vmem_layout.h`. For the full documentation about this file, please see [virtual memory layout documentation](../shelter/memory/vmemlayout.md).
## Usage
The script is located inside `shelter/tools/checker/vmem_layout_checker.py`.
It can be used like that:
``` bash
python shelter/tools/checker/vmem_layout_checker.py <path to header file>
```
## Detailled processus
1) Open provided file, parse it and print the list of valid regions.
2) Generate `cfile.c` to let the compiler do the hard work of parsing macro declaration
3) Compile (the `gcc` compiler must be accessible inside the path) and run the generated C program
4) Recover the output of the program, parse it to obtain start and size of each virtual region
5) Check for overlaps and return an error if something went bad during the processus
The compiled program and the C file are deleted afterward.

40
docs/licenses.md Normal file
View File

@@ -0,0 +1,40 @@
# Contributing and licensing
All license files can be found in the `licenses/` directory.
License texts for Vystem components are derived from [IQAndreas/markdown-licenses](https://github.com/IQAndreas/markdown-licenses), while third-party license texts are extracted from their respective original repositories.
## Licenses
All source code and build-related files are MPL-2.0 unless otherwise explicitly stated.
All `.md` files stored in the root directory and in the `docs` folder are under the MIT license. Code snippets copied from MPL-2.0 licensed source files remain under MPL-2.0.
## Third party licenses
The following third party libraries are used in Vystem:
Original source code | Author | License
--- | --- | ---
[mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3) | mjosaarinen | MIT
[sphincs/sphincsplus](https://github.com/sphincs/sphincsplus) | SPHINCS+ team | MIT-0
[P-H-C/phc-winner-argon2](https://github.com/P-H-C/phc-winner-argon2) | Argon2 team | CC0-1.0
[tianocore/edk2](https://github.com/tianocore/edk2) | Tianocore and all the contributing companies | BSD-2-Clause-Patent
[serge1/ELFIO](https://github.com/serge1/ELFIO) | Serge Lamikhov-Center | MIT
EDK II copyright: Copyright (c) Intel Corporation and other contributors.
## Contribution workflow
Contributors must work through public forks of this repository.
To propose changes for inclusion in the mainline, contributors should open an issue referencing the fork and the relevant commits. Changes are manually reviewed and selectively integrated by the maintainer.
## Contribution licensing grant
By submitting an issue proposing changes from a fork for inclusion in the mainline, the contributor:
- retains full copyright ownership of their contribution;
- agrees that the contribution remains licensed under MPL-2.0 where applicable;
- grants the mainline maintainer a perpetual, worldwide, irrevocable right to copy, modify, merge, redistribute, sublicense, and relicense the contributed changes as part of as part of the mainline project and any future versions or derivative works.
- agrees to be credited in the relevant source files and/or commit history where reasonably possible.

13
docs/roadmap.md Normal file
View File

@@ -0,0 +1,13 @@
# Roadmap
## Version 0.1 (actual)
Initial release
## Version 0.2 (future versions with planned additions)
- Vyld overhaul, support for multiples compilations modes with JSON config files for project settings
- VYX format overhaul, supporting various sections and more feature
- Blastproof: custom keyboard layout driver
- Shelter: Support for IDT, GDT, TSS, interruptions, interruptions handler, ACPI timers table parsing and ACPI driver
- Shelter: Slab allocators overhaul and performance boosts

149
docs/shelter/bench.md Normal file
View File

@@ -0,0 +1,149 @@
# Benchmark results
In this file, you will be able to see examples of benchmark results. These are provided as indicative results. Depending on your configuration, you might get way differents results
## Tests environnement
- Host OS: EndeavourOS x86_64
- Host kernel: Linux 6.19.10-arch1-1
- Host CPU: Ryzen 7 9800x3d @ 5.27 GHz
- Environnement: VM running in qemu-system-x86_64 version 10.2.2
- VM parameters: KVM enabled, 4096M of RAM, cpu set to host, running in single thread
The following tests are shown in the order the kernel run them. Please keep in mind that Pez isn't available until the Pez physical plane benchmark.
## Results for region objects slab allocators
Physical regions object slab allocator:
```
Result for benchmark "allocations" :
Min: 141 | Med: 235 | Avg: 2736 | Max: 3194120 | Total : 27362977 (TSC)
[P00-P90] 141-282 : [##################--] 90%
[P90-P99] 282-282 : [#-------------------] 9%
[P99-Max] 282-3194120 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 141 | Avg: 132 | Max: 376 | Total : 1320700 (TSC)
[P00-P90] 94-141 : [##################--] 90%
[P90-P99] 141-141 : [#-------------------] 9%
[P99-Max] 141-376 : [--------------------] 1%
```
Virtual regions object slab allocator:
```
Result for benchmark "allocations" :
Min: 141 | Med: 235 | Avg: 2781 | Max: 2840445 | Total : 27816339 (TSC)
[P00-P90] 141-282 : [##################--] 90%
[P90-P99] 282-329 : [#-------------------] 9%
[P99-Max] 329-2840445 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 141 | Avg: 130 | Max: 517 | Total : 1304109 (TSC)
[P00-P90] 94-141 : [##################--] 90%
[P90-P99] 141-141 : [#-------------------] 9%
[P99-Max] 141-517 : [--------------------] 1%
```
## Results for radix node slab allocator (PBA based)
Radix node slab allocator (PBA based):
```
Result for benchmark "allocations" :
Min: 141 | Med: 188 | Avg: 441 | Max: 534531 | Total : 4410339 (TSC)
[P00-P90] 141-235 : [##################--] 90%
[P90-P99] 235-282 : [#-------------------] 9%
[P99-Max] 282-534531 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 94 | Avg: 97 | Max: 329 | Total : 973370 (TSC)
[P00-P90] 94-94 : [##################--] 90%
[P90-P99] 94-141 : [#-------------------] 9%
[P99-Max] 141-329 : [--------------------] 1%
```
## Results for radix trees subsystem
Radix trees subsystem:
```
Result for benchmark "insertions into radix trees" :
Min: 2350 | Med: 3196 | Avg: 5194 | Max: 544448 | Total : 51944259 (TSC)
[P00-P90] 2350-3666 : [##################--] 90%
[P90-P99] 3666-134373 : [#-------------------] 9%
[P99-Max] 134373-544448 : [--------------------] 1%
Result for benchmark "reading into radix trees" :
Min: 470 | Med: 517 | Avg: 553 | Max: 31631 | Total : 5537399 (TSC)
[P00-P90] 470-564 : [##################--] 90%
[P90-P99] 564-987 : [#-------------------] 9%
[P99-Max] 987-31631 : [--------------------] 1%
Result for benchmark "searching value with lower bound key" :
Min: 423 | Med: 893 | Avg: 923 | Max: 30832 | Total : 9234184 (TSC)
[P00-P90] 423-1128 : [##################--] 90%
[P90-P99] 1128-1645 : [#-------------------] 9%
[P99-Max] 1645-30832 : [--------------------] 1%
Result for benchmark "deleting values" :
Min: 1645 | Med: 1880 | Avg: 1888 | Max: 35203 | Total : 18889770 (TSC)
[P00-P90] 1645-2021 : [##################--] 90%
[P90-P99] 2021-2162 : [#-------------------] 9%
[P99-Max] 2162-35203 : [--------------------] 1%
```
## Results for Pez physical plane
Pez physical plane:
```
Result for benchmark "allocations for single page" :
Min: 1833 | Med: 2444 | Avg: 3496 | Max: 8225 | Total : 3496518 (TSC)
[P00-P90] 1833-5922 : [##################--] 90%
[P90-P99] 5922-6815 : [#-------------------] 9%
[P99-Max] 6815-8225 : [--------------------] 1%
Result for benchmark "free for single page" :
Min: 1974 | Med: 2820 | Avg: 3807 | Max: 10105 | Total : 3807376 (TSC)
[P00-P90] 1974-6862 : [##################--] 90%
[P90-P99] 6862-8131 : [#-------------------] 9%
[P99-Max] 8131-10105 : [--------------------] 1%
Result for benchmark "allocations for multiple pages" :
Min: 2209 | Med: 5593 | Avg: 5716 | Max: 35861 | Total : 5716798 (TSC)
[P00-P90] 2209-7191 : [##################--] 90%
[P90-P99] 7191-8178 : [#-------------------] 9%
[P99-Max] 8178-35861 : [--------------------] 1%
Result for benchmark "free for multiple pages" :
Min: 1927 | Med: 4935 | Avg: 5095 | Max: 31772 | Total : 5095599 (TSC)
[P00-P90] 1927-7896 : [##################--] 90%
[P90-P99] 7896-10011 : [#-------------------] 9%
[P99-Max] 10011-31772 : [--------------------] 1%
```
## Results for malloc subsystem
Malloc subsystem:
```
Result for benchmark "sh_malloc for small size" :
Min: 188 | Med: 423 | Avg: 535 | Max: 380277 | Total : 5358752 (TSC)
[P00-P90] 188-799 : [##################--] 90%
[P90-P99] 799-1363 : [#-------------------] 9%
[P99-Max] 1363-380277 : [--------------------] 1%
Result for benchmark "sh_free for small size" :
Min: 141 | Med: 329 | Avg: 403 | Max: 27401 | Total : 4039556 (TSC)
[P00-P90] 141-658 : [##################--] 90%
[P90-P99] 658-1128 : [#-------------------] 9%
[P99-Max] 1128-27401 : [--------------------] 1%
Result for benchmark "sh_malloc for pages allocations" :
Min: 4935 | Med: 11468 | Avg: 11890 | Max: 47893 | Total : 11890013 (TSC)
[P00-P90] 4935-16450 : [##################--] 90%
[P90-P99] 16450-20257 : [#-------------------] 9%
[P99-Max] 20257-47893 : [--------------------] 1%
Result for benchmark "sh_free for pages allocations" :
Min: 5593 | Med: 11421 | Avg: 12005 | Max: 50760 | Total : 12005398 (TSC)
[P00-P90] 5593-17296 : [##################--] 90%
[P90-P99] 17296-21432 : [#-------------------] 9%
[P99-Max] 21432-50760 : [--------------------] 1%
```

64
docs/shelter/bootconfig.md Normal file
View File

@@ -0,0 +1,64 @@
# Shelter kernel boot configuration
## Introduction
The Shelter kernel requires a boot configuration in order to start. It doesn't come under the form of a command line but under the form of a C struct. For details on how to transmit the configuration to the kernel, please see [boot contract docs](bootcontract.md).
## Overview
Shelter boot configuration come under the following layout (extracted from `shelter/lib/include/kernel/conf.h`) :
``` C
typedef struct __attribute__((aligned(8))) {
sh_uint8 sig_start[8];
sh_uint8 log_level;
sh_uint16 page_table_allocator_level;
sh_page_PHYSICAL_ADDRESS page_table_pool_pa;
sh_page_VIRTUAL_ADDRESS page_table_pool_va;
sh_bool test_benchmark;
sh_uint64 bench_iterations;
sh_bool log_disable_serial_port;
sh_bool disable_serial_port;
sh_uint16 log_ring_size;
sh_uint8 sig_end[8];
} sh_conf_BOOT_CONFIG;
```
The starting signature must be `ShCfgBeg` in ASCII and the starting signature must be `ShCfgEnd` in ASCII. The alignment to respect for the boot config is 8 bytes.
## List of all keys
**log_level:**
- Type: 1 byte unsigned integer
- Description: represent the minimal (inclusive) log level for any log payload to be logged, can range from 0 to 6. The value of this key is provided by the `kernel_log_level` from the Blastproof boot config
**page_table_allocator_level:**
- Type: 2 bytes unsigned integer
- Description: the value of the bumb allocator after all mapping inside the kernel boot PTP
**page_table_pool_pa:**
- Type: physical address, aka 8 bytes unsigned integer
- Description: the physical address of the first page of the kernel boot PTP
**page_table_pool_va:**
- Type: virtual address, aka 8 bytes unsigned integer
- Description: the virtual address of the first page of the kernel boot PTP, inside kernel virtual memory layout
**test_benchmark:**
- Type: boolean, aka 1 byte unsigned integer
- Description: define if the kernel should test and benchmark his subsystems. The value of this key is provided by the `kernel_test_benchmark` from the Blastproof boot config
**bench_iterations:**
- Type: 8 bytes unsigned integer
- Description: define the amount of iterations to apply for compatible benchmarks. Not all benchmarks will follow this value. The value of this key is provided by the `kernel_bench_iterations` from the Blastproof boot config
**log_disable_serial_port:**
- Type: boolean, aka 1 byte unsigned integer
- Description: if true, the kernel will not output his logs on the serial port. The value of this key is provided by the `kernel_log_disable_serial_port` from the Blastproof boot config
**disable_serial_port:**
- Type: boolean, aka 1 byte unsigned integer
- Description: define if the kernel should allow serial port usage. The value of this key is provided by the `kernel_disable_serial_port` from the Blastproof boot config
**log_ring_size:**
- Type: 2 bytes unsigned integer
- Description: define the amount of pages used for logging ring buffer

66
docs/shelter/bootcontract.md Normal file
View File

@@ -0,0 +1,66 @@
# Shelter boot contract
## Introduction
The Shelter boot contract is the list of steps necessary in order to successfully boot Shelter.
## Prerequisites
We assume the following:
- you understand the concept of pages table pool, Shelter boot configuration and Shelter memory map
- you have a bootloader capable of mapping things into into pages table pool
- your bootloader is capable of generating Shelter boot configuration and Shelter memory map
- your bootloader is capable of allocating or call the UEFI firmware to allocates pages to copy data into them
- your bootloader is capable of parsing VYX executable
- your bootloader is capable of accessing the content of the Shelter VYX executable and keycard binary
While this documentation is redacted like a tutorial, this isn't a tutorial nor intended to be understand as one.
## Step 1: allocate pages
Your bootloader should be allocating 4KB physical pages ranges and store their physical start address for the following elements with the following type if you use the `AllocatePages` from the EFI firmware:
- pages for `.text` section should be `EfiLoaderCode` type
- pages for `.data`, `.rodata`, `.bss` sections should be `EfiLoaderData` type
- pages for stack should be `EfiLoaderCoe` type. The standard amount of stack pages is 64
- page(s) for keycard should be `EfiLoaderCode` type. Keycard should fit into one page, but you are free to allocate more if you want
- page(s) for Shelter boot configuration, memory map and pages table pool should be `EfiLoaderCode`. Shelter boot configuration should fit into one page, but you are free to allocate more if you want
- pages for Shelter logging ring should be `EfiLoaderCode` type
Note: even if the logging ring size specified in the configuration is less than 4 pages, at least 4 pages should be allocated, because the kernel logs some things before loading the boot configuration that specify that log in ring buffer is disabled
## Step 2: prepare pages
The following steps should be completed before mapping the pages:
- zeroing the `.bss` section pages range
- zeroing the stack pages range
- zeroing the entire memory map pages range
- zeroing the entire logging ring buffer
- copying the content of the `.text`, `.data` and `.rodata` section inside their respectives pages range
- copying the content of Keycard inside his pages range
- copying the content of the boot configuration inside his pages range
## Step 3: mapping pages
The prerequisites to this step is having a PTP-compatible page mapping function. A working implementation can be found in the `Blastproof/src/libs/src/vyx.c` file.
The following pages ranges will be mapped at the following virtual address in the following order:
- the pages table pool pages range should be mapped first at the virtual address computed like this `text_base + text_size + data_size + rodata_size + bss_size + 0x200000` where `text_base` is the base VA of the `.text` section and the differents sizes are the size of each section, padded to the nearest 4096 bytes (normally it's already the case in the VYX executable header). It should be mapped in read-write with no execution permissions
- the `.text` section should be mapped at the VA indicated by `text_base`, with execution but read-only permissions
- the `.data` section should be mapped at the VA indicated by `text_base + text_size`, with no execution and read write permissions
- the `.rodata` section should be mapped at the VA indicated by `text_base + text_size + data_size` with no execution and read-only permissions
- the `.bss` section should be mapped at the VA indicated by `text_base + text_size + data_size + rodata_size`, with no execution and read-write permissions
- the 64 pages of the stack should be mapped at the VA indicated by `stack_base`, with no execution and read-write permissions
- Keycard pages ranges should be identity mapped to the same virtual address as his physical address, with execution but read-only permissions
- Shelter boot configuration pages should be mapped at VA `0x00180000`, with no execution and read-write permissions
- memory map pages should be mapped at VA `0x00190000`, with no execution and read-only permissions
- logging ring pages should be mapped at VA `0xFFFFFFFFF0000000`, with no execution and read-write permissions
## Step 4: completing configuration
The boot configuration should be completed with the PTP physical and virtual address as well as the level of the counter allocator of the PTP. Don't forget to update the boot configuration already copied inside the boot configuration pages range.
## Step 5: generating memory map
After obtaining the EFI memory map, the memory map should be generated inside her dedicated pages range. The complete syntax of the Shelter memory map can be found [here](memmap.md).
## Step 6: jump
Once all of this is done, the bootloader can exit the EFI boot services, prepare the register and jump to Keycard. The details about completing the registers is provided [here](keycard.md).

89
docs/shelter/bootprocess.md Normal file
View File

@@ -0,0 +1,89 @@
# Shelter Boot Process
## Introduction
Shelter has a very precise boot process to ensure everything goes to plan. In this file, we detail everything that is done for the kernel boot but it doesn't include how to use the kernel API, algorithms, etc. For that, please dig through the documentation.
## 1) Basic kernel initialization
1.1) Print kernel version
1.2) Detect and parse boot config
1.3) Load basic settings
1.3.1) Load logging ring size
1.3.2) Load serial port settings
1.3.3) Load PTP virtual address
1.3.4) Load kernel log level
---
Starting here, all logs go through the standard log API.
---
## 2) Memory subsystem initialization
### 2.1) Page subsystem initialization
2.1.1) Copying memory map to a secured buffer
2.1.2) Initializing current PTP using boot configuration information
2.1.3) Parsing and validating memory map, initializing physical pages bitmap
### 2.2) Pez dependencies subsystems initialization
2.2.1) Initializing physical region slab allocator
2.2.1.a) Initializing slab allocator structure
2.2.1.b) Adding first slab and verifying slab metadata
2.2.2) Initializing virtual region slab allocator
2.2.2.a) Initializing slab allocator structure
2.2.2.b) Adding first slab and verifying slab metadata
2.2.3) Initializing radix tree node slab allocator
2.2.3.a) Initializing page block allocator structure for slab allocator
2.2.3.b) Initializing slab allocator structure
2.2.3.c) Adding first slab and verifying slab metadata
2.2.4) Benchmarking and testing all slabs allocator
2.2.5) Benchmarking and testing radix tree subsystem
## 2.3) Pez physical subsystem initialization
2.3.1) Creating Pez physical allocation plane
2.3.2) Benchmarking and testing Pez physical allocation plane
## 2.4) Pez virtual subsystem initialization
2.4.1) Creating Pez virtual allocation plane for kernel heap
(Pez virtual plane for kernel heap isn't benchmarked and tested because it's exactly the same algorithm and metadatas under the hood)
## 2.5) Kernel heap initialization
2.5.1) Initializing kernel heap structure
2.5.2) For each slab allocator inside kernel heap:
2.5.2.a) Initializing corresponding page block allocator structure
2.5.2.b) Initializing corresponding slab allocator structure
2.5.2.c) Adding first slab and verifying slab metadata
2.5.3) Benchmarking and testing kernel heap

13
docs/shelter/cpu/asmint.md Normal file
View File

@@ -0,0 +1,13 @@
# ASM instructions
## Introduction
While the Shelter kernel is made in C, somes basic CPU functions require using ASM instructions. For that, we define wrappers that can be used anywhere in the kernel. These wrappers are defined inside the header file `shelter/lib/include/cpu/asm.h` which act as the only place in the entire kernel where the `__asm__` should be used. The subsystem prefix is `sh_asm_`.
## Overview
Here is a list of all the wrappers available. They are all defined as `static inline` function:
- `inb`: take a `sh_uint16` in input for specifying the port and return the inputed byte
- `outb`: take a `sh_uint16` for specifying the port and a `sh_uint8` for specifying the byte to output
- `rdtsc`: take no arguments and return the current TSC as a `sh_uint64`
- `invlpg`: take a `void *` as an address and return nothing

11
docs/shelter/cpu/cpuabstract.md Normal file
View File

@@ -0,0 +1,11 @@
# CPU Abstraction
## Introduction
This component of the Shelter kernel allow for basic CPU functions abstractions, like ASM instruction, serial output and TSC utilities.
## Summary
1) [ASM instructions](asmint.md)
2) [Serial outputing API](serial.md)
3) [TSC API](tsc.md)

12
docs/shelter/cpu/serial.md Normal file
View File

@@ -0,0 +1,12 @@
# Serial outputing API
## Introduction
Shelter provide an abstraction around the serial port for outputing on it. This should act as the central point for outputing on the serial. The API is defined in `shelter/lib/include/cpu/serial.h` and implemented in `shelter/lib/src/cpu/serial.c`. This API obey the `serial_port_disabled` killswitch defined in kernel boot configuration. The subsystem prefix is `sh_serial_`.
## Overview
The API define the following elements:
- `sh_serial_load_serial_port_setting`: should be used only once at the start of the boot process to load the serial port setting using the argument `is_disabled` which is a `sh_bool`
- `sh_serial_send_byte`: safely (wait for previous byte to finish being send) send a byte to the serial port
The default and only serial port available is COM1.

20
docs/shelter/cpu/tsc.md Normal file
View File

@@ -0,0 +1,20 @@
# TSC API
## Introduction
In order to be able to measure and approximate time as soon as the kernel start the boot process, the TSC API is implemented in a volontary minimal way. The TSC API is defined inside `shelter/lib/include/cpu/tsc.h` and implemented inside `shelter/lib/src/cpu/tsc.c`. The API prefix is `sh_tsc_`.
## Overview
The TSC API being intented for measuring time during the boot process, the provided features are extremely basic. In order for any TSC value to start at 0, we define two concept:
- `kernel_init_tsc`: a TSC value initialized at the very start of the kernel boot process
- `kernel_current_tsc`: a TSC value which is the result of `kernel_init_tsc` substracted to the value returned by `sh_asm_rdtsc()`
## API content
The API define the following elements:
- `sh_tsc_TSC_VALUE`: a value of the TSC register, which is a wrapper of a `sh_uint64`
- `sh_tsc_read_value()`: an inline function reading the TSC register using `sh_asm_rdtsc()`, return a `sh_tsc_TSC_VALUE`
- `sh_tsc_init_tsc()`: a function initializing `kernel_init_tsc`. This function should only be called once as soon as the kernel start. Return a `SH_STATUS`
- `sh_tsc_get_kernel_init_tsc()`: return `kernel_init_tsc` under a `sh_tsc_TSC_VALUE`
- `sh_tsc_get_kernel_current_tsc()`: return the result of `sh_asm_rdtsc()` less `kernel_init_tsc`, under a `sh_tsc_TSC_VALUE`

35
docs/shelter/index.md Normal file
View File

@@ -0,0 +1,35 @@
# Shelter Docs
## Introduction
Shelter is an hybrid kernel developped alongside and for the Vystem project.
## Principles
Shelter stricly follow these principles:
- absolutely no external code inside it. All the code that run in ring 0 should be sovereign
- absolutely no UNIX dependency on design philosophy. Shelter follow his own rules for kernel design (but doesn't forbid itself to reuse somes UNIX concepts), even if it mean spending years designing, refining and standardizing them
- a very strict boot process that ensure that everything goes well during the system boot
- fully made in C
- focuses on auditability, self-testing and benchmarking, and determinism
- use his own algorithms for various very important components of every kernel (mainly memory management, scheduling, etc), except on cryptographic algorithms, which follow a very strict integration processus
The Shelter kernel is currently in very early developpement stage and isn't suitable at all for every-day usage.
## Summary
- Key principles to know in order to boot Shelter
- [Kernel bootstrapping using keycard](keycard.md)
- [Introduction to Pages Table Pool](ptp.md)
- [Shelter kernel boot configuration](bootconfig.md)
- [Shelter kernel memory map layout](memmap.md)
- [Shelter boot contract](bootcontract.md)
- [Shelter Boot Process](bootprocess.md)
- [Naming scheme](naming.md)
- Detailled components
- [CPU abstraction](cpu/cpuabstract.md)
- [Shelter standard library](std/std.md)
- [Kernel-specific APIs](kernel/kernel.md)
- [Memory subsystem](memory/index.md)
- [Test-and-benchmark framework](tab.md)
- [Benchmark results](bench.md)

3
docs/shelter/kernel/config.md Normal file
View File

@@ -0,0 +1,3 @@
# Config parsing
In Shelter, the parsing of the config is made by the `SH_STATUS sh_conf_get_boot_config(sh_conf_BOOT_CONFIG **config)` function. It allow for quick and secure configuration parsing. It create a `sh_conf_BOOT_CONFIG` object. These elements are defined inside `shelter/lib/include/kernel/conf.h` and implemented inside `shelter/lib/src/kernel/conf.c`. For the full configuration documentation, please see [boot configuration documentation](../bootconfig.md).

11
docs/shelter/kernel/kernel.md Normal file
View File

@@ -0,0 +1,11 @@
# Kernel-specific APIs
## Introduction
The Shelter kernel define his own specific APIs like logging and configuration parsing.
## Summary
1) [Log API](log.md)
2) [Config parsing](config.md)
3) [Tests utilities](testutils.md)

137
docs/shelter/kernel/log.md Normal file
View File

@@ -0,0 +1,137 @@
# Log API
## Introduction
The logging API is one of the most important of the entire kernel: it allow it to output informations with various level of importance and various sources.
## Output methods
The logging API can log informations with two differents methods:
- serial output: use the serial API to output in real time all the provided informations. Can be disabled by setting it to true using the `sh_log_load_serial_setting(sh_bool is_disabled)` function. The serial API being disabled, this method can also be disabled by that.
- ring buffer output: use the ring buffer API to store all the provided informations into a ring buffer, mapped by the bootloader. Can be disabled by setting the `logging_ring_size` boot configuration key to 0
The following functions provide way to set and get those settings:
- `void sh_log_load_serial_setting(sh_bool is_disabled)`: specify if serial log outputting should be disabled
- `sh_bool sh_log_get_serial_setting()` return the activation status of serial log outputting (`SH_TRUE` mean disabled)
- `void sh_log_load_logging_ring_size(sh_uint16 pages_count)`: set logging ring size
- `sh_uint32 sh_log_get_logging_ring_size()`: return logging ring size
- `sh_uint64 sh_log_get_total_bytes_written()`: return total amount of bytes written inside logging ring
## Output primitives
The following functions doesn't care about log level or log source and are specialized into specific data type:
- `void sh_log_byte(sh_uint8 byte)`: log a byte, basic entry point of the entire logging system
- `SH_STATUS sh_log_string(const char* str)`: log a table of characters
Here are the numbers logging functions under their naming scheme:
```
sh_log_(u)int[8/16/32/64](_hex)
```
Basically, there is 12 functions that can output numbers, in various sizes (8 to 64 bits), signed or unsigned, in decimal or hexadecimal. The hexadecimal variant doesn't exist for signed integers. Examples: `sh_log_uint8`, `sh_log_int16`, `sh_log_uint64_hex`...
There are two mores functions:
- `SH_STATUS sh_log_uint64_hex_fixed(sh_uint64 n)`: log a `sh_uint64` in hexadecimal without removing useless zeros, useful for logging pointers and address
- `SH_STATUS sh_log_double(double value)`: log a double encoded in decimal
## Output channels and log level
The Shelter logging API provide higher level functions to output on specific log channels. Here are all the log channels:
- `SH_LOG_DEBUG`: log level 0
- `SH_LOG_LOG`: log level 1
- `SH_LOG_WARNING`: log level 2
- `SH_LOG_ERROR`: log level 3
- `SH_LOG_CRITICAL`: log level 4
- `SH_LOG_FATAL`: log level 5
- `SH_LOG_TEST`: log level 6
Each log channel has a log level. The log level is set by the `log_level` boot configuration key. The log API will only output log messages with log level equal or higher to current log level. This restriction doesn't apply to log channel `SH_LOG_TEST`.
The `sh_log_OUTPUT_TYPE` type is used to store output type by log level. It's a wrapper of `sh_uint8`. The `sh_log_output_type_valid(sh_log_OUTPUT_TYPE t)` can be used to check if a log level is valid.
Two functions allow to set and get the current log level:
- `SH_STATUS sh_log_load_log_level(sh_uint8 log_level)`
- `sh_uint8 sh_log_get_log_level()`
## Output sources
The Shelter logging API also require a source in order to log informations. Here are all the possible sources:
- `SH_LOG_SOURCE_MAIN`
- `SH_LOG_SOURCE_CONF`
- `SH_LOG_SOURCE_PAGE`
- `SH_LOG_SOURCE_SLAB`
- `SH_LOG_SOURCE_TEST`
- `SH_LOG_SOURCE_PEZ`
- `SH_LOG_SOURCE_PBA`
- `SH_LOG_SOURCE_HEAP`
- `SH_LOG_SOURCE_STD`
The `sh_log_OUTPUT_SOURCE` type is used to store log source. It's a wrapper of `sh_uint16`. The `sh_log_output_source_valid(sh_log_OUTPUT_SOURCE s)` can be used to check if a log level is valid.
## Output payloads
In order to compile all of those informations into one payload, the logging API provide this structure :
``` C
typedef struct {
sh_log_OUTPUT_TYPE output_type;
sh_log_OUTPUT_SOURCE output_source;
sh_tsc_TSC_VALUE tsc_value;
const char* message_pointer;
} sh_log_OUTPUT_PAYLOAD;
```
This allow to contain the message content, the output channel, the output source and the TSC value of any log message.
There is two functions that can manipulate output payloads:
- `SH_STATUS sh_log_payload(sh_log_OUTPUT_PAYLOAD *payload)`: output a payload
- `SH_STATUS sh_log_payload_format(sh_log_OUTPUT_PAYLOAD *payload,va_list args)`: format then output a payload, not directly callable, see below for more informations
## Higher levels logging functions
The following functions will automatically create the payload depending on the parameters they receive.
### Logging with automatic line break
These functions are named like this:
```
sh_log_[test/debug/log/warning/error/critical/fatal]
```
They all need two arguments in the following order (except `sh_log_test` which only need the first one):
- `const char* str`: the text to log
- `sh_log_OUTPUT_SOURCE source`: the source of the log message
These functions will automatically add the `\n` characters at the end of each log message.
### Logging without automatic line break
These functions are named like this:
```
sh_log_l[test/debug/log/warning/error/critical/fatal]
```
They all need two arguments in the following order (except `sh_log_ltest` which only need the first one):
- `const char* str`: the text to log
- `sh_log_OUTPUT_SOURCE source`: the source of the log message
### Logging with formating
In order to allow for formatting log messages, the log API provide functions named like this:
```
sh_log_f[test/debug/log/warning/error/critical/fatal]
```
They all need at least two arguments in the following order (except `sh_log_ltest` which only need the second one):
- `sh_log_OUTPUT_SOURCE source`: the source of the log message
- `const char* format`: the format text
- all the others argument after that are used as content for formatting the log message
All of these functions use `sh_log_payload_format()` under the hood which itself use `SH_STATUS sh_log_format(const char* format,va_list args)` to format log messages.
The format syntax is as follow:
- `%s`: insert a string made of `char`, passed as `char*`
- `%d`: insert a double
- `%x`: insert a integer that need to be provided as a `sh_uint64`, format it as hexadecimal without removing useless zeros. Used to log addresses
- `%c`: insert a `char`
- `%%`: insert a `%`
There is also a specific syntax for formatting integers. It's start with the `%` symbol, then need a amount of bytes (can be 1, 2, 4 or 8) and a output format: `u` for unsigned decimal, `s` for signed decimal, `U` for unsigned decimal. Example: `%1s` output a `sh_int8` in decimal, `%2u` output a `sh_uint16` in decimal, `%4U` output a unsigned `sh_uint32` in hexadecimal.

23
docs/shelter/kernel/testutils.md Normal file
View File

@@ -0,0 +1,23 @@
# Tests utilities
## Introduction
To support the test-and-benchmark framework, the Shelter kernel include a common base of utilities that provide to all tests a basic way to store and expose their results. This file only cover the basic utilities usable by all tests, not the detail and typical results of each test. These utilities are defined inside `shelter/lib/include/kernel/tests/test_utils.h` and implemented inside `shelter/lib/src/kernel/tests/test_utils.c`.
## Overview
The test-and-benchmark (TAB) framework is responsible for testing differents subsystems to ensure their stability and performances over differents devices. But it also provides a common base for showing results and others little things.
Firsty, it define internally a buffer of 10000 `sh_tsc_TSC_VALUE` for storing cycles count and compute statistics. The pointer to this buffer is obtainable through the `sh_tsc_TSC_VALUE* sh_test_get_tsc_values_buffer_ptr()` function.
Then, the TAB framework also define the `void sh_test_load_iterations_count(sh_uint64 iterations_num)` function. This function load the iterations count defined by the `bench_iterations` boot configuration key. Some tests doesn't support this feature, so this function will only set iterations count for tests that support custom iterations count.
Finally, each test uses the `SH_STATUS sh_test_compute_print_stats(char* benchname,sh_tsc_TSC_VALUE *tsc_value_array,sh_uint64 array_size)` function. This function will compute various statistics and print them like this:
```
Result for benchmark "<benchmark name>" :
Min: <min time> | Med: <median time> | Avg: <average time> | Max: <max time> | Total : <total time> (TSC)
[P00-P90] <min time>-P90 : [##################--] 90%
[P90-P99] P90-P99 : [#-------------------] 9%
[P99-Max] P99-<max time> : [--------------------] 1%
```
Here, `PXX` where `XX` is a percentage mean a percentile.

21
docs/shelter/keycard.md Normal file
View File

@@ -0,0 +1,21 @@
# Kernel bootstrapping using keycard
## Introduction
Shelter being a kernel that need to start with pagging enabled, it need to have a intermediary program to safely jump to it: Keycard. Keycard and Shelter have been designed and thinked for x86-64 only.
## Overview
Keycard is a very simple assembly program with the following role in the following order:
- loading the kernel page table and enabling pagging
- loading and aligning the kernel stack
- jumping to kernel entry point
In order to achieve that, it need to be identity mapped into virtual lower half into the kernel page table, using a physical address provided by UEFI firmware. The bootloader map it into a page (keycard compiled code fit into a single page) with allowed execution but read-only. It also have a kind of 'ABI' which specify which value to put into which register before jumping to Keycard:
- `rax`: should contain the stack top virtual address (VA)
- `rbx`: should contain the physical adrdess of the page table root
- `rcx`: should contain the VA of the kernel entry point
Due to security concerns, the `keycard.bin` file contain the compiled code of Keycard and is stored inside the InitFS with the kernel VYX executable.
The Keycard source code is stored inside `Blastproof/src/keycard.asm`.

37
docs/shelter/memmap.md Normal file
View File

@@ -0,0 +1,37 @@
# Shelter kernel memory map layout
## Introduction
In order to transmit efficiently the EFI memory map to the kernel, Shelter have his own memory map layout that need to be created from the EFI memory map in order for Shelter to understand it. For details about mapping the Shelter memory map inside the kernel virtual memory space, please see [boot contract docs](bootcontract.md).
## Header
The Shelter memory map is first assigned a buffer of 64 kilobytes or 16 pages mapped with non-execution and read only permissions. Each memory map has an header described by the following C structure (sourced from `shelter/lib/include/memory/page.h`)
``` C
#pragma pack(1)
typedef struct {
sh_uint8 sig_start[8];
sh_uint64 entry_count;
sh_uint64 entry_size;
sh_uint8 mmap_syntax_version;
} sh_page_MEMORY_MAP_HEADER;
#pragma pack()
```
Each field is packed inside the struct. The signature must be `SheMmapB`. The number of entry is indicated inside the `entry_count` field. The size of an entry is indicated inside the `entry_size` field. Both of these fields are little endian 8 bytes unsigned integer. The `mmap_syntax_version` is always set to `0x01`, as the current version of the Shelter memory map syntax. The kernel also compute the total size of the memory map (header + all entries) and raise an error if it's above the size of the memory map buffer.
## Entries
After the header, there are entries that describe which part of the physical memory is currently unusable (due to various factors like ACPI table, etc). A entry is described by the following C structure:
``` C
#pragma pack(1)
typedef struct {
sh_uint32 type;
sh_uint64 physical_start;
sh_uint64 pages_count;
sh_uint64 attributes;
} sh_page_MEMORY_MAP_ENTRY;
#pragma pack()
```
Each field is packed inside the struct. The `type` field is a 4 bytes unsigned integer that give the type of the region, as the EFI specification specify it. The `physical_start` field is a 8 bytes unsigned integer that contain the physical address of the start of the region. The `pages_count` field is a 8 bytes unsigned integer that contain the number of 4 kilobytes pages that constitute the region. The `attributes` field contain any other attributes provided by the UEFI firmware. All fields are in little endian.

45
docs/shelter/memory/heap.md Normal file
View File

@@ -0,0 +1,45 @@
# Kernel heap manager
## Introduction
In order to allocate small-sized object or medium-sized buffers (approximately between one and one hundred pages), the lower half have been designed for hosting the heap of the kernel. This is an early design subject to changes for perfomances and/or robustess in the coming updates. It's defined inside `shelter/lib/include/memory/heap.h` and implemented inside `shelter/lib/src/memory/heap.c`.
## Overview
The kernel heap manager is based on a dual allocation strategy:
- small objects, less or equal to 1024 bytes goes into slab allocators
- medium buffers, greater than 1024 bytes, are allocated using the Pez allocator for physical and virtual planes management
The heap metadatas are stored inside the `sh_heap_KERNEL_HEAP` structure. The `alloc_size_tree` field is a radix tree mapping virtual pages offset of any pages mapping with their size in pages. This allow for quick retrieval of any pages allocations on the heap.
## Slab allocators
These are described into [generic slab allocator documentation](slabs.md#generic-slab-allocators). Each one is setted up with a PBA containing 12 terabytes of virtual memory
Here are a table describing the virtual pages ranges of all allocators:
Object size range (bytes) | Allocator level | Virtual pages range start address
--- | --- | ---
1-8 | 0 | `0x0000200000000000`
9-16 | 1 | `0x00002C0000000000`
17-32 | 2 | `0x0000380000000000`
33-64 | 3 | `0x0000440000000000`
65-128 | 4 | `0x0000500000000000`
129-256 | 5 | `0x00005C0000000000`
257-512 | 6 | `0x0000680000000000`
513-1024 | 7 | `0x0000740000000000`
## Pages allocations
If the needed amount of bytes is greater than 1024 bytes, pages are allocated on the heap. The number of pages allocated is the nearest amount that can satisfy the demand. The pages are allocated inside a virtual pages range starting at `0x0000100000000000` and contain 16 terabytes of virtual memory.
## API
The heap manager provide the following API, that should only be called by the abstraction provided by `sh_malloc`, except the first three one:
- `sh_heap_KERNEL_HEAP *sh_heap_get_default_heap()`: return a pointer to the kernel heap structure
- `void sh_heap_load_default_heap(sh_heap_KERNEL_HEAP *heap)`: set the pointer to the kernel heap structure
- `SH_STATUS sh_heap_init_heap(sh_pez_PHYSICAL_PLANE *phys_plane,sh_pez_VIRTUAL_PLANE *virt_plane,sh_page_PAGE_TABLE_POOL *kernel_ptp,sh_heap_KERNEL_HEAP *kernel_heap)`: initialize the heap structure, the calling entity need to setup and place the generic slab allocators pointers manually inside the returned structure
- `SH_STATUS sh_heap_allocate_pages(sh_uint32 pages_count,sh_page_VIRTUAL_ADDRESS *address)`: allocate a certain amount of pages, provided by the Pez physical backend and mapped on the heap pages allocation range
- `SH_STATUS sh_heap_free_pages(sh_page_VIRTUAL_ADDRESS va)`: free an allocated region of pages from the heap. Return the pages to Pez physical backend
- `SH_STATUS sh_heap_allocate_object(sh_uint32 size_bytes,sh_page_VIRTUAL_ADDRESS *address)`: allocate a certain object based on his size, automatically uses the most adapted generic slab allocator. Will refuse any size superior to 1024 bytes
- `SH_STATUS sh_heap_free_object(sh_page_VIRTUAL_ADDRESS va)`: free a certain object by providing his VA. Automatically free from the previously used slab allocator. Will refuse any VA not in the range of the object allocation range

16
docs/shelter/memory/index.md Normal file
View File

@@ -0,0 +1,16 @@
# Memory subsystem
## Introduction
The memory subsystem is responsible for handling tasks like physical pages allocations, virtual memory management, initial memory map analysis, pages mapping and unmapping and kernel heap management.
## Summary
1) [Page subsystem](page.md)
2) [Virtual memory layout](vmemlayout.md)
3) [Ring buffer](ring.md)
4) [Pages block allocator](pba.md)
5) [Slabs allocator](slabs.md)
6) [Radix trees subsystem](radix.md)
7) [Pez plane manager](pez.md)
8) [Kernel heap manager](heap.md)

90
docs/shelter/memory/page.md Normal file
View File

@@ -0,0 +1,90 @@
# Page subsystem
## Introduction
The Page subsystem is a part of the memory subsystem responsible for managing initial memory map analysis, physical pages bitmap, pages tables pool management, mapping and unmapping pages as well as memory allocations before Pez initialization and memory statistics. It's defined inside `shelter/lib/include/memory/page.h` and implemented inside `shelter/lib/src/memory/page.c`
## Macros and types
The Page header defines a few usefuls macros like:
- the start and end of the kernel big allocations virtual range
- the size of a page
- the max memory count in bytes as well as the max physical pages count
- the standard size of a pages table pool pages range
- all the types of EFI memory type
- the null value of physical and virtual addresses
- the flags for pages table entry
The Page header also defines the following types:
- `sh_page_MEMORY_TYPE`: a wrapper of `sh_uint32`
- `sh_page_PHYSICAL_ADDRESS` and `sh_page_VIRTUAL_ADDRESS`: wrappers of `sh_uint64`
## Structures
The Page header defines the following structure:
- `sh_page_MEMORY_MAP_ENTRY` and `sh_page_MEMORY_MAP_HEADER`: internal structures used for memory map analysis
- `sh_page_PAGE_TABLE_POOL`: structure for pages table pool, containing the physical and virtual address of a pages table pool as well as the internal bitmap for the internal allocator
- `sh_page_MEM_STATS`: a structure filled by a specific function to get statistics on physical memory using physical pages bitmap
## Pages table pool
It's recommended to read [pages table pool documentation](../ptp.md) before reading this.
The Page subsystem provide the following functions regarding PTP management:
- `SH_STATUS sh_page_load_boot_ptp_va(sh_page_VIRTUAL_ADDRESS pt_pool_va)`: load the virtual address of the boot PTP provided by bootloader. This VA is the address of the root PML4, not the address of the PTP structure. Should only be used once
- `sh_page_VIRTUAL_ADDRESS sh_page_get_boot_ptp_va()`: return the pointer to root PM4 stored inside boot PTP
- `SH_STATUS sh_page_init_ptp(sh_page_PHYSICAL_ADDRESS ptp_pa,sh_page_VIRTUAL_ADDRESS ptp_va,sh_uint64 initial_fill_level,sh_page_PAGE_TABLE_POOL *page_table_pool)`: initialize the boot PTP structure with the fill level of the PTP, provided by bootloader. Doesn't allocate pages for a new PTP but it use the pages already allocated for the boot PTP. Should only be used once for boot PTP
- `SH_STATUS sh_page_dump_ptp_bitmap(sh_page_PAGE_TABLE_POOL *ptp)`: dump the PTP bitmap on the log output, intended for debug purposes
- `sh_page_PHYSICAL_ADDRESS sh_page_ptp_alloc_one_page(sh_page_PAGE_TABLE_POOL *pt_pool)`: allocate one page from the PTP internal bitmap allocator
- `static inline sh_uint64 *sh_page_ptp_pa_to_va(sh_page_PAGE_TABLE_POOL *ptp,sh_uint64 pa)`: convert a PA from the PTP pages range to a usable VA using the current PTP
- `SH_STATUS sh_page_ptp_va_to_pa(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_page_PHYSICAL_ADDRESS *pa)`: return equivalent physical address by searching inside provided PTP
## Memory map analysis
The Page subsystem is responsible for the initial memory map analysis. It provide the following functions:
- `SH_STATUS sh_page_copy_memory_map()`: copy the memory map from her original VA to a dedicated buffer
- `SH_STATUS sh_page_check_memory_map()`: check for memory map signature and detect eventual buffers overflow
- `SH_STATUS sh_page_analyse_memory_map(sh_page_PAGE_TABLE_POOL *ptp)`: analyse memory map, initialize physical pages bitmap by allocating into a free area of the physical memory layout. This allow for dynamic sizing for physical pages bitmap.
All three functions should be run in order and only once.
## Physical pages bitmap
In order to ensure a reliable source of thruth and early allocations, the Page subsystem manage a physical pages bitmap where one bit on mean one page occupied.
The Page subsystem provides the following functions regarding physical pages bitmap:
- `SH_STATUS sh_page_set_pages_range_bitmap(sh_uint8 *bitmap,sh_uint64 page_count_in_bitmap,sh_uint64 page_index,sh_uint64 page_count,sh_bool state)`: allow for manipulation of any bitmap, but should only be used for physical pages bitmap
- `static inline sh_bool sh_page_is_allocated(sh_uint8 *bitmap,sh_uint64 page_index)`: return the status of a specific page inside the bitmap
- `sh_page_VIRTUAL_ADDRESS sh_page_get_physical_bitmap_ptr()`: return physical pages bitmap pointer
## Pages mapping and unmapping
The Page subsystem provide the following functions regarding pages mapping:
- `SH_STATUS sh_page_map_one_page_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_page_PHYSICAL_ADDRESS pa,sh_uint64 flags)`: map one page inside the pages table contained in the provided PTP
- `SH_STATUS sh_page_is_va_mapped_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va)`: return `SH_STATUS_VA_NOT_MAPPED` or `SH_STATUS_VA_MAPPED`, which aren't errors, according to page mapping state
- `SH_STATUS sh_page_is_va_range_mapped_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_uint64 size_bytes)`: return `SH_STATUS_VA_NOT_MAPPED`, `SH_STATUS_VA_FULLY_MAPPED` or `SH_STATUS_VA_PARTIALLY_MAPPED` according to pages range mapping state
- `SH_STATUS sh_page_map_contiguous_pages_range_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_page_PHYSICAL_ADDRESS pa,sh_uint64 flags,sh_uint64 size_bytes)`: map a pages range that has to be contiguous virtually and physically. VAs availability is checked, not physicals pages availability
- `SH_STATUS sh_page_unmap_one_page_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va)`: unmap one page from the pages table contained inside provided PTP
- `SH_STATUS sh_page_unmap_contiguous_pages_range_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_uint64 size_bytes)`: unmap a pages range from the pages table inside provided PTP. Both physical and virtual ranges have to be contiguous. Virtual pages range is checked, not physical ranges occupation
## Basic memory allocations
The Page subsystem implement basic memory allocation, available before Pez initialization, using a first fit algorithm.
The Page subsystem provides the following functions regarding basic memory allocations:
- `sh_uint64 sh_page_get_one_page_na()`: return the index of the first available page inside physical pages bitmap
- `SH_STATUS sh_page_search_available_va_range(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS range_base,sh_page_VIRTUAL_ADDRESS range_size_bytes,sh_uint64 size_bytes,sh_page_VIRTUAL_ADDRESS *address_found)`: search a free region inside PTP using provided size and search area boundaries provided
- `SH_STATUS sh_page_search_physical_contiguous_block_na(sh_uint64 pages_needed,sh_page_PHYSICAL_ADDRESS *pa)`: search a free region of provided size inside physical pages bitmap
- `SH_STATUS sh_page_alloc_contiguous(sh_page_PAGE_TABLE_POOL *ptp,sh_uint64 size_bytes,sh_page_VIRTUAL_ADDRESS* va)`: allocate a specified amount of pages and return the start VA of the allocated region
- `SH_STATUS sh_page_alloc_contiguous_extended(sh_page_PAGE_TABLE_POOL *ptp,sh_uint64 size_bytes,sh_page_VIRTUAL_ADDRESS* va,DEFAULT sh_uint64 flags,DEFAULT sh_page_VIRTUAL_ADDRESS va_range_start,DEFAULT sh_uint64 va_range_size_bytes)`: allow for extensive allocations parameters for selecting mapping flags and VAs search range
- `SH_STATUS sh_page_unalloc_one_page(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va)`: free one page. Check if page is mapped before hand. PA is calculated from searching into PTP
- `SH_STATUS sh_page_unalloc_contiguous(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_uint64 size_bytes)`: free a contiguous memory region. Check if the entire region is allocated before hand
This role of memory allocations is lost once Pez is initialized.
## Memory statistics
The Page subsystem provide the following functions regarding memory statistics:
- `sh_uint64 sh_page_get_physical_memory_amount_pages()`: return amount of physical memory installed in pages
- `sh_uint64 sh_page_get_physical_memory_amount_bytes()`: return amount of physical memory installed in bytes
- `SH_STATUS sh_page_get_memory_stats(sh_page_MEM_STATS *mem_stats)`: provide a extensive amount of statistics on physical memory.
For a more human-readable output, the function `sh_log_mem_stats` can be used.

19
docs/shelter/memory/pba.md Normal file
View File

@@ -0,0 +1,19 @@
# Pages block allocator
## Introduction
The pages block allocator is a simple configurable bumb allocator designed to be the pages source of slabs allocator inside the kernel. It's defined inside `shelter/lib/include/memory/pba.h` and implemented inside `shelter/lib/src/memory/pba.c`.
## Overview
A pages block allocator is conifigurable with various parameters.
Firstly, it need to know the size of each block that are allocated. This is specified inside the `block_pages` field of the PBA structure, under the `sh_uint64` type.
The `start_va` field, a `sh_page_VIRTUAL_ADDRESS`, specify the starting VA of the virtual pages ranges dedicated to the allocator. It need to be aligned on a boundary of a region `block_pages` pages. The `total_pages` field, a `sh_uint64`, specify the size of the decicated virtual pages range, in pages. It need to be a multiple of `block_pages`.
Finally, the `block_count` field, a `sh_uint64`, store the amount of blocks already allocated. The `max_blocks` field, a `sh_uint64`, is computed at initialization and allow to know instantly if the PBA is full, which happen when the dedicated virtual pages range is fully mapped.
All those fields are stored inside the `sh_pba_PAGE_BLOCK_ALLOCATOR` structure. This structure can be manipulated by the following functions:
- `SH_STATUS sh_pba_init(sh_pba_PAGE_BLOCK_ALLOCATOR *pba,sh_page_VIRTUAL_ADDRESS start_va,sh_uint64 area_pages_amount,sh_uint64 block_pages)`: initialize a PBA, using provided parameters. It ensure the previously specified constraints are met with the provided parameters
- `SH_STATUS sh_pba_alloc(sh_pba_PAGE_BLOCK_ALLOCATOR *pba,sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS *ptr)`: allocate a block from the PBA. Take care of the physical region search (using Page subsystem or Pez if it's initialized), VA computation (using provided virtual pages range) and pages mapping, as well as physical pages bitmap update if Pez isn't initialized.

86
docs/shelter/memory/pez.md Normal file
View File

@@ -0,0 +1,86 @@
# Pez plane manager
## Introduction
Pez is the allocator that manages physical pages allocation and virtual pages ranges allocation. It's a custom design, (normally) never seen before. It's defined into `shelter/lib/include/memory/pez/pez.h` and implemented into `shelter/lib/src/memory/pez/pez.c`.
## Planes
Pez manages memory using planes. It can be as granular as a 4KB pages. Since Pez uses `sh_uint32` for representating pages index and regions sizes, a plane can technically manages up to 16 terabytes of memory.
In the implementation, Pez separate the physical plane from virtual planes. They work the same in their logic and algorithm but use differents source of truth and the virtual plane can provide an offset to each address it allocate.
The physical plane uses the physical pages bitmap that the Page subsystem maintain as source of truth. While the source of truth for virtual planes are the corresponding pages mapping into the virtual pages range the virtual plane manages, for the moment, only empty (not allocated at all) virtual planes can be created.
## Metadatas
### Regions objects
The Pez allocator uses regions objects allocated from their dedicated (physical or virtual) slab allocators.
The physical version is:
``` C
#pragma pack(1)
typedef struct {
sh_uint32 start_page_index;
sh_uint32 region_size_pages;
sh_uint8 next_region_index[3];
sh_uint8 flags;
} sh_pez_REGION_PHYSICAL_OBJECT;
#pragma pack()
```
The virtual version is:
``` C
#pragma pack(1)
typedef struct {
sh_uint32 start_page_index;
sh_uint32 region_size_pages;
sh_uint32 next_region_index;
} sh_pez_REGION_VIRTUAL_OBJECT;
#pragma pack()
```
The `start_page_index` field is the index of the first page of the region, starting at 0 from the start of the plane. The `region_size_pages` field is the size of the region in pages. The `next_region_index` field is the index of the next region in the size list. It's one byte longer on virtual region object to account for the 29 bits index. The `flags` field, while not being used in this implementation, can be used to store flags.
These regions objects represente free regions that are ready to be allocated.
### Size lists radix tree
Each plane maintain lists of free regions by their sizes. These are call size lists. They are linked through the `next_region_index` field, using their object index inside their respective slab allocator. These lists only contain regions of exact size. To signal the end of a size list, the last element in the list have their `next_region_index` field set to 0. This prevent the first slot of any of the slab inside the slab allocator to be allocated.
Each plane maintain a radix tree that link sizes of free regions (in page count) to indexes of the first region in this size list. This radix tree is called the size lists radix tree and is 8 levels deep.
This architecture allow for two allocations methods that are stricly time-constant or nearly time-constant:
Each time the allocator need to allocate a region, it search for this size into the corresponding size list. Two possibles results:
- if a free region of exactly this size is found, it's popped of the size list and allocated
- if no free region of exactly this size is found, the algorithm search for another region immediatly above in size of the requested size using the backtracking algorithm. This free region is popped of the size list and splitted in two. The first part is allocated and the second part is inserted into the corresponding size list.
This allocation scheme allow for zero internal fragmentation at the granularity of 4KB pages.
### Boundary radix tree
Each plane maintain what we call a boundary radix tree. This radix tree map the start and end pages index in the plane of each free regions to a boundary entry. For one-page free region, there is only one boundary. The two boundaries for free regions with more than one page are strictly identical.
A boundary is structured like this:
- The lower 32 bits of the boundary are the region object index of the region to which the boundary belong
- The upper 32 bits of the boundary are the previous region object index in the size list of the region to which the boundary belong
This radix tree allow for two things:
- constant-time passive fusion of free regions with deallocated occupied regions, at each free of occupied region. The external fragmentation is repaired as soon as the allocated region that cause it is deallocated.
- instant element suppression of any free region in any size lists
## API
The Pez allocator implementation inside the Shelter kernel provide the following functions:
- `void sh_pez_set_available()`: executed once to signal that Pez has taken over the role of Page for allocating pages
- `sh_bool sh_pez_is_available()`: return the current status of availability of Pez
- `sh_pez_PHYSICAL_PLANE* sh_pez_get_reference_phys_plane()`: return the reference physical plane that is used to manage physical memory
- `SH_STATUS sh_pez_init_physical_plane(sh_uint8 *physical_bitmap,sh_uint64 physical_page_count,sh_slab_reg_phys_SLAB_ALLOCATOR *slab_reg_phys,struct sh_slab_radix_node_SLAB_ALLOCATOR *slab_radix_node,sh_page_PAGE_TABLE_POOL *kernel_ptp,sh_pez_PHYSICAL_PLANE *phys_plane)`: initialize the physical plane by scanning the physical pages bitmap
- `SH_STATUS sh_pez_alloc_physical_pages(sh_pez_PHYSICAL_PLANE *phys_plane,sh_uint32 pages_count,sh_page_PHYSICAL_ADDRESS *address)`: allocate pages range from the physical plane. Doesn't map the pages at all
- `SH_STATUS sh_pez_free_physical_pages(sh_pez_PHYSICAL_PLANE *phys_plane,sh_page_PHYSICAL_ADDRESS *address,sh_uint32 pages_count)`: free a pages range and return them to the physical plane. Doesn't unmap the pages at all
- `SH_STATUS sh_pez_debug_physical(sh_pez_PHYSICAL_PLANE *phys_plane)`: print debugging information about Pez physical plane, intented for debug
- `SH_STATUS sh_pez_init_virtual_plane(sh_page_VIRTUAL_ADDRESS plane_offset,sh_slab_reg_virt_SLAB_ALLOCATOR *slab_reg_virt,struct sh_slab_radix_node_SLAB_ALLOCATOR *slab_radix_node,sh_page_PAGE_TABLE_POOL *kernel_ptp,sh_page_PAGE_TABLE_POOL *reference_ptp,sh_pez_VIRTUAL_PLANE *virt_plane)`: initialize an empty virtual plane
- `SH_STATUS sh_pez_alloc_virtual_pages(sh_pez_VIRTUAL_PLANE *virt_plane,sh_uint32 pages_count,sh_page_VIRTUAL_ADDRESS *address)`: allocate virtual pages range from the provided virtual plane
- `SH_STATUS sh_pez_free_virtual_pages(sh_pez_VIRTUAL_PLANE *virt_plane,sh_page_VIRTUAL_ADDRESS *address,sh_uint32 pages_count)`: free virtual pages range and return them to the provided virtual plane

37
docs/shelter/memory/radix.md Normal file
View File

@@ -0,0 +1,37 @@
# Radix trees subsystem
## Introduction
The memory subsystem require a capable radix trees to unleash the full power of the Pez allocator. It's defined inside `shelter/lib/include/memory/pez/radix.h` and implemented inside `shelter/lib/src/memory/pez/radix.c`.
## Overview
In Shelter, the radix nodes are 128 bytes wide, capable of storing 16 `sh_uint64` values. This mean all radix trees have a 16 fanout, they filter 4 bits per level, and require 16 levels for 64 bits keys and 8 levels for 32 bits keys.
The node is defined by this sructure:
``` C
typedef struct {
sh_page_VIRTUAL_ADDRESS ptr[16];
} sh_radix_NODE;
```
A radix tree is defined into the following structure:
``` C
typedef struct {
sh_radix_NODE *root_node;
sh_uint8 depth;
} sh_radix_TREE;
```
The API for modifying nodes is as follow:
- `SH_STATUS sh_radix_node_read_value(sh_radix_NODE *node,sh_uint8 index,sh_page_VIRTUAL_ADDRESS* value)`: read a value from the node at the following index
- `SH_STATUS sh_radix_node_set_value(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_radix_NODE *node,sh_uint8 index,sh_page_VIRTUAL_ADDRESS value)`: set the value inside the node at the provided node
The API for manipulating radix trees is as follow:
- `SH_STATUS sh_radix_tree_init(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_radix_TREE *tree,sh_uint8 depth)`: initialize a radix tree, fail if depth is greater than 16. Allocate the root node
- `SH_STATUS sh_radix_tree_get_value(sh_radix_TREE *tree,sh_uint64 key,sh_page_VIRTUAL_ADDRESS *value)`: search in a straight line inside the tree to provide the associated value of a key. Stop and return `SH_STATUS_NOT_FOUND` as soon as it hit an empty pointer where there should be one
- `SH_STATUS sh_radix_tree_insert_value(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_radix_TREE *tree,sh_uint64 key,sh_page_VIRTUAL_ADDRESS value)`: insert a value associated with a key inside the tree. Can allocates new nodes and overwrite previously setted value.
- `SH_STATUS sh_radix_tree_delete_value(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_radix_TREE *tree,sh_uint64 key)`: delete a vale and deallocates all nodes (including leaf) that form a path to the leaf if the deletion make them empty
- `SH_STATUS sh_radix_tree_search_smallest_min_bound(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_radix_TREE *tree,sh_uint64 lower_bound_key,sh_page_VIRTUAL_ADDRESS *value)`: backtracking algorithm used to quickly search the value with a key equal or greater than the provided key. Can't allocate new nodes
The radix trees subsystem uses the same slab allocator for all radix trees.

11
docs/shelter/memory/ring.md Normal file
View File

@@ -0,0 +1,11 @@
# Ring buffer
## Introduction
The memory subsystem provide a simple ring buffer API, mainly used by the log API. This implementation doesn't allow for automatic structure initialization with allocation. The structure has to be manually created for it to work. This ring buffer implementation is defined inside `shelter/lib/include/memory/ring.h` and implemented inside `shelter/lib/src/memory/ring.c`
## Overview
The main structure for a ring buffer is `sh_ring_RING_BUFFER_HEADER`. The function provided by the API are volontary very simple for the moment:
- `SH_STATUS sh_ring_write_byte(sh_ring_RING_BUFFER_HEADER *ring_buffer,sh_uint8 byte)`: write a byte inside provided ring buffer
- `SH_STATUS sh_ring_write_string(sh_ring_RING_BUFFER_HEADER *ring_buffer,char *string)`: write a null-terminated string inside provided ring buffer

91
docs/shelter/memory/slabs.md Normal file
View File

@@ -0,0 +1,91 @@
# Slabs allocator
## Introduction
The memory subsystem provides 4 independant but similars slab allocators dedicated to various purposes.
## Regions objects slabs allocator
These slabs allocator are dedicated to the storage of region objects. These objects are 12 bytes in size. There is two slab allocators that store region objects:
- `slab_reg_phys`: the slab allocator for physical region objets. It uses 24 bits index, making it able to store up to 16 millions objects. Defined in `shelter/lib/include/memory/slabs/slab_reg_phys.h` and implemented in `shelter/lib/src/memory/slabs/slab_reg_phys.c`
- `slab_reg_virt`: the slab allocator for virtual region objets. It uses 29 bits index, making it able to store up to 536 millions objects. Defined in `shelter/lib/include/memory/slabs/slab_reg_virt.h` and implemented in `shelter/lib/src/memory/slabs/slab_reg_virt.c`
They uses 3 pages per slab, capable of storing 1024 objects per slabs. They store the slab header outside, into a dedicated, pre allocated area. The slabs are allocated, reused, but never deallocated.
The slab allocator uses indexes to identify objects slots. These indexes are what the user is given upon allocation and need to be provided back in order to free objects slots. The indexes are composed of two parts:
- index in the slab: 10 lower bits
- slab index: 14 or 19 upper bits
The structures `sh_slab_reg_phys_SLAB_STRUCT` and `sh_slab_reg_virt_SLAB_STRUCT` define the headers of each slab. The structures `sh_slab_reg_phys_SLAB_ALLOCATOR` and `sh_slab_reg_virt_SLAB_ALLOCATOR` are used to store allocators metadata. The `sh_slab_reg_phys_OBJECT_INDEX` and `sh_slab_reg_virt_OBJECT_INDEX` are both wrapper of `sh_uint32`.
The only difference between these two are the amount of slabs they can store and their dedicated virtual pages ranges.
The API for physical region objects slab allocator is as follow:
- `SH_STATUS sh_slab_reg_phys_alloc_init(sh_slab_reg_phys_SLAB_ALLOCATOR* slab_alloc,sh_page_PAGE_TABLE_POOL *ptp)`: initialize the allocator, allocate all the headers but does not add the first slab
- `SH_STATUS sh_slab_reg_phys_add_slab(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_phys_SLAB_STRUCT** out_slab)`: add a new slab, can allocate from Page or Pez subsystem, do all the mapping itself
- `void* sh_slab_reg_phys_ref_to_ptr(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_slab_reg_phys_OBJECT_INDEX ref)`: transform an index into a pointer to the referenced object
- `SH_STATUS sh_slab_reg_phys_alloc(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_phys_OBJECT_INDEX* out_index)`: allocate a new object and return his index. Can allocate new slabs
- `SH_STATUS sh_slab_reg_phys_dealloc(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_slab_reg_phys_OBJECT_INDEX index)`: deallocate a object using the provided index. Doesn't delete slabs that are made empty by objects deallocation
The API for virtual region objects slab allocator is as follow:
- `SH_STATUS sh_slab_reg_virt_alloc_init(sh_slab_reg_virt_SLAB_ALLOCATOR* slab_alloc,sh_page_PAGE_TABLE_POOL *ptp)`: initialize the allocator, allocate all the headers but does not add the first slab
- `SH_STATUS sh_slab_reg_virt_add_slab(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_virt_SLAB_STRUCT** out_slab)`: add a new slab, can allocate from Page or Pez subsystem, do all the mapping itself
- `void* sh_slab_reg_virt_ref_to_ptr(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_slab_reg_virt_OBJECT_INDEX ref)`: transform an index into a pointer to the referenced object
- `SH_STATUS sh_slab_reg_virt_alloc(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_virt_OBJECT_INDEX* out_index)`: allocate a new object and return his index. Can allocate new slabs
- `SH_STATUS sh_slab_reg_virt_dealloc(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_slab_reg_virt_OBJECT_INDEX index)`: deallocate a object using the provided index. Doesn't delete slabs that are made empty by objects deallocation
## PBA-based slab allocators
These slab allocators relies on a PBA to automatically allocate and map slab pages inside their dedicated virtual pages ranges. They doesn't create their own PBA. While this responsability fall on the user, this allow the user to set personnalized virtual pages range, even if they aren't intented to have multiples instances of them at the same time.
These slabs allocator also store the header of each slabs inside the slab, making it possible to allocate slabs headers on the fly. The slab header contain pointers that can form a partial slabs linked list.
Finally, these slab allocators provide infinite slabs without any software limits. They reuse but doesn't deallocate slabs. They also don't use indexes but direct pointers.
### Radix node slab allocator
This allocator use 32 pages slabs, and manage 128 bytes object. The slab contain 1024 objects slots. But due to the space taken by the header, each slab only contain 1006 objects due to the header taking space.
Inside the slab header, the slab allocator also maintain a 16 bit extra object per radix node. It serve at a bitmap for radix nodes.
The `sh_slab_radix_node_SLAB` structure describe the header of each slab, as well as each object slot. The `sh_slab_radix_node_SLAB_ALLOCATOR` structure contain the metadatas of the allocator. The header is 256 bytes wide.
The API for radix nodes slab allocator is as follow:
- `SH_STATUS sh_slab_radix_node_alloc_init(struct sh_slab_radix_node_SLAB_ALLOCATOR* slab_alloc,sh_pba_PAGE_BLOCK_ALLOCATOR *pba)`: initialize the slab allocator, doesn't allocate any slab
- `SH_STATUS sh_slab_radix_node_add_slab(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_radix_node_SLAB** out_slab)`: add a new slab, allocate from the PBA
- `SH_STATUS sh_slab_radix_node_alloc(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_radix_NODE** out_obj)`: allocate a new object and return his pointer. Can allocate new slabs
- `SH_STATUS sh_slab_radix_node_dealloc(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_radix_NODE *object_ptr)`: deallocate a object using the provided pointer. Doesn't delete slabs that are made empty by objects deallocation
- `sh_uint16 *sh_slab_radix_node_get_node_bitmap(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_radix_NODE *object_ptr)`: return a pointer to the bitmap of the node
It's defined inside `shelter/lib/include/memory/slabs/slab_radix_node.h` and implemented inside `shelter/lib/src/memory/slabs/slab_radix_node.c`.
### Generic slab allocators
These slab allocators are used to store objects allocated on the heap. The header is always 128 bytes. Each slab contain 512 objects but the actual size of each slab depend on the level of the allocator:
Level | Object size in bytes | Amount of pages per slab | Amount of actual objects per slab
--- | --- | --- | ---
0 | 8 | 1 | 496
1 | 16 | 2 | 504
2 | 32 | 4 | 508
3 | 64 | 8 | 510
4 | 128 | 16 | 511
5 | 256 | 32 | 511
6 | 512 | 64 | 511
7 | 1024 | 128 | 511
The `sh_slab_generic_SLAB` structure describe the header of each slab, as well as each object slot. The `sh_slab_generic_SLAB_ALLOCATOR` structure contain the metadatas of the allocator.
The API for generic slab allocators is as follow:
- `SH_STATUS sh_slab_generic_alloc_init(sh_uint8 level,struct sh_slab_generic_SLAB_ALLOCATOR* slab_alloc,sh_pba_PAGE_BLOCK_ALLOCATOR *pba)`: initialize the slab allocator, doesn't allocate any slab
- `SH_STATUS sh_slab_generic_add_slab(struct sh_slab_generic_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_generic_SLAB** out_slab)`: add a new slab, allocate from the PBA
- `SH_STATUS sh_slab_generic_alloc(struct sh_slab_generic_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,void** out_obj)`: allocate a new object and return his pointer. Can allocate new slabs
- `SH_STATUS sh_slab_generic_dealloc(struct sh_slab_generic_SLAB_ALLOCATOR* alloc,void *object_ptr)`: deallocate a object using the provided pointer. Doesn't delete slabs that are made empty by objects deallocation
It's defined inside `shelter/lib/include/memory/slabs/slab_generic.h` and implemented inside `shelter/lib/src/memory/slabs/slab_generic.c`.
## Notes
The current implementation of all slabs allocator uses a bitmap stored inside the headers to find the first available object slot. It's planned the next update to test the freelist approch. If it's concluant, it will be used as the default approch in the next update.
It's also planned that all slabs allocators uses PBAs in the next update. This switch will not changes the indexes-based approch for region objects slab allocators.

9
docs/shelter/memory/vmemlayout.md Normal file
View File

@@ -0,0 +1,9 @@
# Virtual memory layout
## Introduction
The virtual memory layout is a file `shelter/lib/include/memory/vmem_layout.h` defining which area of virtual memory serve which purposes. It's automatically checked for no overlaps by a python script.
## Overview
Virtual memory regions are defined with macros. Any macro ending with `_VA` will create a new virtual region for the script. The script will then look for the size of this virtual region in another macro that start with the same prefix and end with `_SIZE_BYTES`. If a macro ending with `_VA` doesn't have a corresponding macro ending with `_SIZE_BYTES`, the script will trigger an error and the kernel compilation will fail. If a macro ending with `_SIZE_BYTES` doesn't have a corresponding macro ending with `_VA`, the script will ignore it. The start of each virtual region must be aligned to 4096 bytes and the size must be provided in bytes. Any overlapping virtual region will trigger a compilation error. Consider using this file as the source of trust for everything related to static virtual regions. Any macro that doesn't end with `_VA` or `_SIZE_BYTES` or that doesn't correspong to the behaviour described above will be ignored.

21
docs/shelter/naming.md Normal file
View File

@@ -0,0 +1,21 @@
# Naming scheme
Shelter has a very specific way of organizing subsystems.
First, each part of the kernel (except `main.c` which is responsible for the boot process) is in his own folder. To this date, there is 4 main parts:
- `cpu`: CPU abstractions
- `kernel`: kernel services
- `std`: Shelter standard library
- `memory`: the whole memory subsystem
Then, each of these parts have specific subsystem or API prefix. These prefixes always start by `sh_` and doesn't include the part each subsystem or API is in. For exemple, the serial outputing API prefix is `sh_serial_`.
The `std` part is the only part where, when not precised, the prefix will be shortened to only `sh_`, like `sh_malloc` or `SH_STATUS`.
In all the subsystems and API described in this documentation, the prefix will always be indicated.
These prefixes extand to names of functions, macros and type/structures. Here is the naming style for each of these entity:
- functions: all letters should be lowercase, starting with the prefix in lowercase. Example: `sh_serial_send_byte`
- macros: all letters should be uppercase, starting with the prefix in uppercase. Example: `SH_PAGE_RW`
- types/structure: the name should start with the prefix in lowercase and continue with the type/structure name in uppercase. Example: `sh_page_PAGE_TABLE_POOL`.
Please avoid abreviations/acronyms in the type name, instead define the abreviation/acronym inside the short comment preceding the declaration. Abreviations/acronyms anywhere else is tollerated as long as it's defined somewhere obvious. Don't hesitate to ask if you are not sure.

23
docs/shelter/ptp.md Normal file
View File

@@ -0,0 +1,23 @@
# Introduction to Pages Table Pool
## Introduction
In order to easely manage pages that are used for pages tables (PT), the Vystem project use pages tables pool (PTP). For details about mapping the PTP inside the kernel virtual memory space, please see [boot contract docs](bootcontract.md).
## Overview
In order to map any page into a pages table on x86-64, you need to descend all four layers of the PT. But what happen when you need a new page for an intermediate layer or directly a pages table ? To solve this problem, we have decided to introduce Pages Table Pool (PTP), a very simple way to allocate new pages for pages table.
Each pages table pool is made from a range of pages, contiguous both physically (this is what allow us to instantely know which value to put into a PTE for a intermediate layer) and virtually. The standard size for this range is currently 4096 pages, giving us plenty of rooms for various allocations accross the virtual memory. This size is evolutive and can changed with versions. The PML4 root page is by standard the first page of the PTP, making it suitable to use the starting VA of any PTP (as long as it's aligned on 4096 bytes) as value for the CR3 register.
There is two kinds of PTP:
- modifiable PTP: this is any PTP that is modifiable by the code that run into the virtual layout the PTP describe. To accomplish that, we map the entire pages range of the PTP into itself before putting it into CR3. We use this kind of PTP for the Shelter kernel
- unmodifiable PTP: here, we just don't do the recursive mapping of the pages range of the PTP into itself. This is the kind of PTP that will be used for any userland program
PTP, when created from the bootloader, are mapped within themself read-write and non-execution permissions. There is no primitive to create PTP from the kernel for the moment.
## Allocator
The purpose of any PTP is to have a reserved amount of pages to permit the quick allocation of single page to allow any mapping. There is two kinds of allocator, depending on the needs of the situation:
- bumb/counter allocator: here, we simply increment a counter that represent an index inside the pages range of the PTP, allowing for very quick PTP initialization. Intended for PTP creation and mapping without unmapping by the program creating the PTP. This is the kind of allocator used into the Blastproof bootloader
- bitmap allocator: a bitmap for storing which pages of the pages range is allocated or not. Intended when heavy unmapping is made, to free pages more easely. This is the allocator used in the Shelter Kernel. It can be initialized from the counter value, passed by the bootloader for example. For the moment, automatic pages freeing from the page table isn't implemented.

7
docs/shelter/std/malloc.md Normal file
View File

@@ -0,0 +1,7 @@
# Heap memory allocations
The Shelter standard library provide a single API for memory allocations on the heap. The heap is only initialized at the end of the memory subsystem and can't be used before. It's not adapted for memory mapped I/O or big buffers allocations (larger than one hundrer pages). The heap internal documentation can be found inside the memory subsystem documentation.
The memory allocations API provide the two following functions (defined inside `shelter/lib/include/std/malloc.h` and implemented inside `shelter/lib/src/std/malloc.c`):
- `void* sh_malloc(sh_uint64 size)`: allocate `size` amount of bytes. Return `SH_NULLPTR` if an error occured. The heap internal will trigger a heap crash (essentially a `while (SH_TRUE)` loop) to prevent any further damage if something very bad happen.
- `void sh_free(void *ptr)`: free the memory allocated at `ptr`. The heap internal will trigger a heap crash (essentially a `while (SH_TRUE)` loop) to prevent any further damage if something very bad happen.

6
docs/shelter/std/mem.md Normal file
View File

@@ -0,0 +1,6 @@
# Basic memory operations
The Shelter standard library provide very basic memory operations primitives, defined in `shelter/lib/include/std/mem.h` and implemented inside `shelter/lib/src/std/mem.c`:
- `SH_STATUS sh_mem_compare(const void *a,const void *b,sh_uint64 size)`: compare two memory regions with the same size. Return `SH_STATUS_SUCCESS` if both regions are equal, or `SH_STATUS_MEM_NOT_EQUAL` if one byte is different.
- `SH_STATUS sh_mem_copy(const void *destination,const void *source,sh_uint64 size)`: copy one region of memory to another. Return `SH_STATUS_SUCCESS`
- `SH_STATUS sh_mem_set_8(sh_uint8 *ptr,const sh_uint8 byte,sh_uint64 count)`: set a provided amount of bytes to the value of one provided byte. Return `SH_STATUS_SUCCESS`

20
docs/shelter/std/status.md Normal file
View File

@@ -0,0 +1,20 @@
# Return status
## Introduction
The vast majority of functions in Shelter return status code know as the `SH_STATUS` type. All the status codes are defined inside `shelter/lib/include/std/status.h`.
## Overview
Shelter define a status code with the `SH_STATUS` type. It's a wrapper of `sh_int64`.
The sign of a status play a big role:
- If the sign is negative, the status code isn't an error but rather an indication of something not that bad that happened during the function call. It can also serve the purpose of indicating a more detailled result.
- If the status code is 0, it's equivalent to a success, defined as `SH_STATUS_SUCCESS`
- If the sign is positive, the status code signify an error. You can use the `sh_status_error(SH_STATUS status)` function to know if a status code is an error. Return `SH_TRUE` if so.
Please check `shelter/lib/include/std/status.h` for the list of all status codes.
In Shelter, most function will returned to the caller the status code of all the functions calls they made if an error happened. Sometimes, it's necessary to dig through the code to understand from where an error come from.
If at any moment, a function that return a `SH_STATUS` call a function that return a pointer and the pointer is `SH_NULLPTR`, the status code `SH_STATUS_ERROR_NULLPTR_RETURNED` (not all error start with `SH_STATUS_ERROR`) will be returned.

14
docs/shelter/std/std.md Normal file
View File

@@ -0,0 +1,14 @@
# Shelter standard library
## Introduction
In order to function properly, any kernel need a standard library. Shelter define his own minimal standard library with his own particularities.
## Summary
1) [Basic types](types.md)
2) [Return status](status.md)
3) [Basic memory operations](mem.md)
4) [Heap memory allocations](malloc.md)
You can include the file `shelter/lib/include/std/stdlib.h` to include all necessary headers to access the Shelter standard library.

35
docs/shelter/std/types.md Normal file
View File

@@ -0,0 +1,35 @@
# Basic types
The Shelter standard library define the following type in `shelter/lib/include/std/type.h`:
- `sh_int8`: 1 byte signed integer
- `sh_uint8`: 1 byte unsigned integer
- `sh_int16`: 2 bytes signed integer
- `sh_uint16`: 2 bytes unsigned integer
- `sh_int32`: 4 bytes signed integer
- `sh_uint32`: 4 bytes unsigned integer
- `sh_int64`: 8 bytes signed integer
- `sh_uint64`: 8 bytes unsigned integer
- `sh_bool`: wrapper of `sh_uint8`
- `sh_iter64`: wrapper of `sh_uint64`, used for loops
- `va_list`: wrapper of `__builtin_va_list`
The Shelter standard library the following macros and values in `shelter/lib/include/std/type.h`:
- `SH_INT8_MIN`: the minimum possible value of `sh_int8`
- `SH_INT8_MAX`: the maximum possible value of `sh_int8`
- `SH_UINT8_MAX`: the maximum possible value of `sh_uint8`
- `SH_INT16_MIN`: the minimum possible value of `sh_int16`
- `SH_INT16_MAX`: the maximum possible value of `sh_int16`
- `SH_UINT16_MAX`: the maximum possible value of `sh_uint16`
- `SH_INT32_MIN`: the minimum possible value of `sh_int32`
- `SH_INT32_MAX`: the maximum possible value of `sh_int32`
- `SH_UINT32_MAX`: the maximum possible value of `sh_uint32`
- `SH_INT64_MIN`: the minimum possible value of `sh_int64`
- `SH_INT64_MAX`: the maximum possible value of `sh_int64`
- `SH_UINT64_MAX`: the maximum possible value of `sh_uint64`
- `SH_TRUE`: a wrapper of `(sh_bool)1`
- `SH_FALSE`: a wrapper of `(sh_bool)0`
- `SH_DEFVALUE`: the default value for any argument marked by `DEFAULT`
- `SH_NULLPTR`: wrapper of `(void*)0`
- `va_start(v,l)`: wrapper of `__builtin_va_start(v,l)`
- `va_end(v)`: wrapper of `__builtin_va_end(v)`
- `va_arg(v,l)`: wrapper of `__builtin_va_arg(v,l)`

53
docs/shelter/tab.md Normal file
View File

@@ -0,0 +1,53 @@
# Test-and-benchmark framework
## Introduction
The Shelter kernel includes his own test-and-benchmark (TAB) framework to support the measurements of performances and reliability of differents subsystems. This file describe the logic behind all the tests included inside the Shelter kernel and the embarqued payload, not the common API base described in [tests utilities](kernel/testutils.md) or the benchmark results.
A payload is a file generated at compilation, mostly random, used for accurate testing in specific situations.
## Slab allocators tests
These tests are designed to test the reliability and performances of the three main slab allocators behind the Pez subsystem: physical and virtual region objects slab allocators as well as the radix tree nodes slab allocators. They are defined inside `shelter/lib/include/kernel/tests/test_slabs.h` and implemented inside `shelter/lib/src/kernel/tests/test_slabs.c`. This test doesn't require a payload.
They consist in allocating 10000 objects from these three slabs and deallocating them afterward. This allocate a few slabs for each allocator, meaning they will consume a few pages at the start but nothing too expensive for the systems. This also allow for quick allocations of the first few thousands objects of these allocators since the slabs are already allocated.
## Radix tree subsystem tests
These tests are designed to test the reliability and performances of the radix tree subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_radix.h` and implemented inside `shelter/lib/src/kernel/tests/test_radix.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_radix_payload.h`.
The payload contain the following datas:
- `test_keys`: an array of 10000 random `sh_uint64` serving as keys for the pairs
- `test_values`: an array of 10000 random `sh_uint64` serving as values for the pairs
- `test_search`: an array of 10000 random `sh_uint64` serving as keys for the lower bound keys for the backtracking algorithm
All tests are run on a 16-deep unique radix tree. The test are in the following order:
- Insertion: trying to insert 10000 pairs of keys-values defined inside `test_keys` and `test_values`
- Reading: trying to obtain the associated value of each of the 10000 pairs by providing the values of `test_keys`, checking if the correct value is returned by comparing to the values of `test_values`
- Searching: trying to search using the backtracking algorithm with lower bound key using the values of `test_search`
- Deleting: trying to delete all pairs using the keys contained inside `test_keys`
## Pez subsystem tests
This test is designed to test the reliability and performances of the Pez subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_pez.h` and implemented inside `shelter/lib/src/kernel/tests/test_pez.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_pez_payload.h`.
This specific test has been designed to see how Pez react under high pressure. It's only apply it on Pez's physical plane, as the virtual plane uses the same logic.
The payload contain the following datas:
- `test_pez_physical_size`: an array of 2000 `sh_uint64` serving as sizes in pages for the allocation. Half of them are 1. The other half is between 2 and 1000.
- `test_pez_physical_alloc`: an array of 4000 `sh_uint64`, serving as reference for when to allocate and when to deallocate each allocated region. These numbers range from 1 to 2000 and appear exactly 2 times for each.
The test work like this: we iterate on all the value of `test_pez_physical_alloc`. When we hit for the first time a number, we allocate the corresponding size inside `test_pez_physical_size` by using the allocation number as index. When we hit for the second time a number, we deallocate this allocation.
The test will separate single page allocations and frees from multi-pages allocations and frees when showing the results.
## Malloc subsystem test
These tests are design to test the reliability and performances of the Pez subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_malloc.h` and implemented inside `shelter/lib/src/kernel/tests/test_malloc.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_malloc_payload.h`.
The payload contain the following datas:
- `test_malloc_small_size`: contain 10000 sizes for small allocations (under or equal to 1024 bytes)
- `test_malloc_big_size`: contain 1000 sizes in pages, ranging from 1 to 25, serving the same role as `test_pez_physical_size`
- `test_malloc_big_alloc`: contain 2000 values, serving the same role and generated like `test_pez_physical_alloc`
The malloc test combine two tests: 10000 allocations and deallocations of variously sized small objects using the generic slab allocators, and 1000 allocations and frees of pages on the heap, on the same logic of the Pez subsystem test, but adapted to the quantity used in this test.

5
docs/vyld/index.md Normal file
View File

@@ -0,0 +1,5 @@
# Vyld Docs
Vyld, standing for Vystem Linker, is a specialised linker designed to convert ELF binaries to the VYX executable format. This documentation is divided into two parts:
- [The VYX executable format](vyx.md)
- [The Vyld utility](vyld.md)

67
docs/vyld/vyld.md Normal file
View File

@@ -0,0 +1,67 @@
# Vyld Docs
## Introduction
Vyld, standing for Vystem Linker, is the utility responsible for creating VYX executable from ELF binaries. Vyld has been intended to be used only on a regular Linux system. For a better understanding, it's recommanded you read the [VYX format docs](vyx.md) first.
## Detailled informations
Folder: `vyld`
Source code file: `vyld.cpp`
For building the `vyld` utility, use the following command:
``` bash
g++ vyld.cpp -o vyld
```
External library:
- ELFIO, made by Serge Lamikhov-Center, sourced from [serge1/ELFIO](https://github.com/serge1/ELFIO), under the MIT license
## Usage
Requirement: having the `gcc` compiler and `ld` linker reachable in path. For the moment, only the `gcc` compiler is supported.
The `vyld` utility can be used like this:
``` bash
vyld <path to C file 1> <path to C file 2> ... <path to C file n> <path to output file>
```
Each path provided except the last one has to be C file, or else the compilation might fail. The last path can be any extension, but for standard respect, it should be a VYX file.
Vyld can accept custom compilation flags for provided C files if they are passed by using the `VYLD_COMPILATION_FLAGS` environnement variable. They are added to the mandatory compilation flags for provided C files but they will not replace them. The mandatory compilation flags for provided C files are:
``` bash
-fno-pic -m64 -mcmodel=large
```
The mandatory compilation flag for the `_vyx_start.c` entry point are:
``` bash
-ffreestanding -nostdlib -nostartfiles -Wall -Wextra -fno-stack-protector -fno-builtin -fno-pic -m64 -mcmodel=large
```
These flags aren't applied to provided C files unless passed inside `VYLD_COMPILATION_FLAGS`.
Vyld need to have access to the `_vyx_start.c` file in his executable directory in order to compile any VYX binary. This file is the entry point of any VYX executable.
Command example:
``` bash
vyld hello.c io.c binary.vyx
VYLD_COMPILATION_FLAGS="-Wall -Wextra -Werror -Wpedantic" vyld hello.c io.c binary.vyx
```
We plane to expand drastically the capabilities and versatility of the `vyld` utility and the VYX format in the future, but for the moment, it's sufficient for what we need (loading the Shelter kernel).
## Detailled generation processus
1) Obtaining his own executable path by reading `/proc/self/exe` in order to obtain a deterministic working directory
2) Searching for flags provided into the `VYLD_COMPILATION_FLAGS`
3) Checking for existence of `_vyx_start.c ` and compiling it into `_vyx_start.o` using mandatory compilation flags for entry point. The resulting object file is stored inside the `vyld` executable directory
4) Iterating for each provided C files: finding his location and generating his output name under the format `<file name without .c extension>.o`
5) Iterating for each provided C files: compiling each C file into an object file using mandatory compilation flags for provided C files and flags passed through `VYLD_COMPILATION_FLAGS`. The resulting objects files are stored inside the `vyld` executable directory
6) Combining all objects files (entry point and all provided files under their object form) into one object file named `combined.o` and stored inside the `vyld` executable directory. For that, we use the `gcc -r` feature of `gcc`. The entry point object file is specifically added in first position
7) Opening `combined.o` and obtaining sections size for `.text`, `.data`, `.rodata` and assimiled sections as well as `.bss` section. Padding these sizes to the nearest multiple of 4096 bytes superior to these sizes
8) Generating the `.text` section base virtual address (VA) and stack base VA, aligned on 4096 bytes pages
9) Generating a custom linker script to correctly link the `combined.o` file with the right VAs inside the code and putting in order all the sections into the final binary
10) Linking the `combined.o` file into an ELF executable named `compiled.elf` and stored inside the `vyld` executable folder
11) Opening the `compiled.elf` binary, extracting all sections, generating VYX header and executable
The range for generating `.text` section base is currently `0xFFFF800000000000` to `0xFFFF900000000000-1`, then padded to the nearest page boundary.
The range for generating stack base is currently `0xFFFFF00000000000` to `0xFFFFFF8000000000-0x100000`, then padded to the nearest page boundary.

32
docs/vyld/vyx.md Normal file
View File

@@ -0,0 +1,32 @@
# The VYX executable format
## Introduction
The VYX format, standing for Vystem Executable, is a very simple binary format made for simple binary loading from any environnement. As long as you can access the `.vyx` file, you can load a VYX binary.
## Header
Because VYX format has been designed to provide a simple way to load any kind of binary, we wanted to make it as easely updatable as possible. For the moment, it's just a very simple format built for loading the Shelter kernel, but in a few updates, it will be use for loading all kind of binaries. No spoiler on that. The header is designed like that:
- Signature: three bytes spelling `VyX` in ASCII
- VYX format version: the version of the VYX format used in the binary, represented as an uint16_t. Depending on the version number, the following structure will probably changed a lot. The current version is `0x0001`.
- `.text` base address: the virtual address (VA) at which the `.text` should be loaded. It should be aligned on the boundary of a 4 kilobytes page. Represented as an uint64_t
- stack base: the VA at which the stack base should be setuped. Be careful, it indicate the base, not the top. It should be aligned on the boundary of a 4 kilobytes page. Represented as an uint64_t
- the size of the `.text` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the size of the `.data` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the size of the `.rodata` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the size of the `.bss` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
All the multibytes integers should be written in little endian. The VYX executable format has been made for x86-64 only binary.
## Sections organizations
The VYX format only support three sections inside the binary that should be in the following order: `.text`, `.data`, `.rodata`. The `.bss` should be initialized and zeroed by the loader just after the `.rodata` section. The space occupied by the three sections inside the binary should occupy a multiple of 4096 bytes, according to the size indicated in the header, padded with zeroes if necessary.
For example, if each sections take 2 pages, after being padded to the nearest multiple of 4096 bytes superior to their real size, and that the `.text` base is 0x1000, `.text` should be located at 0x1000, `.data` at 0x3000, `.rodata` at 0x5000 and `.bss` at 0x7000
## Specifications
The VYX binaries use the SystemV ABI. They doesn't support rellocations nor dynamic linking. While relative addressing into the binary code is theorically working, all the Vystem binaries will be compiled 64 bits static addressing with position-dependent code (non-PIC), especially the kernel. That why respecting the indicated `.text` section base and order of sections loading is extremely important.
Obviously, the right privileges should be applied in term of pagging: no execution unless `.text` pages, read only `.text` and `.rodata` sections, read-write `.data` and `.bss` sections.
For the moment, the entry point is located at the base of the `.text` section.
The stack size can be decided at the convenience of the loader, but the stack size used for the Shelter kernel is, for the moment, 256 kilobytes. The stack growing downward, the initial stack pointer should be set to stack base + stack size.
A reference implementation of a VYX loader, exploiting all currently documented features, can be found into the Blastproof bootloader source code.