PastaBoy12345/vystem-b
2
0
Fork 0
forked from lolo859/vystem
Code Pull Requests Activity

Vystem 0.2

Browse Source
This commit is contained in:
lolo859
2026-05-27 19:34:54 +02:00
parent a43c08b893
commit d238606b75
372 changed files with 51320 additions and 83217 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/blastproof/bootconfig.md
View File

@@ -2,7 +2,7 @@
## Introduction
Blastproof store his configuration inside a text file named `bp.conf` and stored inside `\EFI\BlastProofFiles` inside the EFI partition. Due to being protected by SBFIE, the config is immutable after compilation and any change to it will cause the boot to fail, even if provided password was good. A example configuration is provided as `Blastproof/config/bp_template.conf`
Blastproof store his configuration inside a text file named `bp.conf` and stored inside `\EFI\BPFILES` inside the EFI partition. Due to being protected by SBFIE, the config is immutable after compilation and any change to it will cause the boot to fail, even if provided password was good. A example configuration is provided as `Blastproof/config/bp_template.conf`
## Syntax and behaviour
@@ -38,7 +38,7 @@ Any non-ascii character will cause the boot to fail. All the keys that are liste
**font:**
- Description: Give the name of the file containing the font the bootloader will atempt to use.
- Possible value: any value representing a file name stored inside `EFI\BlastProofFiles`
- Possible value: any value representing a file name stored inside `EFI\BPFILES`
- Default value: `bitra-ascii-medium.fbm`
**disable_boot_animation:**
@@ -109,3 +109,8 @@ Any non-ascii character will cause the boot to fail. All the keys that are liste
- Description: define the amount of pages used for logging ring buffer. Setting this pair to 0 will disable the logging ring.
- Possible value: any integer ranging from 0 to 65535
- Default value: `2048`
**kernel_kbd_events_queue_capacity**
- Description: define the amount of keyboards events each keyboard events can hold
- Possible value: any integer ranging from 64 to 65535
- Default value: `256`

9
docs/blastproof/bootprocess.md
View File

@@ -2,10 +2,7 @@
Blastproof boot process follow a very rigid sequences of steps that are detailled below:
- Printing informations:
- Bootloader version
- Firmware informations
- CPU informations
- Verifying VFTM
- Reading boot password from EFI console
- Verifying integrity of integrated keys (see SBFIE)
- Acquiring access to ESP partition
@@ -20,14 +17,14 @@ Blastproof boot process follow a very rigid sequences of steps that are detaille
- Loading and rendering bitmap font according to selected graphic output format
- Loading, rendering and playing boot animation if configuration allow it
- Loading InitFS and SignSyst
- Listing all availables BlockIO protocol
- Locating handles for InitFS and SignSyst
- Parsing the GPT table on which the ESP is located
- Allocating memory for them
- Loading them into RAM
- Parsing their headers
- Performing cryptographic checks to validate their headers and contents
- Loading the kernel
- Loading Shelter and Keycard binary
- Obtaining ACPI pointer
- Generating Shelter boot configuration
- Allocating pages for kernel sections
- Creating page table

21
docs/blastproof/bplib.md
View File

@@ -29,16 +29,17 @@ The Blastproof bootloader is based on the EDK II framework:
Blastproof library is divided into several components, some relying on the external libraries cited above, and all relying on the EDK II framework. Here is the list of all components with their headers:
- `conf.h`: manage configuration loading and parsing
- `console.h`: provide a simple way to enter a password inside the EFI shell
- `cpu.h`: provide a simple way to print CPU ID informations
- `crypto.h`: provide the cryptographic abstraction layer to use algorithms such as Argon2, SHA3 or Sphincs+
- `debug.h`: provide outputting primitives to the serial port
- `default.h`: provide macros for default configuration values
- `disk.h`: provide functions to read files from ESP partition
- `font.h`: provide functions to load, parse and render FBM fonts
- `graphic.h`: provide basic primitives to interact with GOP framebuffers
- `console.h`: provides a simple way to enter a password inside the EFI shell
- `cpu.h`: provides a simple way to print CPU ID informations
- `crypto.h`: provides the cryptographic abstraction layer to use algorithms such as Argon2, SHA3 or Sphincs+
- `debug.h`: provides outputting primitives to the serial port
- `default.h`: provides macros for default configuration values
- `disk.h`: provides functions to read files from ESP partition
- `font.h`: provides functions to load, parse and render FBM fonts
- `graphic.h`: provides basic primitives to interact with GOP framebuffers
- `initfs.h`: InitFS and SignSyst drivers
- `ui.h`: provide primitives to render text and boot animation in the GOP framebuffer
- `vyx.h`: provide a very basic Vyx-based kernel loader
- `ui.h`: provides primitives to render text and boot animation in the GOP framebuffer
- `vyx.h`: provides a very basic Vyx-based kernel loader
- `vftm.h`: verify VFTM authenticity and informations
Additional headers can be generated at compilation time and aren't included in this list, such as `key.h`.

2
docs/blastproof/fbm.md
View File

@@ -50,4 +50,4 @@ When rendering the font, the renderer will take two values: background color and
## Provided font
Vystem provide a basic font named `bitra`, with 12 pixels as width and 20 as height, only supporting ASCII characters. This font is stored in the file `bitra-medium-ascii.fbm`.
Vystem provides a basic font named `bitra`, with 12 pixels as width and 20 as height, only supporting ASCII characters. This font is stored in the file `FONT.FBM`.

8
docs/blastproof/fs.md
View File

@@ -15,7 +15,7 @@ The header of any InitFS filesystem is as follow:
- Bootloader version: an uint16_t indicating the bootloader version, currently set to `0x0001`
- InitFS version: an uint16_t indicating the InitFS version, currently set to `0x0001`
- OS version: an uint32_t indicating the OS version, currently set to `0x00000001`
- Installation ID: an array of 48 completely random bytes. His SHA3-512 hash is stored inside the `initfs-footprint.bin` file, protected by SPFIE
- Installation ID: an array of 48 completely random bytes. His SHA3-512 hash is stored inside the `INITFSFP.BIN` file, protected by SBFIE
- InitFS Size: the amount of bytes in the total filesystem, not forcefully rounded to the disk block size. Stored as an uint64_t
- Files table size: the amount of bytes used for the filesystem files table. Stored as an uint64_t
- Files content size: the amount of bytes for the filesystem files content area. Stored as an uint64_t
@@ -26,11 +26,11 @@ The header of any InitFS filesystem is as follow:
- Entropy check: 8 bytes of a value generated with this formula : `header.entropy_check=(header.entropy*0x9E3779B185EBCA87)^header.entropy`. Stored as an uint64_t
- Files table hash: an array of 64 bytes containing the SHA3-512 hash of the files table
- Files content area hash: an array of 64 bytes containig the SHA3-512 hash of the files content area
- Installation ID double hash: an array of 64 bytes containing the SHA3-512 hash of the content of the `initfs-footprint.bin` file.
- Installation ID double hash: an array of 64 bytes containing the SHA3-512 hash of the content of the `INITFSFP.BIN` file.
- Padding: 128 bytes of padding generated with a specific pattern (see below)
- Header hash: an array of 64 bytes containing the SHA3-512 of the InitFS header, not including the last 64 bytes
This header is exactly 512 bytes. The mecanism decribed above with the installation ID and the `initfs-footprint.bin` is to ensure that an installation of Blastproof find his associated InitFS.
This header is exactly 512 bytes. The mecanism decribed above with the installation ID and the `INITFSFP.BIN` is to ensure that an installation of Blastproof find his associated InitFS.
### Files table and files entries
@@ -67,7 +67,7 @@ The header of any SignSyst is as follow:
- Padding: 288 bytes of padding generated with the same pattern as InitFS
- Header hash: an array of 64 bytes containing the SHA3-512 of the SignSyst header, not including the last 64 bytes
The header is exactly 512 bytes. The file `signsyst-hash.bin`, protected by SPFIE, contain the full hash of the SignSyst header.
The header is exactly 512 bytes. The file `SSHASH.BIN`, protected by SBFIE, contain the full hash of the SignSyst header.
### Signatues area

14
docs/blastproof/index.md
View File

@@ -2,20 +2,22 @@
## Introduction
Blastproof is Vystem's bootloader. It is a UEFI x86-64 only bootloader that rely on EDK II for all hardware interactions. It is able to confirm the integrity of his files, locate the kernel and load it into memory, using the specific boot protocol designed for Shelter.
Blastproof is Vystem's bootloader. It is a UEFI x86-64 only bootloader that rely on EDK II for all hardware interactions. It is able to confirm the integrity of his files and partition, locate the kernel and load it into memory, using the specific boot protocol designed for Shelter.
## Blastproof architectural overview
The Blastproof bootloader rely mainly on the EDK II framework for printing to the EFI console, interacting with disks/partitions and FAT32 filesystem, graphic output protocol, and memory allocation. However, it uses Vystem standard formats for fonts, filesystems, security and integrity checks, graphic renderer, and kernel loading.
The Blastproof bootloader rely mainly on the EDK II framework for printing to the EFI console, interacting with disks and FAT32 filesystem, graphic output protocol, and memory allocation. However, it uses Vystem standard formats for fonts, filesystems, security and integrity checks, graphic renderer, and kernel loading.
## Ressources
1) [Bootprocess](bootprocess.md)
2) [Keyboard layouts](kbdlayout.md)
2) [Boot configuration](bootconfig.md)
3) [Security and integrity model at boot level](security.md)
4) [Secure Boot Files Integrity Enforcement (SBFIE)](sbfie.md)
5) [Custom filesystems](fs.md)
6) [Font Bitmap](fbm.md)
7) [Blastproof library](bplib.md)
4) [Vystem FAT Trusted Manifest](vftm.md)
5) [Secure Boot Files Integrity Enforcement (SBFIE)](sbfie.md)
6) [Custom filesystems](fs.md)
7) [Font Bitmap](fbm.md)
8) [Blastproof library](bplib.md)
For detailled informations about kernel loading, please see Vyld docs.

71
docs/blastproof/kbdlayout.md Normal file
View File

@@ -0,0 +1,71 @@
# Keyboard layouts
## Introduction
Blastproof supports the addition of custom keyboards layout. This file describe how they are structured and how to create one.
## Implementation
The `layout.h` header, located in `Blastproof/src/libs/include`, contain the utilities needed to create custom keyboards layout. Two main structures are needed:
``` C
typedef struct {
CHAR16 char_source;
CHAR16 char_dest;
} bp_layout_CHAR_CONV;
```
The `char_source` field must be the character that Blastproof will receive from the UEFI firmware. In the immense majority of case, the firmware interprets each pressed key using the QWERTY layout. The `char_dest` field must be the character that replaced the inputed character.
``` C
typedef struct {
UINTN conv_count;
bp_layout_CHAR_CONV conv_table[];
} bp_layout_CONV_TABLE;
```
This structure is defined at compilation time and should indicate the amount of characters conversions as well as all the characters conversions, stored in an array.
## Usage
In the immense majority of cases, the firmware uses the QWERTY layout. If this is the case, you should use the keyboard layout of your keyboard. Here are the supported keyboard layout for the moment:
- `QWERTY`
- `AZERTY`
In the case where the firmware already use the keyboard layout of your keyboard, please select the `QWERTY` layout : it doesn't contain any key translation.
## Creating your own layout
In order to create your own keyboard layout, you can follow these steps:
### Creating the header file
1) Create a header file named `<keyboard layout name>.h` inside the `Blastproof/src/libs/include/layouts` folder.
2) Paste it the following template:
``` C
// SPDX-License-Identifier: MPL-2.0
#ifndef BP_LIB_LAYOUT_LAYOUTNAME_H
#define BP_LIB_LAYOUT_LAYOUTNAME_H
#include "../layout.h"
bp_layout_CONV_TABLE layoutname_convs={
.conv_count=0,
.conv_table={
}
};
#endif
```
3) Replace every `layoutname` and `LAYOUTNAME` with the name of your layout.
4) Add as many character conversions as you want. A character conversion is set under the form `{L'a',L'q'}`. In this example, we translate the character `a` to `q`.
5) Don't forget to update the `conv_count` field with the amount of characters conversions.
### Registering the layout
Go inside the `layout_registry.h` file, located inside `Blastproof/src/libs/include/layouts` and add the following line where indicated:
- `#include "<keyboard layout name>.h"` : replace `<keyboard layout name>` with the name of your layout
- add the string that will be displayed (generally the layout name) to the user inside
`layout_list`. Don't forger the `L` string modifier before the first quote
- `&<keyboard layout name>_conv` : add this element inside the `layout_conversions_table` variable. Don't forget the `&` operator

19
docs/blastproof/sbfie.md
View File

@@ -12,7 +12,7 @@ SBFIE has been conceived with the following limitations in mind:
- due to the will of it being compatible with a large variety of devices, it doesn't integrate TPM utilization (but support in an eventual future is planned)
- any user of a non-QWERTY keyboard will have a hard time typing a password in an UEFI console
- it use the EFI console as input (support of self-made input methods and protocol will be implemented in a near future)
- any missing file that is supposed to be protected by SPFIE will cause the boot process to fail
- any missing file that is supposed to be protected by SBFIE will cause the boot process to fail
## How it work
@@ -27,15 +27,16 @@ The `key.h` header contain the following datas:
- `bp_key_pkblob`: an array of `64*N` bytes containing `N` Sphincs+ public keys, one for each file
- `bp_key_pwdsalt`: an array of 32 bytes containing the salt to use for hashing the boot password
- `bp_key_files`: an array of `N` strings indicating the order in which boot files should be verified for each public key from `sh_key_pkblob` to correspond with each file
- `bp_key_files_sign`: an array of `N` strings indicating the corresponding signature file for each file of `bp_key_files`
We assume that all files are in the `\EFI\BlastProofFiles` folder with their corresponding names as specified in `bp_key_files` and that their Sphincs+ signatures are stored as `<file name>.sig` inside of the `\EFI\BlastProofSign` folder.
We assume that all files are in the `\EFI\BPFILES` folder with their corresponding names as specified in `bp_key_files` and that their Sphincs+ signatures are stored inside `bp_key_files_sign` as `<file name without extension>.sig` inside of the `\EFI\BPSIGN` folder.
### Detailled steps
The steps in which SPFIE check the files is as follow:
The steps in which SBFIE check the files is as follow:
1. The user enter the boot password
2. The boot password is hashed using Argon2 using 262144 of memory cost and 3 of time cost. The salt is stored inside `bp_key_pwdsalt`The output is a 96 digest.
2. The boot password is hashed using Argon2 using 262144 of memory cost and 3 of time cost. The salt is stored inside `bp_key_pwdsalt`. The output is a 96 digest.
3. The digest is used as seed for generating a Sphincs+ keypair, immediately erasing the private key and keeping only the public key.
4. The generated public key is used to verify the signature inside `bp_key_mainsig` with the message being `bp_key_pkblob`
5. If the verification is successfull, the boot process continue. If not, the boot process is halted.
@@ -46,8 +47,8 @@ The steps in which SPFIE check the files is as follow:
## Concerned files
To this date, the following files generated by build script are protected by SPFIE:
- `bp.conf`
- `bitra-ascii-medium.fbm`
- `initfs-footprint.bin`
- `signsyst-hash.bin`
To this date, the following files generated by build script are protected by SBFIE:
- `BP.CFG`
- `FONT.FBM`
- `INITFSFP.BIN`
- `SSHASH.BIN`

5
docs/blastproof/security.md
View File

@@ -7,9 +7,10 @@ The Vystem project has been designed for the ground up for security and integrit
## Overview
In order to achieve a strong amount of security without sacrificing compatibility with existing devices, we have chosen to integrate the following elements into the boot sequence:
- Secure Boot Files Integrity Enforcement (SBFIE): protect cores bootloader files assuming bootloader integrity with a user-defined password, without storing it directly but integrating it very deeply in the integrity check chain. See [SPFIE docs](spfie.md) for more informations
- Vystem FAT Trusted Manifest (VFTM): ensure that the FAT32 filesystem on the ESP partition was produced by an authorized entity, that the ESP partition hasn't been changed from disk, that the FAT32 filesystem hasn't been corrupted or modified, and allow for the bootloader to check his own binary without EFI Secure Boot. It's signed by Keygen and verified by the bootloader himself. See [VFTM docs](vftm.md) for more informations.
- Secure Boot Files Integrity Enforcement (SBFIE): protect cores bootloader files assuming bootloader integrity with a user-defined password, without storing it directly but integrating it very deeply in the integrity check chain. See [SBFIE docs](sbfie.md) for more informations
- Robust post-quantum cryptography integrated directly into the bootloader: while we rely on EDK II for files and disk IO as well as password input, all the cryptography stack is integrated into the bootloader, not relying on firmware cryptography at any time. We use post-quantum signing algorithm like SPHINCS+ and strong hashing primitives like Argon2 and SHA3, with their sources codes directly taken from their official implementation, only modified to integrate with EDK II types and memory allocation system
- Signing of every system files: every sensitive system file is integrated into the InitFS, signed into a trio of check: each Blastproof binary (and so SPFIE keys), InitFS and SignSyst is uniquely associated (mainly by installation ID), meaning that the whole system can only work if all of them are properly setup on the same machine device. InitFS integrity check are also cryptographically linked to SPFIE verification.
- Signing of every system files: every sensitive system file is integrated into the InitFS, signed into a trio of check: each Blastproof binary (and so SBFIE keys), InitFS and SignSyst is uniquely associated (mainly by installation ID), meaning that the whole system can only work if all of them are properly setup on the same machine device. InitFS integrity check are also cryptographically linked to SBFIE verification.
## Limitations

92
docs/blastproof/vftm.md Normal file
View File

@@ -0,0 +1,92 @@
# Vystem FAT Trusted Manifest
## Introduction
Vystem FAT Trusted Manifest (VFTM) is a structure that can be found in the reserved sectors of any FAT32 filesystem generated by VyBuild (please see the conditions for VyBuild to generate this manifest in the VyBuild documentation). It's the first layer of defense in the Vystem Secure Boot Chain, being verified independantly at each boot.
## Position and structure
VTFM can be divided into three parts:
- the VFTM manifest itself, containing all importants informations and hashs. It's stored somewhere between the sector 8 and 31 of the FAT32 filesystem
- the manifest public key. It's stored somewhere between the sector 8 and 31 of the FAT32 filesystem, but never on the same sector of the manifest itself
- the manifest signature. It's stored after the FAT32 filesystem, just before the end of the ESP partition
The manifest and manifest public key have their position determined by this function:
``` C
void generate_vftm_sectors(const uint8_t* part_guid,int* sector_manifest,int* sector_public_key) {
uint32_t h=0x811c9dc5;
for (int i=0;i<16;i++) h=(h^part_guid[i])*0x01000193;
*sector_manifest=8+(h%24);
*sector_public_key=8+(((h>>16)%23+(*sector_manifest-7))%24);
}
```
The `part_guid` argument is an array of 16 bytes containing the ESP partition unique GUID. That way, the position of these components of VFTM is linked to the GPT table.
## Generation of the keys
Two Sphincs+ keypairs are generated by the Keygen utility. They are named `root` and `manifest`. We use `sk_root` to sign `pk_manifest`, generating the signature `sig_root`. `sig_root` and `pk_root` are injected into the `key.h` header generated by Keygen, alongside the SBFIE stuff. `sk_root` is immediately destroyed and never leave Keygen memory space. Then, using UNIX socket named `vftm.sock`, that should be created in the current directory by the calling entity of Keygen, Keygen transmit the keypair `manifest` to the calling entity. It should always be the entity that will generate the FAT32 filesystem (in this case, VyBuild).
VyBuild will sign the generated manifest with `sk_manifest`, and erase it immediately after. `pk_manifest` became the public key that will be injected into one of the reserved sector of the filesystem, alongside the manifest. The generated `sig_manifest` is injected at the end of the ESP partition, right after the FAT32 manifest.
## Manifest structure
The manifest structure can be represented like this:
``` C
#pragma pack(1)
struct vystem_fat_trusted_manifest {
uint8_t sig[8];
uint16_t manifest_version;
uint16_t bootloader_version;
uint32_t os_version;
uint8_t fat_hash[64];
uint8_t fat_build_id[32];
uint64_t clusters_count;
uint64_t bytes_per_sector;
uint64_t sectors_per_cluster;
uint64_t fat_size_bytes;
uint64_t fat_offset_bytes;
uint64_t bootloader_cluster_count;
uint64_t bootloader_cluster_offset;
uint64_t bootloader_build_id;
uint8_t fat_header_hash[64];
uint8_t bootloader_binary_hash[64];
uint64_t signature_offset_bytes;
uint64_t signature_size;
uint8_t gpt_disk_guid[16];
uint8_t gpt_efi_part_unique_guid[16];
uint8_t manifest_hash[64];
};
#pragma pack()
```
The expected signature is `VyFatSig` in ASCII.
## VFTM detailled verification procedure
Before doing anything, the bootloader will do all these steps to verify the authenticity, then the content of the manifest:
### VFTM assets gathering
The bootloader will gather the following assets in order, using various EDK2 protocols:
- the disk GUID from the GPT header
- the ESP partition unique GUID, that will serve to determine where to extract the manifest and public key
- the manifest and public key
- the manifest signature
### Keys verification
The booloader will then do the following steps to confirm the manifest authenticity:
1) Using `pk_root` and `sig_root`, it verify the extracted `pk_manifest`. If this fail, the bootprocess is halted.
2) Using `pk_manifest` and `sig_manifest`, it verify the extracted manifest. If this fail, the bootprocess is halted.
### Manifest contents verification
Finally, it can use the manifest content to verify various things in this order:
1) Verify the indicated manifest signature, various versions and GUID provided into the manifest.
2) Verify the FAT32 header by comparing his hash and fields with what is provided into the manifest.
3) Verify the FATs by hashing the FATs found in the filesystem and comparing them with the hash provided into the manifest.
4) Verify the bootloader binary by hashing it (retrieving his content through metadatas contained inside the manifest, without using the SimpleFilesystemProtocol) and comparing it to the hash provided into the manifest
5) Verify the hash of the manifest
All hashs use the SHA3-512 algorithm.

18
docs/boottools/commoncrypto.md Normal file
View File

@@ -0,0 +1,18 @@
# Common crypto docs
## Introduction
The `commoncrypto` library is responsible for providing all cryptographic functions necessary to `keygen`, `initfsgen` and later VyBuild
## Detailled informations
Folder: `Blastproof/commoncrypto`
It's build using VyBuild, as both a static and shared library. The provided build file doesn't allow yet for the library to be build outside of Vystem build process.
The `-Ofast -march=native` is used on every source file, to make it faster in every use case.
The library include the following external cryptographic library:
- SPHINCS+, made by the SPHINCS+ team, sourced from [sphincs/sphincsplus](https://github.com/sphincs/sphincsplus), under the MIT-0 license
- tiny_sha3, made by mjosaarinen, sourced from [mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3), under the MIT license
- Argon2, made by the Argon2 team, sourced from [P-H-C/phc-winner-argon2](https://github.com/P-H-C/phc-winner-argon2), under the CC0-1.0 license

11
docs/boottools/index.md
View File

@@ -1,10 +1,11 @@
# Bootloader tools Docs
In order to generate all necessary components for the build of a Vystem disk image, we have created several C++ tools designed to help generating these components. All of these tools have been intented to be used only in a normal Linux environnement. Here is the list of bootloader tools:
1) [bootanim](bootanim.md)
2) [fontgen](fontgen.md)
3) [initfsgen](initfsgen.md)
4) [keygen](keygen.md)
In order to generate all necessary components for the build of a Vystem disk image, we have created several C++ tools and library designed to help generating these components. All of these tools have been intented to be used only in a normal Linux environnement. Here is the list of bootloader tools:
1) [commoncrypto](commoncrypto.md)
2) [bootanim](bootanim.md)
3) [fontgen](fontgen.md)
4) [initfsgen](initfsgen.md)
5) [keygen](keygen.md)
Here a few precisions about the global mecanisms inside each tools:
- when a tool is iterating on the content of a folder, it rely on the order provided by the filesystem. But tool like `keygen` get there files list through arguments specifying direct files path rather than a folder path, making it a deterministic order if the build script doesn't change

12
docs/boottools/initfsgen.md
View File

@@ -9,9 +9,9 @@ The `initfsgen` utility is responsible for generating the InitFS and SignSyst fi
Folder: `Blastproof/initfsgen`
Source code file: `initfsgen.cpp`
For building `initfsgen`, use the provided build script `build.sh` while being in the utility directory.
For building `initfsgen`, you will need to use the provided build file. But for the moment, the provided build file doesn't allow yet for the utility to be build outside of Vystem build process. The build file will require `libcommoncrypto.a` to be provided inside the `initfsgen` folder.
External library:
External library provided by `commoncrypto` and used inside `initfsgen`:
- SPHINCS+, made by the SPHINCS+ team, sourced from [sphincs/sphincsplus](https://github.com/sphincs/sphincsplus), under the MIT-0 license
- tiny_sha3, made by mjosaarinen, sourced from [mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3), under the MIT license
@@ -35,16 +35,16 @@ initfsgen ./initfs_dir
2) Collect 48 bytes of entropy for installation ID and 8 bytes of entropy for entropy check, by opening `/dev/urandom`. No others method of entropy generation are or will be supported.
3) Copying installation ID inside InitFS and SignSyst headers, filling all possible informations about files and signatures quantity and sizes inside the headers.
4) Generating entropy check, putting it into InitFS header.
5) Computing installation ID hash and saving it into `initfs-footprint.bin`, saved into the current directory.
6) Computing hash of the content of `initfs-footprint.bin` and saving it into the InitFS header.
5) Computing installation ID hash and saving it into `initfsfp.bin`, saved into the current directory.
6) Computing hash of the content of `initfsfp.bin` and saving it into the InitFS header.
7) Iterate for each file: loading the entire file, generating InitFS file name, copying file data into files content area, computing file content hash, generating SPHINCS+ keypair unique to this file, generating signature for file by using private key, securely erasing private key (without compiler optimization), saving file hash and public key into file entry, saving file entry into files table and adding generated signature to SignSyst
8) Generating files table, files content area and signatures area hash, saving them into headers by xoring them with part of the installation ID
9) Generating and saving padding for InitFS and SignSyst headers. Computing headers hash and finalizing filesystems raw data.
10) Saving InitFS under `initfs.bin`, hashing SignSyst header and saving it into `signsyst-hash.bin`, saving SignSyst under `signsyst.bin`
10) Saving InitFS under `initfs.bin`, hashing SignSyst header and saving it into `sshash.bin`, saving SignSyst under `signsyst.bin`
For the full and detailled layout of InitFS and SignSyst, please see [customs filesystems docs](../blastproof/fs.md).
All hashing operations use SHA3-512. All entropy generations use secure entropy from `/dev/urandom`.
This utility generate four files:
- `initfs.bin` and `signsyst.bin` are the filesystems expected to be flashed into their respectives partitions on disk
- `initfs-footprint.bin` and `signsyst-hash.bin` are files expected to be protected by SPFIE and put inside the ESP partition
- `initfsfp.bin` and `sshash.bin` are files expected to be protected by SBFIE and put inside the ESP partition

27
docs/boottools/keygen.md
View File

@@ -2,16 +2,16 @@
## Introduction
The `keygen` utility is responsible for generating SPFIE keys, files and boot password. This file focus on how these ressources are generated and setuped, not how the verification is made at boot time. The explenations in this file are complementary to the informations in [SPFIE docs](../blastproof/spfie.md).
The `keygen` utility is responsible for generating VFTM keys, SBFIE keys, files and boot password. This file focus on how these ressources are generated and setuped, not how the verification is made at boot time. The explenations in this file are complementary to the informations in [SBFIE docs](../blastproof/sbfie.md) and [VFTM docs](../blastproof/vftm.md).
## Detailled informations
Folder: `Blastproof/keygen`
Source code file: `keygen.cpp`
For building `keygen`, first make sure that you have the `libargon2.a` static library file inside the utility directory. This one and the source of Argon2 for the `keygen` isn't included inside the main Vystem repository but downloaded and compiled by the build script. So you may have to compile Argon2 yourself before compiling `keygen`. To compile `keygen`, use the provided build script `build.sh` while being in the utility directory.
For building `keygen`, use the provided build file. The provided build file doesn't allow yet for the utility to be build outside of Vystem build process.
External libraries:
External libraries provided by `commoncrypto` and used inside `keygen`:
- SPHINCS+, made by the SPHINCS+ team, sourced from [sphincs/sphincsplus](https://github.com/sphincs/sphincsplus), under the MIT-0 license
- tiny_sha3, made by mjosaarinen, sourced from [mjosaarinen/tiny_sha3](https://github.com/mjosaarinen/tiny_sha3), under the MIT license
- Argon2, made by the Argon2 team, sourced from [P-H-C/phc-winner-argon2](https://github.com/P-H-C/phc-winner-argon2), under the CC0-1.0 license
@@ -24,19 +24,22 @@ sudo keygen <path to file 1> <path to file 2> ... <path to file n>
```
You can provide as much paths as you want, as long as they all lead to regular files.
It's recommanded the `keygen` utility be launched as root.
It's necessary the `keygen` utility be launched as root. If the user launched `keygen` using sudo, all created files and folder will be changed of owner to be owned by the user that launched the `keygen` utility.
To transmit the `manifest` keypair to VyBuild, `keygen` expect the `vftm.sock` UNIX socket to be created inside the current folder of execution. The generation process will fail if this socket isn't created by calling entity.
The `VYSTEM_KEYGEN_PASSWORD` environnement variable can be used to transmit a password. If this variable isn't defined of empty, the utility will default on standard input method. This method of transmiting the password should only be used if you are in a test/developpement environnement, and not in production.
## Detailled generation processus
1) Firstly, the `keygen` processus disable all possibilty of process memory dumps or memory swapping before doing anything else. Due the very thight security here, `keygen` should be running as root to ensure his own security.
2) Then, it check for the existence of all the provided files and lock private and public keys buffer as well as signature buffer to ensure that all the secrets aren't dumped on disk due to process fault or swapping.
3) Iterating for all the provided files: loading the file into memory, generating SPHINCS+ keypair for it, computing and signing the SHA3-512 hash of the file content if it's above 1 megabyte, or signing directly the file if under 1 megabyte, securely erasing all temporary keys and signature buffer, saving signature and public key into their dedicated buffer.
4) Saving all signatures into a directory named `sign`, created if non existing, under the name format `<original name>.sig`, then securely erasing the signatures buffer
5) Letting the user enter a boot password. If password is empty, it will ask for a number multiple of 2 to generate this many hexadecimal characters that will serve as password, the default amount of hexadecimal characters being 16. If the user enter a password, the user will have to confirm it
6) Checking that the password fill all criterias: should be (not enforced) at least 12 characters, can (enforced) only include ASCII characters due to EFI shell limitations, shouldn't be longer that 512 characters (enforced), and shoudn't use the numerical pad at all (not well supported in the EFI shell, but the user can still use numbers)
7) Offering the user the possibility to translate their password into a QWERTY representation of it, if they doesn't use a QWERTY keyboard. That way, the user can enter his password like he would do on his keyboard but the password that will be hashed will be the password translated in QWERTY.
1) Firstly, the `keygen` processus disable all possibilty of process memory dumps or memory swapping before doing anything else. Due the very thight security here, `keygen` must be running as root to ensure his own security.
2) It generate all the necessary keys for VFTM. Please see VFTM docs for more informations.
3) Verifying if the `VYSTEM_KEYGEN_PASSWORD` environnement variable exists and acting accordingly.
3) Then, it check for the existence of all the provided files and lock private and public keys buffer as well as signature buffer to ensure that all the secrets aren't dumped on disk due to process fault or swapping.
4) Iterating for all the provided files: loading the file into memory, generating SPHINCS+ keypair for it, computing and signing the SHA3-512 hash of the file content if it's above 1 megabyte, or signing directly the file if under 1 megabyte, securely erasing all temporary keys and signature buffer, saving signature and public key into their dedicated buffer.
5) Saving all signatures into a directory named `sign`, created if non existing, under the name format `<original name>.sig`, then securely erasing the signatures buffer
**Warning:** This feature is still very experimental, the only translation supported is AZERTY to QWERTY and the user experience is still very unsettling. For testing purposes, it's recommanded using a password with no characters differences between keyboard layout such as `test`, for example compatibility between the AZERTY and QWERTY keyboard. A better password inputing experience with support for various keyboard layout is planned for future release.
The 2 following steps are only executed if the standard password input method is selected:
6) Letting the user enter a boot password. If password is empty, it will ask for a number multiple of 2 to generate this many hexadecimal characters that will serve as password, the default amount of hexadecimal characters being 16. If the user enter a password, the user will have to confirm it
7) Checking that the password fill all criterias: should be (not enforced) at least 12 characters, can (enforced) only include ASCII characters due to EFI shell limitations, shouldn't be longer that 512 characters (enforced), and shoudn't use the numerical pad at all (not well supported in the EFI shell, but the user can still use numbers)
8) Converting the final password to UTF16LE for EFI environnement easier compatibilty. Obtaining 32 bytes of entropy from `/dev/urandom`. Hashing the password with the Argon2id algorithm using no parralelism, time cost of 3 and memory cost of 262144, with the salt, resulting in a 96 bytes output.
9) Using the password as seed for generating a master SPHINCS+ keypair. Generating a signature of the public keys of each files concatenated into a single block. Securely erasing both keys of the master keypair.

53
docs/build.md
View File

@@ -2,53 +2,56 @@
## Introduction
In this file, we will see how you can build and boot successfully into a VM containing a Vystem disk image. It's recommanded to read the whole documentation for better understanding of why certains parts are necessary.
In this file, we will see how you can build and boot successfully into a VM containing a Vystem disk image or boot a Vystem instance on a real computer. It's recommanded to read the whole documentation for better understanding of why certains parts are necessary.
## Prerequisites
You should have cloned the full repository on your device. Please keep in mind that Vystem hasn't been tested on real hardware yet, so the only way to test it is through a virtual machine.
You should have cloned the full repository on your device. You can uses Vystem in two cases:
- in a VM
- on a real computer
Vystem has been designed to be build and run from a standard Linux system. Any Linux distro should work. A WSL environnement can also work but the VM might be very slow, because of KVM unavailability. Make sure you have KVM enable and accessible before starting, or else remove the `--enable-kvm` argument at the end of the build script.
In both cases, Vystem has been designed to be build and run from a standard Linux system. Any Linux distro should work. A WSL environnement can also work but, if you are testing in a VM, the VM might be very slow or unstable in the benchmark results, because of KVM unavailability. Make sure you have KVM enable and accessible before starting, or else remove the `-accel kvm` argument at the end of the `build.json` file.
Before starting, please make sure that you have the following binary reachable in your terminal path:
```
python3 python nasm g++ gcc iasl git make sed wget unzip fallocate parted mkfs.fat sgdisk partprobe losesetup qemu-system-x86_64
python3 python nasm g++ gcc iasl git make unzip
```
Finally, please make sure that you have an internet connexion (required for downloading additionnal assets like EDK II and Argon2) as well as root access/sudo permissions, as it is recommanded for running `keygen` and required for creating the disk image. Root permissions is only obtained as soon as it's needed. If you don't want to give root permissions, you can also run the commands inside the build script one by one just to be sure.
You may also want `qemu-system-x86_64` if you are testing Vystem in a VM.
Finally, please make sure that you have an internet connexion if EDK II isn't yet setuped as well as root access/sudo permissions, as it is necessary for running `keygen`. Root permissions is only obtained as soon as it's needed.
## Build and run
Once you are ready, run the following command while being in the root folder:
``` bash
./build.sh
vybuild/build.sh
```
This will download, compile, and assemble every asset together. This can take a while depending on your internet connexion, since EDK II is quite heavy and will be downloaded from it's source repository and locally compiled, with OVMF. For the moment, using packaged versions of EDK II and/or OVMF provided by packages managers isn't supported.
This script is designed to be used only from the root folder.
This will build the `vybuild` executable inside the `vybuild` folder. This part isn't multithreaded and can take up to a whole minute depending on your device.
Then, to start the build process, you can start VyBuild on the `build.json` file in the root folder like this:
``` bash
vybuild/vybuild build.json
```
This will compile, assemble and generate every asset together. This can take a while depending on your internet connexion, since EDK II is quite heavy and will be downloaded from it's source repository and locally compiled, with OVMF. For the moment, using packaged versions of EDK II and/or OVMF provided by packages managers isn't supported.
The VM will automatically be launched with 4 gigabytes of RAM (you can obviously raise or lower that amount depending on your device), KVM enabled and serial port output redirected inside the terminal.
## Detailled build process
To generate the disk image without starting the VM, you can use the following command:
``` bash
vybuild/vybuild build.json img
```
The folder used for InitFS base is named `initfs_dir` and is placed in the root folder of the repository.
The build process is as follow:
1) Checking virtual memory layout
2) Generating payloads for TAB subsystem
3) Compiling Keycard and putting it into `initfs_dir`
4) Compiling Vyld
5) Compiling Shelter and putting it into `initfs_dir`
6) Cloning EDK II, building required tools and OVMF
7) Compiling `bootanim` and generating boot animation
8) Compiling `fontgen` and generating font
9) Compiling `initfsgen` and generating InitFS and SignSyst
10) Patching bootloader configuration template to insert random partition GUID and fixed partition type GUID
11) Cloning and building Argon2, compiling `keygen` and generating required files for SPFIE
12) Compiling bootloader
13) Generating disk image
14) Launching virtual machine
Then, just flash the generated disk image on a USB disk and boot from it on real hardware.
**Warning:** depending the properties of the firmware and available video cards on the hardware used to test Vystem, the EFI GOP framebuffer might not survive after ExitBootServices.
## Troubleshooting
If anything goes wrong, the script will stop and show you the error. Sometimes it's just a dependency that isn't reachable. Carefully analyse the error message and try to install the missing dependencies, also check that the required dependencies listed above are all installed.
If anything goes wrong, VyBuild will stop and show you the error. Sometimes it's just a dependency that isn't reachable. Carefully analyse the error message and try to install the missing dependencies, also check that the required dependencies listed above are all installed.
If it's related to EDK II (which occur the majority of the time), it could be that a new update in EDK II has broken something. Don't hesitate to open an issue, and this should be fix fairly quickly.

9
docs/commonlib.md Normal file
View File

@@ -0,0 +1,9 @@
# Commonlib
## Introduction
Commonlib is a very simple C library that contains informations useful to all parts of Vystem. All the source code of Commonlib is stored inside the `common` folder.
## Versions
The header `versions.h` defines the versions number of various Vystem components.

10
docs/index.md
View File

@@ -1,6 +1,6 @@
# Vystem Docs
**Warning:** this documentation only include currently implemented elements and doesn't include plannned future elements, except some confirmed planned updates scattered within the documentation
**Warning:** this documentation only includes currently implemented elements and doesn't include plannned future elements, except some confirmed planned updates scattered within the documentation
## Introduction
@@ -11,9 +11,11 @@ Welcome to Vystem documentation. This documentation is divided into severals par
Vystem is an OS made of severals components which can be divided into severals categories:
1) [Blastproof](blastproof/index.md): the bootloader
2) [Bootloader tools](boottools/index.md): all the tools needed to generate components for the boot process
3) [Vyld](vyld/index.md): the VYX executable format linker and VYX executable format specification
4) [Shelter](shelter/index.md): the kernel
5) [Kernel tools](kerneltools/index.md): all the `.py` scripts used for various usage around the kernel
3) [Vyx](vyx.md): Vystem Executable format
4) [VyBuild](vybuild/index.md): Vystem build system
5) [Commonlib](commonlib.md): very simple library that contains informations useful to all parts of Vystem
6) [Shelter](shelter/index.md): the kernel
7) [Kernel tools](kerneltools/index.md): all the `.py` scripts used for various usage around the kernel
## Ressources

1
docs/licenses.md
View File

@@ -21,6 +21,7 @@ Original source code | Author | License
[P-H-C/phc-winner-argon2](https://github.com/P-H-C/phc-winner-argon2) | Argon2 team | CC0-1.0
[tianocore/edk2](https://github.com/tianocore/edk2) | Tianocore and all the contributing companies | BSD-2-Clause-Patent
[serge1/ELFIO](https://github.com/serge1/ELFIO) | Serge Lamikhov-Center | MIT
[nlohmann/json](https://github.com/nlohmann/json) | Niels Lohmann | MIT
EDK II copyright: Copyright (c) Intel Corporation and other contributors.

61
docs/roadmap.md
View File

@@ -1,13 +1,60 @@
# Roadmap
## Version 0.1 (actual)
## Version 0.1 (previous)
Initial release
## Version 0.2 (future versions with planned additions)
## Version 0.2 (actual)
- Vyld overhaul, support for multiples compilations modes with JSON config files for project settings
- VYX format overhaul, supporting various sections and more feature
- Blastproof: custom keyboard layout driver
- Shelter: Support for IDT, GDT, TSS, interruptions, interruptions handler, ACPI timers table parsing and ACPI driver
- Shelter: Slab allocators overhaul and performance boosts
- Blastproof:
- New boot password screen, with power options
- Support for custom keyboard layouts using correspondance tables
- Config is now stored in .cfg files
- VFTM support
- Partition tables is now parsed without relying on UEFI firmware
- New configuration key: `kernel_kbd_events_queue_capacity`
- Vyx:
- Support for payloads with custom mappings options
- Initfsgen:
- The `initfs-footprint.bin` file is now named `initfsfp.bin`
- The `signsyst-hash.bin` file is now named `sshash.bin`
- Keygen:
- Removed layout conversion feature
- The utility will now check if a password is provided in the `VYSTEM_KEYGEN_PASSWORD` environnement variable
- The manual password setup will still be proposed if `VYSTEM_KEYGEN_PASSWORD` isn't defined or is empty
- This method of inputing password via `VYSTEM_KEYGEN_PASSWORD` should ONLY be used in a developpement environnement
- Keygen now generate root and manifest keypairs, and expect a `vftm.sock` UNIX socket to transmit manifest keypair to a compatible build system
- VyBuild:
- A new build system, see documentation for more informations
- Commoncrypto:
- A new library made of three cryptographic libraries, required for Keygen, Initfsgen and VyBuild
- Vystem FAT Trusted Manifest:
- A new signed manifest, injected inside the reserved sectors of the ESP partition FAT32
- It's signed using a combinaison of Keygen and VyBuild
- It allow to verify the link between ESP partition and current disk, as well as FAT32 integrity and self bootloader verification
- Common Lib:
- In the `common` folder, new headers that will be accessible to all components of Vystem will be put here
- For the moment, it only includes the reference version of each component
- Shelter:
- Can now manage TSS, GDT and IDT
- Can now parse root ACPI table and MADT
- Can now interact with LAPIC, estimating CPU and LAPIC frequency
- Can now start APs
- Can now manages IOAPIC devices and GSI lines as well as legacy IRQs
- Added thread safety to print, timers and malloc. Added iprint/iprintf/sprintf.
- Added Device System, endpoints from APIC, TSC, kernel configuration, memory and SMP subsystem are available
- Device System is not available for the input subsystem yet, as it would requires Ring 3 architecture, that isn't ready for now
- Add very basic framebuffer support for early kernel progress bar
- Image build processus:
- The signatures are now stored as `<filename without original extension>.sig` files
- The signatures are now stored inside `EFI/BPSIGN` folder
- The bootloader files are now stored inside `EFI/BPFILES` folder
## Version 0.3 (future release with planned additions)
- Updated `SH_STATUS` struct
- New Modular Daemon Unit with the first type being implemented: kernel library. Run in ring 3 with very basic API
- First features of the VFS, with InitFS drivers
- MCFG ACPI table parsing, PCIe devices enumeration, first MSI/MSI-X implementation
- SATA driver, volumes enumeration with integration to the VFS
- Upgrade to the Device System, with new APIs for the input subsystem and VFS, as well as Device System querying through the VFS

151
docs/shelter/bench.md
View File

@@ -5,10 +5,10 @@ In this file, you will be able to see examples of benchmark results. These are p
## Tests environnement
- Host OS: EndeavourOS x86_64
- Host kernel: Linux 6.19.10-arch1-1
- Host kernel: Linux 7.0.10-arch1-1
- Host CPU: Ryzen 7 9800x3d @ 5.27 GHz
- Environnement: VM running in qemu-system-x86_64 version 10.2.2
- VM parameters: KVM enabled, 4096M of RAM, cpu set to host, running in single thread
- Environnement: VM running in qemu-system-x86_64 version 11.0.0
- VM parameters: KVM enabled, 4096M of RAM, cpu set to host
The following tests are shown in the order the kernel run them. Please keep in mind that Pez isn't available until the Pez physical plane benchmark.
@@ -17,31 +17,31 @@ The following tests are shown in the order the kernel run them. Please keep in m
Physical regions object slab allocator:
```
Result for benchmark "allocations" :
Min: 141 | Med: 235 | Avg: 2736 | Max: 3194120 | Total : 27362977 (TSC)
[P00-P90] 141-282 : [##################--] 90%
[P90-P99] 282-282 : [#-------------------] 9%
[P99-Max] 282-3194120 : [--------------------] 1%
Min: 141 | Med: 235 | Avg: 3051 | Max: 3175320 | Total : 30517523 (TSC)
[P00-P90] 141-235 : [##################--] 90%
[P90-P99] 235-282 : [#-------------------] 9%
[P99-Max] 282-3175320 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 141 | Avg: 132 | Max: 376 | Total : 1320700 (TSC)
Min: 94 | Med: 141 | Avg: 131 | Max: 611 | Total : 1313415 (TSC)
[P00-P90] 94-141 : [##################--] 90%
[P90-P99] 141-141 : [#-------------------] 9%
[P99-Max] 141-376 : [--------------------] 1%
[P99-Max] 141-611 : [--------------------] 1%
```
Virtual regions object slab allocator:
```
Result for benchmark "allocations" :
Min: 141 | Med: 235 | Avg: 2781 | Max: 2840445 | Total : 27816339 (TSC)
[P00-P90] 141-282 : [##################--] 90%
[P90-P99] 282-329 : [#-------------------] 9%
[P99-Max] 329-2840445 : [--------------------] 1%
Min: 141 | Med: 235 | Avg: 3051 | Max: 3175320 | Total : 30517523 (TSC)
[P00-P90] 141-235 : [##################--] 90%
[P90-P99] 235-282 : [#-------------------] 9%
[P99-Max] 282-3175320 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 141 | Avg: 130 | Max: 517 | Total : 1304109 (TSC)
Min: 94 | Med: 141 | Avg: 131 | Max: 611 | Total : 1313415 (TSC)
[P00-P90] 94-141 : [##################--] 90%
[P90-P99] 141-141 : [#-------------------] 9%
[P99-Max] 141-517 : [--------------------] 1%
[P99-Max] 141-611 : [--------------------] 1%
```
## Results for radix node slab allocator (PBA based)
@@ -49,16 +49,16 @@ Min: 94 | Med: 141 | Avg: 130 | Max: 517 | Total : 1304109 (TSC)
Radix node slab allocator (PBA based):
```
Result for benchmark "allocations" :
Min: 141 | Med: 188 | Avg: 441 | Max: 534531 | Total : 4410339 (TSC)
Min: 141 | Med: 235 | Avg: 426 | Max: 624395 | Total : 4264498 (TSC)
[P00-P90] 141-235 : [##################--] 90%
[P90-P99] 235-282 : [#-------------------] 9%
[P99-Max] 282-534531 : [--------------------] 1%
[P99-Max] 282-624395 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 94 | Avg: 97 | Max: 329 | Total : 973370 (TSC)
Min: 94 | Med: 94 | Avg: 95 | Max: 611 | Total : 954664 (TSC)
[P00-P90] 94-94 : [##################--] 90%
[P90-P99] 94-141 : [#-------------------] 9%
[P99-Max] 141-329 : [--------------------] 1%
[P99-Max] 141-611 : [--------------------] 1%
```
## Results for radix trees subsystem
@@ -66,28 +66,28 @@ Min: 94 | Med: 94 | Avg: 97 | Max: 329 | Total : 973370 (TSC)
Radix trees subsystem:
```
Result for benchmark "insertions into radix trees" :
Min: 2350 | Med: 3196 | Avg: 5194 | Max: 544448 | Total : 51944259 (TSC)
[P00-P90] 2350-3666 : [##################--] 90%
[P90-P99] 3666-134373 : [#-------------------] 9%
[P99-Max] 134373-544448 : [--------------------] 1%
Min: 2350 | Med: 3196 | Avg: 5364 | Max: 668575 | Total : 53644531 (TSC)
[P00-P90] 2350-3713 : [##################--] 90%
[P90-P99] 3713-134608 : [#-------------------] 9%
[P99-Max] 134608-668575 : [--------------------] 1%
Result for benchmark "reading into radix trees" :
Min: 470 | Med: 517 | Avg: 553 | Max: 31631 | Total : 5537399 (TSC)
Min: 470 | Med: 517 | Avg: 543 | Max: 22466 | Total : 5432213 (TSC)
[P00-P90] 470-564 : [##################--] 90%
[P90-P99] 564-987 : [#-------------------] 9%
[P99-Max] 987-31631 : [--------------------] 1%
[P90-P99] 564-658 : [#-------------------] 9%
[P99-Max] 658-22466 : [--------------------] 1%
Result for benchmark "searching value with lower bound key" :
Min: 423 | Med: 893 | Avg: 923 | Max: 30832 | Total : 9234184 (TSC)
[P00-P90] 423-1128 : [##################--] 90%
[P90-P99] 1128-1645 : [#-------------------] 9%
[P99-Max] 1645-30832 : [--------------------] 1%
Min: 470 | Med: 846 | Avg: 902 | Max: 156134 | Total : 9021368 (TSC)
[P00-P90] 470-1081 : [##################--] 90%
[P90-P99] 1081-1363 : [#-------------------] 9%
[P99-Max] 1363-156134 : [--------------------] 1%
Result for benchmark "deleting values" :
Min: 1645 | Med: 1880 | Avg: 1888 | Max: 35203 | Total : 18889770 (TSC)
[P00-P90] 1645-2021 : [##################--] 90%
[P90-P99] 2021-2162 : [#-------------------] 9%
[P99-Max] 2162-35203 : [--------------------] 1%
Min: 1598 | Med: 1833 | Avg: 1860 | Max: 33276 | Total : 18607018 (TSC)
[P00-P90] 1598-1974 : [##################--] 90%
[P90-P99] 1974-2256 : [#-------------------] 9%
[P99-Max] 2256-33276 : [--------------------] 1%
```
## Results for Pez physical plane
@@ -95,55 +95,66 @@ Min: 1645 | Med: 1880 | Avg: 1888 | Max: 35203 | Total : 18889770 (TSC)
Pez physical plane:
```
Result for benchmark "allocations for single page" :
Min: 1833 | Med: 2444 | Avg: 3496 | Max: 8225 | Total : 3496518 (TSC)
[P00-P90] 1833-5922 : [##################--] 90%
[P90-P99] 5922-6815 : [#-------------------] 9%
[P99-Max] 6815-8225 : [--------------------] 1%
Min: 1786 | Med: 2444 | Avg: 3348 | Max: 6862 | Total : 3348656 (TSC)
[P00-P90] 1786-5593 : [##################--] 90%
[P90-P99] 5593-6721 : [#-------------------] 9%
[P99-Max] 6721-6862 : [--------------------] 1%
Result for benchmark "free for single page" :
Min: 1974 | Med: 2820 | Avg: 3807 | Max: 10105 | Total : 3807376 (TSC)
[P00-P90] 1974-6862 : [##################--] 90%
[P90-P99] 6862-8131 : [#-------------------] 9%
[P99-Max] 8131-10105 : [--------------------] 1%
Min: 1927 | Med: 2820 | Avg: 3840 | Max: 33981 | Total : 3840417 (TSC)
[P00-P90] 1927-6721 : [##################--] 90%
[P90-P99] 6721-8037 : [#-------------------] 9%
[P99-Max] 8037-33981 : [--------------------] 1%
Result for benchmark "allocations for multiple pages" :
Min: 2209 | Med: 5593 | Avg: 5716 | Max: 35861 | Total : 5716798 (TSC)
[P00-P90] 2209-7191 : [##################--] 90%
[P90-P99] 7191-8178 : [#-------------------] 9%
[P99-Max] 8178-35861 : [--------------------] 1%
Min: 2303 | Med: 5405 | Avg: 5568 | Max: 33793 | Total : 5568231 (TSC)
[P00-P90] 2303-7050 : [##################--] 90%
[P90-P99] 7050-8131 : [#-------------------] 9%
[P99-Max] 8131-33793 : [--------------------] 1%
Result for benchmark "free for multiple pages" :
Min: 1927 | Med: 4935 | Avg: 5095 | Max: 31772 | Total : 5095599 (TSC)
[P00-P90] 1927-7896 : [##################--] 90%
[P90-P99] 7896-10011 : [#-------------------] 9%
[P99-Max] 10011-31772 : [--------------------] 1%
Min: 2115 | Med: 4841 | Avg: 5066 | Max: 29892 | Total : 5066506 (TSC)
[P00-P90] 2115-7849 : [##################--] 90%
[P90-P99] 7849-9635 : [#-------------------] 9%
[P99-Max] 9635-29892 : [--------------------] 1%
```
## Results for malloc subsystem
Malloc subsystem:
```
Result for benchmark "sh_malloc for small size" :
Min: 188 | Med: 423 | Avg: 535 | Max: 380277 | Total : 5358752 (TSC)
[P00-P90] 188-799 : [##################--] 90%
[P90-P99] 799-1363 : [#-------------------] 9%
[P99-Max] 1363-380277 : [--------------------] 1%
Result for benchmark "allocations for single page" :
Min: 1786 | Med: 2444 | Avg: 3348 | Max: 6862 | Total : 3348656 (TSC)
[P00-P90] 1786-5593 : [##################--] 90%
[P90-P99] 5593-6721 : [#-------------------] 9%
[P99-Max] 6721-6862 : [--------------------] 1%
Result for benchmark "sh_free for small size" :
Min: 141 | Med: 329 | Avg: 403 | Max: 27401 | Total : 4039556 (TSC)
[P00-P90] 141-658 : [##################--] 90%
[P90-P99] 658-1128 : [#-------------------] 9%
[P99-Max] 1128-27401 : [--------------------] 1%
Result for benchmark "free for single page" :
Min: 1927 | Med: 2820 | Avg: 3840 | Max: 33981 | Total : 3840417 (TSC)
[P00-P90] 1927-6721 : [##################--] 90%
[P90-P99] 6721-8037 : [#-------------------] 9%
[P99-Max] 8037-33981 : [--------------------] 1%
Result for benchmark "sh_malloc for pages allocations" :
Min: 4935 | Med: 11468 | Avg: 11890 | Max: 47893 | Total : 11890013 (TSC)
[P00-P90] 4935-16450 : [##################--] 90%
[P90-P99] 16450-20257 : [#-------------------] 9%
[P99-Max] 20257-47893 : [--------------------] 1%
Result for benchmark "allocations for multiple pages" :
Min: 2303 | Med: 5405 | Avg: 5568 | Max: 33793 | Total : 5568231 (TSC)
[P00-P90] 2303-7050 : [##################--] 90%
[P90-P99] 7050-8131 : [#-------------------] 9%
[P99-Max] 8131-33793 : [--------------------] 1%
Result for benchmark "sh_free for pages allocations" :
Min: 5593 | Med: 11421 | Avg: 12005 | Max: 50760 | Total : 12005398 (TSC)
[P00-P90] 5593-17296 : [##################--] 90%
[P90-P99] 17296-21432 : [#-------------------] 9%
[P99-Max] 21432-50760 : [--------------------] 1%
Result for benchmark "free for multiple pages" :
Min: 2115 | Med: 4841 | Avg: 5066 | Max: 29892 | Total : 5066506 (TSC)
[P00-P90] 2115-7849 : [##################--] 90%
[P90-P99] 7849-9635 : [#-------------------] 9%
[P99-Max] 9635-29892 : [--------------------] 1%
```
## Results for queues
Keyboard events queue:
```
Result for benchmark "pushing objects" :
Min: 141 | Med: 141 | Avg: 221 | Max: 546892 | Total : 2215580 (TSC)
[P00-P90] 141-188 : [##################--] 90%
[P90-P99] 188-235 : [#-------------------] 9%
[P99-Max] 235-546892 : [--------------------] 1%
```

41
docs/shelter/bootconfig.md
View File

@@ -19,6 +19,10 @@ typedef struct __attribute__((aligned(8))) {
sh_bool log_disable_serial_port;
sh_bool disable_serial_port;
sh_uint16 log_ring_size;
sh_uint64 acpi_rsdp;
sh_uint8 acpi_ver;
sh_uint16 kbd_events_queue_capacity;
sh_conf_FB_CONFIG fb_config;
sh_uint8 sig_end[8];
} sh_conf_BOOT_CONFIG;
```
@@ -62,3 +66,40 @@ The starting signature must be `ShCfgBeg` in ASCII and the starting signature mu
**log_ring_size:**
- Type: 2 bytes unsigned integer
- Description: define the amount of pages used for logging ring buffer
**acpi_rsdp:**
- Type: 8 bytes unsigned integer
- Description: the physical address of the ACPI RDSP
**acpi_ver:**
- Type: 1 byte unsigned integer
- Description: define the version of ACPI provided by the firmware, can be 1 or 2 depending on the ACPI version GUID found by Blastproof
**kbd_events_queue_capacity:**
- Type: 2 bytes unsigned integer
- Description: the capacity of each keyboard events queue. The value of this key is provided by the `kernel_kbd_events_queue_capacity` from the Blastproof boot config
**fb_config:**
- Type: custom struct
- Description: informations on the early boot framebuffer, intended to work with GOP framebuffer information
## FB config structure
The `fb_config` structure is defined like that:
``` C
typedef struct __attribute__((aligned(8))) {
sh_bool fb_present;
sh_page_VIRTUAL_ADDRESS fb_pa;
sh_uint32 white_pixel_value;
sh_uint32 gray_pixel_value;
sh_uint64 size_in_bytes;
sh_uint32 fb_height;
sh_uint32 fb_width;
sh_uint16 text_x;
sh_uint16 text_y;
sh_uint16 text_width;
sh_uint16 text_height;
} sh_conf_FB_CONFIG;
```
The `fb_present` field should be set to `false` if anything impeach the framebuffer to be fully operational. The `fb_pa` field is the physical address of the framebuffer. The kernel will map the framebuffer itself. The `white_pixel_value` and `gray_pixel_value` contain the encoded pixel data for the color white and gray. The `text_x`, `text_y`, `text_width` and `text_height` serve the purpose of defining an area that the kernel should erase before starting using the progress bar. They are named like that because most of the time, it will be a text placed by the bootloader that the kernel should erase. Others fields are self-explenatory.

5
docs/shelter/bootcontract.md
View File

@@ -11,7 +11,7 @@ We assume the following:
- you have a bootloader capable of mapping things into into pages table pool
- your bootloader is capable of generating Shelter boot configuration and Shelter memory map
- your bootloader is capable of allocating or call the UEFI firmware to allocates pages to copy data into them
- your bootloader is capable of parsing VYX executable
- your bootloader is capable of parsing VYX executable, including payloads
- your bootloader is capable of accessing the content of the Shelter VYX executable and keycard binary
While this documentation is redacted like a tutorial, this isn't a tutorial nor intended to be understand as one.
@@ -24,6 +24,7 @@ Your bootloader should be allocating 4KB physical pages ranges and store their p
- page(s) for keycard should be `EfiLoaderCode` type. Keycard should fit into one page, but you are free to allocate more if you want
- page(s) for Shelter boot configuration, memory map and pages table pool should be `EfiLoaderCode`. Shelter boot configuration should fit into one page, but you are free to allocate more if you want
- pages for Shelter logging ring should be `EfiLoaderCode` type
- pages for Shelter VYX binary payloads should be `EfiLoaderCode` type or `EfiLoaderData` type depending on the type of payload
Note: even if the logging ring size specified in the configuration is less than 4 pages, at least 4 pages should be allocated, because the kernel logs some things before loading the boot configuration that specify that log in ring buffer is disabled
@@ -37,6 +38,7 @@ The following steps should be completed before mapping the pages:
- copying the content of the `.text`, `.data` and `.rodata` section inside their respectives pages range
- copying the content of Keycard inside his pages range
- copying the content of the boot configuration inside his pages range
- parsing the VYX binary for payloads, allocating pages for them, zeroing those pages and copying the content of each payload into their respectives pages
## Step 3: mapping pages
@@ -52,6 +54,7 @@ The following pages ranges will be mapped at the following virtual address in th
- Shelter boot configuration pages should be mapped at VA `0x00180000`, with no execution and read-write permissions
- memory map pages should be mapped at VA `0x00190000`, with no execution and read-only permissions
- logging ring pages should be mapped at VA `0xFFFFFFFFF0000000`, with no execution and read-write permissions
- each payload should be mapped at the indicated VA, with the indicated permissions
## Step 4: completing configuration

96
docs/shelter/bootprocess.md
View File

@@ -34,7 +34,9 @@ Starting here, all logs go through the standard log API.
2.1.2) Initializing current PTP using boot configuration information
2.1.3) Parsing and validating memory map, initializing physical pages bitmap
2.1.3) Using the framebuffer informations, initializing the framebuffer subsystem
2.1.4) Parsing and validating memory map, initializing physical pages bitmap
### 2.2) Pez dependencies subsystems initialization
@@ -62,19 +64,19 @@ Starting here, all logs go through the standard log API.
2.2.5) Benchmarking and testing radix tree subsystem
## 2.3) Pez physical subsystem initialization
### 2.3) Pez physical subsystem initialization
2.3.1) Creating Pez physical allocation plane
2.3.2) Benchmarking and testing Pez physical allocation plane
## 2.4) Pez virtual subsystem initialization
### 2.4) Pez virtual subsystem initialization
2.4.1) Creating Pez virtual allocation plane for kernel heap
(Pez virtual plane for kernel heap isn't benchmarked and tested because it's exactly the same algorithm and metadatas under the hood)
## 2.5) Kernel heap initialization
### 2.5) Kernel heap initialization
2.5.1) Initializing kernel heap structure
@@ -87,3 +89,89 @@ Starting here, all logs go through the standard log API.
2.5.2.c) Adding first slab and verifying slab metadata
2.5.3) Benchmarking and testing kernel heap
## 3) Basic CPU features configuration
### 3.1) TSS structure initialization
3.1.1) Initializing TSS structure
### 3.2) GDT structure initialization
3.2.1) Generating GDT structure
3.2.2) Loading GDT register
3.2.3) Loading TSS register
### 3.3) IDT structure initialization
3.3.1) Generating IDT structure
3.3.2) Loading IDT register
## 4) ACPI parsing
### 4.1) Parsing RSDP
### 4.2) Parsing XSDT
### 4.3) Parsing all availables tables
### 4.4) Parsing MADT table
4.4.1) Counting all LAPICs, IOAPICs and ISOs
4.4.2) Initializing devices system for LAPICs and IOAPICs
4.4.3) Registering all informations inside MADT table
4.4.3.a) Registering, initializing and binding all LAPICs
4.4.3.b) Registering, initializing and binding all IOAPICs
4.4.3.c) Registering all ISOs
## 5) Devices bootstrap
### 5.1) Applications= processors (AP) bootstrap
5.1.1) Disabing all GSI lines
5.1.2) Checking somes CPU ID flags for logging
5.1.2.a) Checking hypervisor flag
5.1.2.b) Checking TSC constant flag
5.1.3) Trying to obtain CPU frequency
5.1.3.a) Trying to obtain CPU frequency from CPU ID
5.1.3.b) Trying to obtain CPU frequency from TSC and PIT
5.1.4) Initializing LAPIC for bootstrap CPU
5.1.5) Estimating LAPIC frequency
5.1.6) Loading 32 bits mode GDT for AP bootstrap
5.1.7) Preparing for AP launch by filling various structures
5.1.8) Launching all APs in an infinite loop
---
Starting here, all logs go through the standard library print functions.
---
5.1.9) Switching legacy IRQ management from PIT to IOAPIC subsystem
### 5.2) Keyboard bootstrap
5.2.1) Initializing keyboard subsystem
5.2.2) Initializing PS2 keyboard driver
5.2.3) Disabing PS2 keyboard scanning for the rest of kernel boot process

116
docs/shelter/cpu/ap.md Normal file
View File

@@ -0,0 +1,116 @@
# Application Processors subsystem
## Introduction
The Application Processors (AP) subsystem is responsible for managing SMP-related CPU abstractions, including CPU descriptors and per-CPU data structures. It also handles the preparation and bootstrap sequence required to bring up secondary processors. This subsystem is defined inside `shelter/lib/include/cpu/ap.h` and implemented inside `shelter/lib/include/cpu/ap.c`.
## Overview
This subsystem is responsible for three things:
- defining the structures used to describe CPUs and storing per-CPU metadatas
- preparing all necessary structures in order to start the APs
- starting the APs
### CPU structures
The AP subsystem defines two structures related to CPU metadatas:
**`sh_ap_CPU_STRUCT`:**
``` C
#pragma pack(1)
typedef struct {
sh_page_VIRTUAL_ADDRESS c_entry_point_stack_top_va;
sh_uint32 lapic_id;
sh_uint16 tss_selector;
sh_ap_PER_CPU *per_cpu;
} sh_ap_CPU_STRUCT;
#pragma pack()
```
The CPU struct is mainly used for the AP trampoline: it contain informations specific to each AP that will allow them to reach the C entry point. It also contains a pointer to the per-CPU struct of each CPU.
**`sh_ap_PER_CPU`:**
``` C
#pragma pack(1)
typedef struct {
sh_uint64 cpu_id;
sh_uint32 bytes_outputed;
sh_bool timer_state;
char *temp_buffer;
sh_uint64 temp_buffer_size;
} sh_ap_PER_CPU;
#pragma pack()
```
The per-CPU struct is used for per-CPU metadatas like LAPIC timer state, per-CPU temporary buffer, logical CPU id and various per-CPU metadatas for differents kernel subsystems.
The logical CPU id is an id assigned to each processor. The bootstrap processor always obtain the id 0, and there is no gap in the logical CPU id allocation space. It allow for an alternative to LAPIC ids and ACPI processors id, which can contain gaps in their allocation space.
The GS base register of each AP contain a pointer to the CPU struct of each AP. The CPU and per-CPU struct of the bootstrap processor is created at the start of the kernel entry point and then elaborated during the AP bootstrap procedure. It isn't stored inside the AP's CPU struct array.
### Preparing all necessary structures for AP bootstrap
Since all APs start in 16 bits mode, Shelter uses the following **physical** address to put importants compoments there:
- `0x7000`: the physical address of the AP trampoline, shared by all APs
- `0x8000`: the physical address of the 32 bits GDT, necessary for the 32 bits far jump
- `0x9000`: the physical address of the AP bootstrap struct
All those addresses are identity mapped in the bootstrap CPU page table. Additionally, the AP trampoline loaded from the VYX binary of Shelter is mapped at the virtual address `0xA000`.
The AP bootstrap struct is a structure that contains all commons addresses used by all APs:
``` C
#pragma pack(1)
typedef struct {
sh_page_PHYSICAL_ADDRESS lapic_base_pa;
sh_page_VIRTUAL_ADDRESS c_entry_point_va;
sh_page_PHYSICAL_ADDRESS page_table_pa;
sh_page_VIRTUAL_ADDRESS cpu_struct_base_area;
sh_page_VIRTUAL_ADDRESS shared_gdt_64_va;
sh_gdt_GDTR gdt_64_gdtr;
sh_idt_IDT *idt;
} sh_ap_AP_BOOTSTRAP;
#pragma pack()
```
It contain things such as the LAPIC memory mapped area PA, the C entry point VA, the page table PA, the CPU struct array VA, the shared GDT VA for the 64 bits far jump, the GDTR for the said GDT and the IDT VA.
During the kernel boot process, all APs share the same page table as the bootstrap processor for simplicity.
First, the 32 bits GDT is generated and placed at PA `0x8000` by `sh_ap_load_gdt_32()`. Then the rest of the AP bootstrap preparation is handled by `sh_ap_prepare_for_smp_launch()`
Here is the detailled processus:
1) Identity-mapping the page at PA `0x9000` to initialize obvious values like C entry point VA, page table PA and LAPIC base PA.
2) Allocating the array of CPU struct on the heap (indexed by LAPIC id), and initializing the corresponding value in the AP bootstrap struct.
3) Allocating the array of TSS on the heap.
4) Looping on all availables LAPIC to fill all CPU struct: allocating the stack for each AP, filling the LAPIC id, filling the TSS of each AP, allocating the multiple stacks for each TSS, filling the per-CPU struct of each AP.
5) Filling the global GDT for all APs with all TSS, completing the AP bootstrap struct, initializing the AP start state.
### Starting the APs
Once everything is ready, `sh_ap_start_ap_boot_procedure()` is called once to start all APs. The procedure is as follow:
1) Copying the AP trampoline from VA `0xA000` to VA/PA `0x7000`.
2) Filling CPU struct and per-CPU struct of the bootstrap CPU.
3) Sending INIT IPI and waiting 10 millisecond.
4) Sending SIPI and waiting for all APs to start. At this point, the `sh_print` standard library feature is ready.
All APs will start in 16 bits mode and execute the following sequence of action:
1) Disabling interrupts
2) Load 32 bits GDT, enabling 32 bits mode in CR0 and executing the far jump to get to 32 bits mode
3) Reading LAPIC base, extracting LAPIC id and keeping it in a register
4) Enabling PAE, loading page table PA into CR3, enabling LME and NXE bits
5) Enabling pagging, and executing the far jump to get to 64 bits mode
6) Locating the CPU struct for this CPU using the CPU LAPIC id
7) Loading the local CPU stack
8) Loading 64 bits shared GDT, reloading CS register
9) Loading task register and GS register
10) Jumping to C entry point
The C entry point is the function named `sh_ap_entry_point()`. It does the following things:
1) Initialize the LAPIC of the AP
2) Load the shared IDT
3) Confirm to the bootstrap CPU that everything is ready for this AP by updating the corresponding AP start state and goes into an infinite loop
If the bootstrap CPU doesn't receives any confirmation from one of the AP, the boot process is halted.

9
docs/shelter/cpu/asmint.md
View File

@@ -11,3 +11,12 @@ Here is a list of all the wrappers available. They are all defined as `static in
- `outb`: take a `sh_uint16` for specifying the port and a `sh_uint8` for specifying the byte to output
- `rdtsc`: take no arguments and return the current TSC as a `sh_uint64`
- `invlpg`: take a `void *` as an address and return nothing
- `lgdt`: take a `void *` as an address and return nothing
- `ltr`: take a `sh_uint16` as a TSS selector and return nothing
- `sidt`: take a `sh_idt_IDTR*` as a pointer and return nothing
- `sti`: take no arguments and return nothing
- `cli`: take no arguments and return nothing
- `cpuid`: take 2 `sh_uint32`: `leaf`, `subleaf`, and 2 `sh_uint32*`: `eax`, `ebx`, `ecx`, `edx`, return nothing
- `mfence`: take no arguments and return nothing
- `lfence`: take no arguments and return nothing
- `sfence`: take no arguments and return nothing

3
docs/shelter/cpu/cpuabstract.md
View File

@@ -9,3 +9,6 @@ This component of the Shelter kernel allow for basic CPU functions abstractions,
1) [ASM instructions](asmint.md)
2) [Serial outputing API](serial.md)
3) [TSC API](tsc.md)
4) [PIC API](pic.md)
5) [PIT API](pit.md)
6) [Application processors subsystem](ap.md)

13
docs/shelter/cpu/pic.md Normal file
View File

@@ -0,0 +1,13 @@
# PIC API
## Introduction
The PIC API is used to correctly setup the PIC, before the initialization of the IOAPIC subsystem. It provides a minimal API to manage only the part where the PIC is needed in the boot process. Otherwise, the PIC is just ignored once all legacy IRQs are managed by the IOAPIC subsystem. The PIC API is defined in `shelter/lib/include/cpu/pic.h` and implemented in `shelter/lib/include/cpu/pic.c`.
## API overview
The API provides the following functions:
- `sh_pic_remap()`: remap all legacy IRQs on IDT vectors 32 to 47, should only be called once before any use of the PIC
- `sh_pic_unmask()`: take an IRQ number and unmask this IRQ line
- `sh_pic_mask()`: take an IRQ number and mask this IRQ line
- `sh_pic_send_eoi()`: send End Of Interrupt signal to the PIC

6
docs/shelter/cpu/pit.md Normal file
View File

@@ -0,0 +1,6 @@
# PIT API
In Shelter, the PIT is only used for one purpose: estimating the CPU frequency. It's only used with the PIC, never with the IOAPIC, so the API is volontarily very simple:
- `sh_pit_set_frequency()`: expect the frequency as argument. Set the PIT to uses this frequency and return nothing
This API is defined inside `shelter/lib/include/cpu/pit.h` and implemented inside `shelter/lib/include/cpu/pic.c`.

2
docs/shelter/cpu/serial.md
View File

@@ -2,7 +2,7 @@
## Introduction
Shelter provide an abstraction around the serial port for outputing on it. This should act as the central point for outputing on the serial. The API is defined in `shelter/lib/include/cpu/serial.h` and implemented in `shelter/lib/src/cpu/serial.c`. This API obey the `serial_port_disabled` killswitch defined in kernel boot configuration. The subsystem prefix is `sh_serial_`.
Shelter provides an abstraction around the serial port for outputing on it. This should act as the central point for outputing on the serial. The API is defined in `shelter/lib/include/cpu/serial.h` and implemented in `shelter/lib/src/cpu/serial.c`. This API obey the `serial_port_disabled` killswitch defined in kernel boot configuration. The subsystem prefix is `sh_serial_`.
## Overview

12
docs/shelter/cpu/tsc.md
View File

@@ -2,14 +2,16 @@
## Introduction
In order to be able to measure and approximate time as soon as the kernel start the boot process, the TSC API is implemented in a volontary minimal way. The TSC API is defined inside `shelter/lib/include/cpu/tsc.h` and implemented inside `shelter/lib/src/cpu/tsc.c`. The API prefix is `sh_tsc_`.
In order to be able to measure and approximate time as soon as the kernel start the boot process, the TSC API is implemented in a volontarily minimal way. The TSC API is defined inside `shelter/lib/include/cpu/tsc.h` and implemented inside `shelter/lib/src/cpu/tsc.c`. The API prefix is `sh_tsc_`.
## Overview
The TSC API being intented for measuring time during the boot process, the provided features are extremely basic. In order for any TSC value to start at 0, we define two concept:
One of the TSC API goal being measuring time during the boot process, the provided features regarding this goal are extremely basic. In order for any TSC value to start at 0, we define two concept:
- `kernel_init_tsc`: a TSC value initialized at the very start of the kernel boot process
- `kernel_current_tsc`: a TSC value which is the result of `kernel_init_tsc` substracted to the value returned by `sh_asm_rdtsc()`
Starting with Shelter 0.2, the TSC API also provides primitives to estimate CPU frequency using PIT and TSC, or CPU ID if available.
## API content
The API define the following elements:
@@ -18,3 +20,9 @@ The API define the following elements:
- `sh_tsc_init_tsc()`: a function initializing `kernel_init_tsc`. This function should only be called once as soon as the kernel start. Return a `SH_STATUS`
- `sh_tsc_get_kernel_init_tsc()`: return `kernel_init_tsc` under a `sh_tsc_TSC_VALUE`
- `sh_tsc_get_kernel_current_tsc()`: return the result of `sh_asm_rdtsc()` less `kernel_init_tsc`, under a `sh_tsc_TSC_VALUE`
- `sh_tsc_estimate_cpu_freq()`: estimate CPU frequency using PIT and TSC, should only be called once the IDT has been initialized and PIT remapped on the correct IDT vector, and before the start of all APs. It return the estimated frequency under a `sh_uint64`
- `sh_tsc_load_cpu_freq()`: take a `sh_uint64` and load it as CPU frequency, return nothing
- `sh_tsc_has_hypervisor()`: return `SH_TRUE` if the hypervisor bit in CPU ID is set, take no parameters
- `sh_tsc_is_tsc_constant()`: return `SH_TRUE` if the TSC is constant, according to CPU ID
- `sh_tsc_get_cpu_freq_cpuid()`: return in a variable given as a pointer the value of the CPU frequency given by CPU ID id available. Return a `SH_STATUS`
- `sh_tsc_devs_query()`: TSC DevS entry point, should only be called by `sh_devs_query`

36
docs/shelter/devs/abstracted/ps2.md Normal file
View File

@@ -0,0 +1,36 @@
# PS2 keyboard
The PS2 keyboard driver is defined in `shelter/lib/include/devs/input/ps2.h` and implemented in `shelter/lib/src/devs/input/ps2.c`. This documentation only describes the public API, the one that is safe to access from external subsystems.
## Driver initialization
The PS2 is initialized using `sh_ps2_driver_init()`. The pre-requirement for this function include the keyboard input subsystem being initialized. Here is the exact procedure:
1) Disable interrupts for safety
2) Flush output buffer
3) Debug log status and config byte initial value
4) Disable keyboard and mouse PS2 ports
5) Reflush output buffer for safety
6) Disable both ports IRQs
7) Perform the PS2 controller self test
8) Enable port 1
9) Enable scancodes translation
10) Initialize driver internal state
11) Enable IRQ for port 1
12) Register interrupt handler for legacy IRQ1
13) Enable interrupts
14) Disable scanning
15) Reset leds state
16) Identify keyboard
17) Register PS2 keyboard to the keyboard input subsystem
18) Enable scanning
For the moments, commands responses handling and scancodes parsing is working but there are a few limitations:
- since we only parsed a few ACPI tables, we are unable to detect with certainty the presence of a PS2 controller. So the system consider that the PS2 controller is available and working and the keyboard device registered to the keyboard input subsystem will always be marked as present
- any error in the driver initialization will be interpreted as a failure of the boot process
- for the moment, we are unable to defer the leds state update when a scancode change the NumLock/CapsLock state, so these leds won't update
## API
A few additionnal functions are provided:
- `sh_ps2_disable_scanning()`: disable PS2 keyboard scanning
- `sh_ps2_enable_scanning()`: enable PS2 keyboard scanning

43
docs/shelter/devs/acpi.md Normal file
View File

@@ -0,0 +1,43 @@
# ACPI Parsing
Shelter only support the ACPI v2 and newer ACPI specifications. That mean the XSDT pointer must exists.
## Root structures parsing
The root structures parsing is defined in `shelter/lib/include/devs/acpi.h` and implemented in `shelter/lib/src/devs/acpi.c`.
The `sh_acpi_parse_rsdp()` function takes the RSDP and completes a `sh_apci_ACPI_POINTERS_ROOT` structure that look like this:
``` C
typedef struct {
sh_uint64 rsdt;
sh_uint64 xsdt;
} sh_apci_ACPI_POINTERS_ROOT;
```
Then, the `sh_acpi_parse_xsdt()` function takes the XSDT pointer and parses the XSDT header. It ensures that the structure isn't corrupted, and map all the entries. At the end, it completes this structure:
``` C
typedef struct {
sh_uint64 *entries;
sh_uint64 entry_count;
} sh_acpi_ACPI_XSDT_BODY;
```
Finally, the `sh_acpi_parse_tables()` takes the `sh_acpi_ACPI_XSDT_BODY` structure and parses all the entries. It ensures that at least the first pages of each table is mapped. All the tables pointers are sorted in a structure that look like this:
``` C
typedef struct {
sh_uint64 madt_physical_address;
} sh_acpi_ACPI_TABLES;
```
## Tables parsing
The parsing of ACPI tables is delegated to several functions, one for each tables. Here are the tables that are currently supported:
- MADT
All parsing functions are defined in the files inside `shelter/lib/include/devs/acpi_tables` and in the files inside `shelter/lib/src/devs/acpi_tables`.
### MADT parsing
The MADT is parsed using the `sh_acpi_madt_parse(sh_uint64 madt_ptr)`, defined in `shelter/lib/include/devs/acpi_tables/madt.h` and implemented in `shelter/lib/src/devs/acpi_tables/madt.c`.
This function logs all the entries and register all LAPIC devices, IOAPIC devices and ISOs.

142
docs/shelter/devs/devs.md Normal file
View File

@@ -0,0 +1,142 @@
# DevS querying
## Introduction
The Device System provides a central way to query various kernel subsystems. It is defined in `shelter/lib/include/devs/devs.h` and implemented in `shelter/lib/src/devs/devs.c`.
## Entry point
All DevS related API call goes through a standard entry point:
``` C
sh_devs_query(char *path,sh_devs_RESULT *result)
```
The result is stored inside `result`, which is a structure like this:
``` C
typedef struct {
enum sh_devs_RESULT_TYPE type;
sh_uint64 value;
} sh_devs_RESULT;
```
There are severals returned types:
- `SH_DEVS_VALUE`: a 64 bits value
- `SH_DEVS_BOOL`: a boolean stored inside `value` by casting the `sh_bool` to `sh_uint64`
- `SH_DEVS_LAPIC`: `value` should be casted to `sh_lapic_DEVICE*`, which points directly to a LAPIC device structure
- `SH_DEVS_IOAPIC`: `value` should be casted to `sh_ioapic_DEVICE*`, which points directly to a IOAPIC device structure
Each endpoint has a specific syntax, but here are the common syntax elements:
- each path starts with the `$` sign
- each subsystem has its own prefix
Unless you are querying for a pointer (in this case, you can modify the value inside the pointed structure), you can't write anything in the DevS endpoints.
## Endpoint categories
All the kernel subsystems that expose datas through the DevS provide a function to parse the rest of the path without the subsystem prefix once `sh_devs_query()` dispatched the query.
These functions aren't documented in the documentation of these subsystems and shouldn't be called directly.
### LAPIC endpoints
Function:
``` C
sh_lapic_devs_query(char *sub_path,sh_devs_RESULT *result)
```
Prefix: `$apic/lapic`
Endpoints:
Path with syntax | Returned type | Description
-----------------|---------------|------------
`$apic/lapic/by-apic-id/<id>` | `SH_DEVS_LAPIC` | Return the struct of the LAPIC device correspoding to the provided LAPIC id. **Can return `SH_NULLPTR` with a success due to sparses id**
`$apic/lapic/by-acpi-id/<id>` | `SH_DEVS_LAPIC` | Return the struct of the LAPIC device correspoding to the provided ACPI processor id. **Can return `SH_NULLPTR` with a success due to sparses id**
`$apic/lapic/count` | `SH_DEVS_VALUE` | Return the number of LAPIC devices registered
`$apic/lapic/timer-frequency` | `SH_DEVS_VALUE` | Return the estimated frequency of the LAPIC timer
`$apic/lapic/max-apic-id` | `SH_DEVS_VALUE` | Return the biggest LAPIC id among all registered LAPIC devices
`$apic/lapic/max-acpi-id` | `SH_DEVS_VALUE` | Return the biggest ACPI processor id among all registered LAPIC devices
`$apic/lapic/base-address` | `SH_DEVS_VALUE` | Return the physical address of the memory mapped pages for LAPICs
`$apic/lapic/this-cpu-lapic` | `SH_DEVS_LAPIC` | Return the struct of the LAPIC device corresponding to this CPU
### IOAPIC endpoints
Function:
``` C
sh_ioapic_devs_query(char *sub_path,sh_devs_RESULT *result)
```
Prefix: `$apic/ioapic`
Endpoints:
Path with syntax | Returned type | Description
-----------------|---------------|------------
`$apic/ioapic/by-ioapic-id/<id>` | `SH_DEVS_ioAPIC` | Return the struct of the IOAPIC device correspoding to the provided IOAPIC id. **Can return `SH_NULLPTR` with a success due to sparses id**
`$apic/lapic/by-gsi/<gsi>` | `SH_DEVS_IOAPIC` | Return the struct of the IOAPIC device that manages the corresponding GSI. **Can return `SH_NULLPTR` with a success due to sparses GSI**
`$apic/lapic/count` | `SH_DEVS_VALUE` | Return the number of IOAPIC devices registered
`$apic/lapic/max-ioapic-id` | `SH_DEVS_VALUE` | Return the biggest IOAPIC id among all registered IOAPIC devices
### TSC endpoints
Function:
``` C
sh_tsc_devs_query(char *sub_path,sh_devs_RESULT *result)
```
Prefix: `$tsc`
Endpoints:
Path with syntax | Returned type | Description
-----------------|---------------|------------
`$tsc/kernel-init-tsc` | `SH_DEVS_VALUE` | Return the TSC value saved at the start of the kernel C entry point, before AP bootstrap
`$tsc/cpu-freq` | `SH_DEVS_VALUE` | Return the CPU frequency, either estimated or provided by CPU ID
`$tsc/is-cpu-freq-provided` | `SH_DEVS_BOOL` | Return `SH_TRUE` if CPU frequency is provided by CPU ID, `SH_FALSE` if CPU frequency is estimated
### Kernel config endpoints
Function:
``` C
sh_conf_devs_query(char *sub_path,sh_devs_RESULT *result)
```
Prefix: `$kernel/conf`
Endpoints:
Path with syntax | Returned type | Description
-----------------|---------------|------------
`$kernel/conf/log-level` | `SH_DEVS_VALUE` | Return the log level of the kernel
`$kernel/conf/test-benchmark` | `SH_DEVS_BOOL` | Return the test-benchmark state, `SH_TRUE` if enabled
`$kernel/conf/bench-iterations` | `SH_DEVS_VALUE` | Return the amount of iterations used for benchmarking
`$kernel/conf/disable-serial-port` | `SH_DEVS_BOOL` | Return `SH_TRUE` if serial port is disabled, `SH_FALSE` otherwise
`$kernel/conf/kbd-events-queue-capacity` | `SH_DEVS_VALUE` | Return the amount of events a keyboard events queue can store
### Memory subsystem endpoints
Function:
``` C
sh_memory_devs_query(char *sub_path,sh_devs_RESULT *result)
```
Prefix: `$memory`
Endpoints:
Path with syntax | Returned type | Description
-----------------|---------------|------------
`$memory/free-pages` | `SH_DEVS_VALUE` | Return the amount of free pages in the Pez physical plane
`$memory/used-pages` | `SH_DEVS_VALUE` | Return the amount of allocated pages in the Pez physical plane
`$memory/total-pages` | `SH_DEVS_VALUE` | Return the amount of physical pages detected when parsing the memory map
`$memory/installed-pages` | `SH_DEVS_VALUE` | Return the amount of pages detected usable when parsing the memory map. Can be slightly smaller than total pages
### SMP endpoints
Function: integrated to `sh_devs_query()`
Prefix: `$smp`
Path with syntax | Returned type | Description
-----------------|---------------|------------
`$smp/cpu-count` | `SH_DEVS_VALUE` | Return the number of processors

18
docs/shelter/devs/drivers.md Normal file
View File

@@ -0,0 +1,18 @@
# Device drivers
## Overview
The current device drivers are scattered through the `shelter/lib/include/devs` folder. There is two kinds of devices drivers:
- integrated: these one are used as a core part of the kernel, and aren't exposed through subsystem abstraction
- abstracted: these one are abstracted using a specific subsystem depending on the device type that provides a standard API both for drivers and consumers
## Integrated drivers
For the moment, integrated drivers are implemented for the following devices:
- [LAPIC devices](integrated/lapic.md)
- [IOAPIC devices](integrated/ioapic.md)
## Abstracted drivers
For the moment, abstracted drivers are implemented for the following devices:
- [PS2 keyboard](abstracted/ps2.md)

13
docs/shelter/devs/index.md Normal file
View File

@@ -0,0 +1,13 @@
# Device System
## Introduction
The Device System (also called DevS) is the entire subsystem managing everything related to ACPI parsing and devices abstractions.
## Overview
The DevS is splitted in several subparts:
- [ACPI parsing](acpi.md)
- [Device drivers](drivers.md)
- [Keyboard input subsystem](kbdinput.md)
- [DevS querying](devs.md)

35
docs/shelter/devs/integrated/ioapic.md Normal file
View File

@@ -0,0 +1,35 @@
# IOAPIC devices
This driver is defined inside `shelter/lib/include/devs/apic/ioapic.h` and implemented inside `shelter/lib/src/devs/apic/ioapic.c`.
## Driver initialization
During the MADT parsing, all IOAPIC entries are counted. Then, a call to `sh_ioapic_init_devs(sh_uint8 max_lapic_id)` is made to initialize the driver internal state. This call allocates an array indexed by IOAPIC id.
## Device initialization
After that, each time a IOAPIC entry is parsed, a call to `sh_ioapic_init(sh_uint32 base,sh_uint8 ioapic_id,sh_uint32 gsi_base,sh_ioapic_DEVICE *ioapic)` is made. This does three things:
1) Identity map the memory space for the LAPIC
2) Fill the structure for the LAPIC device
3) Initialize the device without modifying the IOREDTBL for the moment
Then, a call to `sh_ioapic_bind(sh_ioapic_DEVICE *ioapic_dev)` is made, registering the LAPIC structure in the array.
## Basic IOAPIC management
For basic IOAPIC management, the following function are provided:
- `sh_ioapic_get_dev_ioapic_id(sh_uint64 ioapic_id)`: return a `sh_ioapic_DEVICE*` pointing to the IOAPIC structure corresponding to the provided IOAPIC id
- `sh_ioapic_get_dev_by_gsi(sh_uint32 gsi)`: return a `sh_ioapic_DEVICE*` pointing to the IOAPIC structure of the IOAPIC that manages the provided GSI
- `sh_ioapic_read_ioredtbl_entry(sh_ioapic_DEVICE *ioapic_dev,sh_uint32 gsi,sh_ioapic_IOREDTBL_ENTRY *entry)`: read a IOREDTBL entry and put its value into `entry`. If the GSI isn't managed by the provided device, raise an error
- `sh_ioapic_write_ioredtbl_entry(sh_ioapic_DEVICE *ioapic_dev,sh_uint32 gsi,sh_ioapic_IOREDTBL_ENTRY *entry)`: write and IOREDTBL entry corresponding to the provided GSI. If the GSI isn't managed by the provided device, raise an error
IOREDTBL read and write are thread-safe operations.
## Higher level operations
The following higher level operations are provided:
- `sh_ioapic_mask_gsi(sh_ioapic_DEVICE *dev,sh_uint32 gsi)`: mask the provided GSI. If the GSI isn't managed by the provided device, raise an error
- `sh_ioapic_unmask_gsi(sh_ioapic_DEVICE *dev,sh_uint32 gsi)`: unmask the provided GSI. If the GSI isn't managed by the provided device, raise an error
- `sh_ioapic_mask_all()`: mask all the GSI for all registered IOAPIC.
If you are looking for an even more abstracted API, please check the GSI subsystem.

81
docs/shelter/devs/integrated/lapic.md Normal file
View File

@@ -0,0 +1,81 @@
# LAPIC devices
This driver is defined inside `shelter/lib/include/devs/apic/lapic.h` and implemented inside `shelter/lib/src/devs/apic/lapic.c`.
## Driver initialization
During the MADT parsing, all LAPIC entries are counted. Then, a call to `sh_lapic_init_devs(sh_uint16 max_lapic_id,sh_uint16 max_acpi_processor_id)` is made to initialize the driver internal state. This call allocates two arrays: one indexed by LAPIC id and one indexed by ACPI processors id.
## Device initialization
After that, each time a LAPIC entry is parsed, a call to `sh_lapic_init(sh_uint64 lapic_phys,sh_uint8 spurious_vector,sh_uint8 apic_id,sh_uint8 apic_processor_id,sh_uint32 flags,sh_lapic_DEVICE *lapic)` is made. This does two things:
1) Identity map the memory space for the LAPIC
2) Fill the structure for the LAPIC device
Then, a call to `sh_lapic_bind(sh_lapic_DEVICE *lapic_dev)` is made, registering the LAPIC structure in both arrays.
Finally, when the MADT parsing is over, a call to `sh_lapic_init_dev(sh_uint8 spurious_vector,sh_lapic_DEVICE *lapic)` initialize the LAPIC by setuping a few of its registers.
## Basic LAPIC management
For basic LAPIC management, the following functions are provided:
- `sh_lapic_eoi(sh_lapic_DEVICE *lapic)`: send a End Of Interrupt (EOI) to the LAPIC
- `sh_lapic_get_dev_apic_id(sh_uint64 apic_id)`: return a `sh_lapic_DEVICE*` pointing to the LAPIC structure corresponding to the provided APIC id
- `sh_lapic_get_dev_acpi_cpu_id(sh_uint64 acpi_processor_id)`: return a `sh_lapic_DEVICE*` pointing to the LAPIC structure corresponding to the provided ACPI processor id
- `sh_lapic_get_by_apic_id_array()`: return a `sh_lapic_DEVICE**` of the LAPIC devices pointers array, indexed by LAPIC id
- `sh_lapic_get_max_apic_id()`: return the max LAPIC id
- `sh_lapic_get_max_acpi_processor_id()`: return the max ACPI processor id
- `sh_lapic_get_lapic_count()`: return the number of registered LAPICs
- `sh_lapic_get_current_core()`: return the LAPIC id of the current core, return -1 if error happened
## LAPIC timer
To launch a one shot timer without knowing the frequency, the `sh_lapic_timer_one_shot(sh_lapic_DEVICE *lapic_dev,sh_uint32 initial_value)` can be used.
But the LAPIC driver provides a way to calibrate the LAPIC in order to obtain its frequency. For this, we use the following process:
1) Launch a timer with max value
2) Read the start TSC and current LAPIC value, iterate for 10 millions times, read the end TSC and current LAPIC value.
3) Compute `tsc_delta` and `lapic_delta`
4) Using the following formula, estimate the LAPIC frequency: `lapic_freq = (lapic_delta * cpu_freq) / tsc_delta`
This calibration process is implemented in `sh_lapic_calibrate(sh_lapic_DEVICE *lapic_dev,sh_uint64 cpu_freq)`. But the TSC frequency estimation must have been achevied before that.
When the LAPIC frequency is obtained, two new functions are unlocked:
- `sh_lapic_get_frequency()`: return the estimated LAPIC frequency
- `sh_lapic_timer_one_shot_us(sh_lapic_DEVICE *lapic_dev,sh_uint64 microseconds_count)`: launch a one shot timer by taking a value in microsecond. Compute the initial value using the estimated frequency before hand.
## IPI management
The LAPIC driver provides a helper to know if the LAPIC IPI feature is busy: `sh_lapic_ipi_is_busy(sh_lapic_DEVICE *lapic_dev)`.
### Sending a fixed IPI
The primitive to send a fixed IPI is:
``` C
sh_lapic_send_fixed_ipi(sh_lapic_DEVICE *lapic_dev,sh_uint8 vector,sh_uint32 destination_mode,sh_int16 target_lapic_id);
```
`lapic_dev` must be the struct of the LAPIC of the current CPU.
`destination_mode` can either be `SH_LAPIC_IPI_DESTINATION_SPECIFIC`, `SH_LAPIC_IPI_DESTINATION_SELF`, `SH_LAPIC_IPI_DESTINATION_ALL` or `SH_LAPIC_IPI_DESTINATION_ALL_EXCLUDING_SELF`.
If `destination_mode` isn't `SH_LAPIC_IPI_DESTINATION_SPECIFIC`, `target_lapic_id` must be `SH_LAPIC_IPI_NO_DESTINATION`.
### Sending any others IPI
The primitive to send any other IPI than a fixed IPI is:
``` C
sh_lapic_send_ipi(sh_lapic_DEVICE *lapic_dev,sh_uint32 ipi_type,sh_uint32 destination_mode,sh_int16 target_lapic_id,sh_page_PHYSICAL_ADDRESS start_address);
```
`lapic_dev` must be the struct of the LAPIC of the current CPU.
`ipi_type` can either be `SH_LAPIC_IPI_TYPE_INIT`, `SH_LAPIC_IPI_TYPE_STARTUP` or `SH_LAPIC_IPI_TYPE_NMI`.
If `ipi_type` is `SH_LAPIC_IPI_TYPE_STARTUP`, `start_address` must contain the starting address. Address constraint (compatible with 16 bits mode) are checked.
If `ipi_type` isn't `SH_LAPIC_IPI_TYPE_STARTUP`, `start_address` must be 0.
`destination_mode` can either be `SH_LAPIC_IPI_DESTINATION_SPECIFIC`, `SH_LAPIC_IPI_DESTINATION_SELF`, `SH_LAPIC_IPI_DESTINATION_ALL` or `SH_LAPIC_IPI_DESTINATION_ALL_EXCLUDING_SELF`.
If `destination_mode` isn't `SH_LAPIC_IPI_DESTINATION_SPECIFIC`, `target_lapic_id` must be `SH_LAPIC_IPI_NO_DESTINATION`.

134
docs/shelter/devs/kbdinput.md Normal file
View File

@@ -0,0 +1,134 @@
# Keyboard input subsystem
## Introduction
The keyboard input subsystem is responsible for abstracting keyboard devices and keyboards events. It's defined in `shelter/lib/include/devs/input/kbd.h` and implemented in `shelter/lib/src/devs/input/kbd.c`. It's still in the early stage and isn't finished for the moment.
## Overview
The keyboard input subsystem is based on two key concepts:
- keyboard devices: this is the standard abstraction for any keyboard-like devices
- keyboard events: provides a standard way to interpret any events from any keyboard devices
The keyboard input subsystem is initialized using `sh_kbd_init_devs()`.
### Keyboard devices
### Driver side
The keyboard device structure look like this:
``` C
typedef struct {
sh_uint8 kbd_id;
sh_queue_KBD_EVENT *events_queue;
sh_bool present;
sh_bool ver_maj;
sh_bool ver_num;
sh_bool shift;
sh_bool alt;
sh_bool ctrl;
sh_bool win;
sh_SPIN_LOCK spinlock;
} sh_kbd_DEVICE;
```
Each keyboard device is identified with a keyboard id. When a keyboard is unregistered, the keyboard id can be recycled for a new keyboard device. There is a maximum of 256 keyboards id, and so 256 maximum registered keyboard devices at the same time.
The keyboard id 0 is reserved for the PS2 keyboard. It's always valid, even if there is no PS2 controller.
The `present` boolean defines if the keyboard is present. If `present` is `SH_FALSE`, no actions except unregistering the keyboard is allowed on the keyboard device.
The `present` boolean for the keyboard od keyboard id 0 indicates if the PS2 controller is available.
There is two primitives to register keyboards devices:
- `sh_kbd_register_ps2(sh_bool is_present,sh_kbd_DRIVER_HANDLE **handle)`: register the PS2 keyboard, reserved for the PS2 driver
- `sh_kbd_register(sh_bool is_present,sh_kbd_DRIVER_HANDLE **handle)`: register a new keyboard device
Upon registration, a `sh_kbd_DRIVER_HANDLE` struct is completed. It became non valid when the keyboard is unregistered. It is reserved for driver usage only.
Two primitives accessible to drivers require this driver handle:
- `sh_kbd_set_flag(sh_kbd_DRIVER_HANDLE *handle,sh_uint8 flag,sh_bool value)`: set a flag about the keyboard corresponding to the provided driver handle
- `sh_kbd_push_events(sh_kbd_DRIVER_HANDLE *handle,sh_kbd_EVENT *event)`: push an event in the events queue of the keyboard device corresponding to the provided driver handle
There is, for the moment, no primitive to unregister a keyboard.
### Consumer side
Four functions are available for consumers:
- `sh_kbd_enumerate_devices(sh_kbd_ENUMERATION *enumeration)`: complete a `sh_kbd_ENUMERATION` structure, each bit set to 1 correspond to a valid keyboard id
- `sh_kbd_get_handle(sh_uint8 kbd_id,sh_kbd_CONSUMER_HANDLE **handle)`: return a valid consumer handle for the provided keyboard id. It uses `sh_malloc`
- `sh_kbd_destroy_handle(sh_kbd_CONSUMER_HANDLE **handle)`: free the provided consumer handle
- `sh_kbd_read_flag(sh_kbd_DRIVER_HANDLE *d_handle,sh_kbd_CONSUMER_HANDLE *c_handle,sh_uint8 flag,sh_bool *value)`: read a flag about the keyboard corresponding to the provided handle. Accepts either a driver or consumer handle, but fails if both handle are provided.
The possible flags to read/write are: `SH_KBD_FLAG_PRESENT`, `SH_KBD_FLAG_VER_MAJ`, `SH_KBD_FLAG_VER_NUM`, `SH_KBD_FLAG_SHIFT`, `SH_KBD_FLAG_ALT`, `SH_KBD_FLAG_CTRL`, `SH_KBD_FLAG_WIN`.
There is, for the moment, no way to read keyboards events.
## Keyboard events
A keyboard event is represented by this structure:
``` C
typedef struct {
sh_uint16 scancode;
sh_bool pressed;
sh_uint8 context_shift_win;
sh_uint8 context_alt;
sh_uint8 context_ctrl;
sh_uint8 context_lock_status;
sh_uint8 event_type;
sh_uint64 timestamp;
} sh_kbd_EVENT;
```
All non-pause events contain a snapshot of keyboard context state at emission time, before the state of corresponding flag in the keyboard device changes. These fields remain valid even for `SH_EVENT_TYPE_KBD_CONTEXT`:
- `context_shift_win`:
- Bit 0: shift active, checked with macro `SH_KBD_EVENT_SHIFT_ACTIVE`
- Bit 1: win active, checked with macro `SH_KBD_EVENT_WIN_ACTIVE`
- `context_alt`:
- Bit 0: right alt active, checked with macro `SH_KBD_EVENT_RIGHT_ACTIVE`
- Bit 1: left alt active, checked with macro `SH_KBD_EVENT_LEFT_ACTIVE`
- `context_ctrl`:
- Bit 0: right control active, checked with macro `SH_KBD_EVENT_RIGHT_ACTIVE`
- Bit 1: left control active, checked with macro `SH_KBD_EVENT_LEFT_ACTIVE`
- `context_lock_status`:
- Bit 0: numlock active, checked with `SH_KBD_EVENT_NUMLOCK_ACTIVE`
- Bit 0: capslock active, checked with `SH_KBD_EVENT_CAPSLOCK_ACTIVE`
There is a precise way of interpreting what this structure mean. It start by checking the value of `event_type`.
### `SH_EVENT_TYPE_KBD_PAUSE`
The event is a press of the pause key. The `scancode` and `pressed` fields should be ignored.
### `SH_EVENT_TYPE_KBD_CONTEXT`
This event is a press of a modifier key. It allows to abstract modifiers keys events accross all types of keyboards.
The scancode must be one of the following:
- `SH_KBD_CONTEXT_LSHIFT`
- `SH_KBD_CONTEXT_RSHIFT`
- `SH_KBD_CONTEXT_LCTRL`
- `SH_KBD_CONTEXT_RCTRL`
- `SH_KBD_CONTEXT_LALT`
- `SH_KBD_CONTEXT_RALT`
- `SH_KBD_CONTEXT_WIN`
- `SH_KBD_CONTEXT_NUMLOCK`
- `SH_KBD_CONTEXT_CAPSLOCK`
You need to check the `pressed` boolean to know if the key was pressed or released.
### `SH_EVENT_TYPE_PS2_NORMAL`
This event type represents a standard PS2 Set1 scancode. The scancode is put in the low 8 bits of the `scancode` fields. The high 8 bits are set to 0.
You need to check the `pressed` boolean to know if the key was pressed or released.
### `SH_EVENT_TYPE_PS2_E0`
This event type represents an extended PS2 Set1 scancode prefixed by `0xE0`. The scancode is put in the low 8 bits of the `scancode` fields. The high 8 bits are set to `0xE0`.
You need to check the `pressed` boolean to know if the key was pressed or released.
### Notes
For `SH_EVENT_TYPE_PS2_NORMAL` and `SH_EVENT_TYPE_PS2_E0` events type, the scancode, when `pressed` is `SH_FALSE`, keeps its bit 8 to 1.

6
docs/shelter/index.md
View File

@@ -10,12 +10,14 @@ Shelter stricly follow these principles:
- absolutely no external code inside it. All the code that run in ring 0 should be sovereign
- absolutely no UNIX dependency on design philosophy. Shelter follow his own rules for kernel design (but doesn't forbid itself to reuse somes UNIX concepts), even if it mean spending years designing, refining and standardizing them
- a very strict boot process that ensure that everything goes well during the system boot
- fully made in C
- fully made in C (and a bit of ASM only where necessary)
- focuses on auditability, self-testing and benchmarking, and determinism
- use his own algorithms for various very important components of every kernel (mainly memory management, scheduling, etc), except on cryptographic algorithms, which follow a very strict integration processus
The Shelter kernel is currently in very early developpement stage and isn't suitable at all for every-day usage.
x2APIC isn't supported yet, so all of the code related to CPU manipulation is optimized for maximum 256 CPUs.
## Summary
- Key principles to know in order to boot Shelter
@@ -32,4 +34,6 @@ The Shelter kernel is currently in very early developpement stage and isn't suit
- [Kernel-specific APIs](kernel/kernel.md)
- [Memory subsystem](memory/index.md)
- [Test-and-benchmark framework](tab.md)
- [Devices System](devs/index.md)
- [IRQ subsystem](irq/index.md)
- [Benchmark results](bench.md)

98
docs/shelter/irq/gdt.md Normal file
View File

@@ -0,0 +1,98 @@
# GDT
The GDT management is defined in `shelter/lib/include/irq/gdt.h` and implemented in `shelter/lib/include/irq/gdt.c`.
## 64 bits GDT
The 64 bits GDT can contains two types of entries:
- `sh_gdt_GDT_ENTRY_64`: standard GDT entry
- `sh_gdt_GDT_ENTRY_128`: special 128 bits GDT entry for TSS
The standard 64 bits GDT used during the boot process is contained in this structure:
``` C
#pragma pack(1)
typedef struct {
sh_gdt_GDT_ENTRY_64 null;
sh_gdt_GDT_ENTRY_64 kernel_code;
sh_gdt_GDT_ENTRY_64 kernel_data;
sh_gdt_GDT_ENTRY_64 user_code;
sh_gdt_GDT_ENTRY_64 user_data;
sh_gdt_GDT_ENTRY_128 tss;
} sh_gdt_GDT;
#pragma pack()
```
We also use this struct for generating the 10 bytes to use with `lgdt`:
``` C
#pragma pack(1)
typedef struct {
sh_uint16 limit;
sh_uint64 base;
} sh_gdt_GDTR;
#pragma pack()
```
Regarding the API, we provide the following functions:
- `sh_gdt_fill_access_byte(sh_bool present,sh_uint8 dpl,sh_bool descriptor_type,sh_uint8 segment_type)`: generate access byte
- `sh_gdt_fill_flags_byte(sh_bool g,sh_bool db,sh_bool l,sh_bool avl)`: generate flags byte
- `sh_gdt_fill_gdt_entry_64(sh_uint32 limit,sh_uint32 base,sh_uint8 access,sh_uint8 flags)`: fill a 64 bits GDT entry, expect flags as `sh_gdt_fill_flags_byte` would return it
- `sh_gdt_fill_gdt_entry_128(sh_uint32 limit,sh_uint64 base,sh_uint8 access,sh_uint8 flags)`: fill a 128 bits GDT entry, expect flags as `sh_gdt_fill_flags_byte` would return it
- `sh_gdt_fill_gdt(sh_tss_TSS *tss,sh_gdt_GDT *gdt)`: fill the standard 64 bits GDT with the provided TSS
- `sh_gdt_load_gdtr(sh_gdt_GDT *gdt)`: load the standard 64 bits GDT using `lgdt`
- `sh_gdt_reload_registers()`: an ASM stub to reload segment registers
All these functions are intended to be used with the standard 64 bits GDT defined with `sh_gdt_GDT`, which should only be used on the bootstrap CPU
## APs GDT
The APs use a special GDT containing sufficient spaces for the TSS entries of all APs.
For that we use a special shared 64 bits GDT between all APs, defined in this struct:
``` C
#pragma pack(1)
typedef struct {
sh_gdt_GDT_ENTRY_64 null;
sh_gdt_GDT_ENTRY_64 kernel_code;
sh_gdt_GDT_ENTRY_64 kernel_data;
sh_gdt_GDT_ENTRY_64 user_code;
sh_gdt_GDT_ENTRY_64 user_data;
sh_gdt_GDT_ENTRY_128 tss[256];
} sh_gdt_GDT_AP;
#pragma pack()
```
The APs GDT is generated using the function `sh_gdt_fill_gdt_ap(sh_tss_TSS *tss,sh_uint64 tss_count,sh_gdt_GDT_AP *gdt)`. This allows to fill a GDT with up to 256 TSS entries. The GDTR for this shared GDT is generated using `sh_gdt_make_gdtr_ap(sh_gdt_GDT_AP *gdt)`.
## 32 bits GDT
In order for the APs to exit 16 bits mode, they need a 32 bits GDT. Here is the structure for a 32 bits GDT entry:
``` C
#pragma pack(1)
typedef struct {
sh_uint16 limit_low;
sh_uint16 base_low;
sh_uint8 base_middle;
sh_uint8 access;
sh_uint8 granularity;
sh_uint8 base_high;
} sh_gdt_GDT_ENTRY_32;
#pragma pack()
```
The 32 bits GDT used to exit 16 bits mode is structured like this:
``` C
#pragma pack(1)
typedef struct {
sh_gdt_GDT_ENTRY_32 null;
sh_gdt_GDT_ENTRY_32 code_64;
sh_gdt_GDT_ENTRY_32 code;
sh_gdt_GDT_ENTRY_32 data;
} sh_gdt_GDT_32;
#pragma pack()
```
The 32 bits GDT implementation provides the following functions:
- `sh_gdt_fill_access_byte_32(sh_bool present,sh_uint8 dpl,sh_bool descriptor_type,sh_uint8 segment_type)`: generate access byte
- `sh_gdt_fill_granularity_byte_32(sh_bool granularity,sh_bool db,sh_bool l,sh_bool avl)`: generate granularity byte
- `sh_gdt_fill_gdt_entry_32(sh_uint32 limit,sh_uint32 base,sh_uint8 access,sh_uint8 granularity)`: fill a GDT entry for a 32 bits GDT, expect flags as `sh_gdt_fill_flags_byte_32` would return it
- `sh_gdt_fill_gdt_32(sh_gdt_GDT_32 *gdt_32)`: fill the 32 bits GDT structure

68
docs/shelter/irq/gsi.md Normal file
View File

@@ -0,0 +1,68 @@
# GSI subsystem
## Introduction
This subsystem is responsible for registering all the interrupts sources overrides (ISO) and providing an abstraction to manipulate all the GSI lines and legacy IRQs.
## ISOs registering
The subsystem can be initialized using `sh_gsi_iso_array_init()`. This allocate an array of 256 `sh_gsi_ISO` in which all the ISOs will be stored.
The ISO struct is the following:
``` C
typedef struct {
sh_uint32 gsi;
sh_bool valid;
sh_uint8 source_irq;
enum sh_gsi_POLARITY polarity;
enum sh_gsi_TRIGGER_MODE trigger_mode;
sh_uint8 bus;
} sh_gsi_ISO;
```
Any `sh_gsi_ISO` without the valid boolean to `SH_TRUE` should be considered as not occupied in the array.
The `sh_gsi_POLARITY` enum can take the following values:
``` C
enum sh_gsi_POLARITY {
SH_GSI_POLARITY_NOT_SET=0,
SH_GSI_POLARITY_BUS_DEFAULT,
SH_GSI_POLARITY_ACTIVE_HIGH,
SH_GSI_POLARITY_ACTIVE_LOW
};
```
The `sh_gsi_TRIGGER_MODE` enum can take the following values:
``` C
enum sh_gsi_TRIGGER_MODE {
SH_GSI_TRIGGER_MODE_NOT_SET=0,
SH_GSI_TRIGGER_MODE_BUS_DEFAULT,
SH_GSI_TRIGGER_MODE_EDGE,
SH_GSI_TRIGGER_MODE_LEVEL
};
```
Bus defaults are resolved when configuring the IOREDTBL of each IOAPIC.
During the MADT table parsing, ISOs are registered using the `sh_gsi_iso_register(sh_uint8 bus,sh_uint8 source,sh_uint32 gsi,sh_uint16 flags)` function, which expects the arguments as they are parsed in the MADT table.
Two functions can be used to search for ISOs:
- `sh_gsi_get_iso_by_irq(sh_uint8 irq)`: return a `*sh_gsi_ISO` with the provided legacy IRQ number. Return `SH_NULLPTR` if not found or error
- `sh_gsi_get_iso_by_gsi(sh_uint32 gsi)`: return a `*sh_gsi_ISO` with the provided GSI number. Return `SH_NULLPTR` if not found or error
## IOAPIC abstraction
The GSI subsystem provides an abstraction to manipulate IOREDTBL entries based on the GSI or legacy IRQ number without interacting directly with the IOAPIC subsystem. The following functions are provided:
- `sh_gsi_get(sh_uint32 gsi,sh_ioapic_IOREDTBL_ENTRY *entry)`: return the IOREDTBL entry corresponding to the provided GSI
- `sh_gsi_set(sh_uint32 gsi,sh_ioapic_IOREDTBL_ENTRY *entry)`: set the IOREDTBL entry corresponding to the provided GSI with the provided entry. Automatically mask the IOREDTBL entry before writing the entry
- `sh_gsi_irq_set(sh_uint8 irq,sh_ioapic_IOREDTBL_ENTRY *entry)`: do the same as `sh_gsi_set()` but retrieves the original GSI before hand, taking into account registered ISOs
- `sh_gsi_mask(sh_uint32 gsi)`: mask the IOREDTBL entry corresponding to the provided GSI
- `sh_gsi_unmask(sh_uint32 gsi)`: unmask the IOREDTBL entry corresponding to the provided GSI
- `sh_gsi_irq_mask(sh_uint8 irq)`: do the same as `sh_gsi_mask()` but retrieves the original GSI before hand, taking into account registered ISOs
- `sh_gsi_irq_unmask(sh_uint8 irq)`: do the same as `sh_gsi_unmask()` but retrieves the original GSI before hand, taking into account registered ISOs
## Legacy IRQs switch
What we call the legacy IRQs switch is the mapping of all the legacy IRQs to interrupt vectors 32 to 47, exactly like the PIT, and the transfer of legacy IRQs management from the PIT to the IOAPIC.
It's operated by `sh_gsi_irq_switch()`. This is the only function that is allowed to call `sh_irq_switch_irq_management()`. By default, all the IOREDTBL entries corresponding to the GSI lines used by all legacy IRQs will be mask.

49
docs/shelter/irq/idt.md Normal file
View File

@@ -0,0 +1,49 @@
# IDT generation and loading
The IDT management is defined inside `shelter/lib/include/irq/idt.h` and implemented inside `shelter/lib/src/irq/idt.c`. The current IDT generator generate the IDT according to boot process needs, and is therefore not definitive.
## IDT generation
The IDT entry is structured with this structure:
``` C
#pragma pack(1)
typedef struct {
sh_uint16 offset_low;
sh_uint16 selector;
sh_uint8 ist;
sh_uint8 type_attr;
sh_uint16 offset_mid;
sh_uint32 offset_high;
sh_uint32 reserved;
} sh_idt_IDT_ENTRY;
#pragma pack()
```
The full IDT is stored in this structure:
``` C
#pragma pack(1)
typedef struct {
sh_idt_IDT_ENTRY entries[256];
} sh_idt_IDT;
#pragma pack()
```
IDT entries are generated using the following functions:
- `sh_idt_fill_type_attr_byte(sh_bool present,sh_uint8 dpl,sh_uint8 type)`: generate the `type_attr` byte
- `sh_idt_fill_idt_entry(sh_uint64 offset,sh_uint16 selector,sh_uint8 ist,sh_uint8 type_attr)`: generate an IDT entry, expect the `type_attr` argument as `sh_idt_fill_type_attr_byte()` would generate it
The IDT is generated by `sh_idt_fill_idt(sh_idt_IDT *idt)`.
Using the following structure:
``` C
#pragma pack(1)
typedef struct {
sh_uint16 limit;
sh_uint64 base;
} sh_idt_IDTR;
#pragma pack()
```
`sh_idt_load_idtr(sh_idt_IDT *idt)` load the IDT.
The following vectors are filled for the moment: 0 to 21, 28 to 30, 32 to 47, 254 and 255. Please see the [IRQ docs](irq.md) to see how the handlers work.

19
docs/shelter/irq/index.md Normal file
View File

@@ -0,0 +1,19 @@
# IRQ subsystem
## Introduction
The IRQ subsystem contains all things related to interrupts management.
## Overview
The IRQ subsystem manages the following elements:
- [TSS structure generation and loading](tss.md)
- [GDT generation for bootstrap and applications processors, and GDT loading](gdt.md)
- [IDT generation and loading](idt.md)
- [IRQ handlers](irq.md)
- [ISO registering and GSI abstractions](gsi.md)
Please note the following:
- this subsystem is only targeting the x86-64 architecture
- this subsystem should only be used after the full memory subsystem initialization
- all the assets generated by this subsystem is, for the moment, only useful for the task performed during the boot process

126
docs/shelter/irq/irq.md Normal file
View File

@@ -0,0 +1,126 @@
# IRQ handlers
The current IRQ handlers are defined in `shelter/lib/include/irq/irq.h` and implemented in `shelter/lib/src/irq/irq.c`. The current handlers are tailored to fit the needs of the boot process and are therefore not complete at all.
## Overview
All IRQ handling is done through the macros defined inside `shelter/lib/src/irq/irq_handler.asm`.
The current IRQ handlers are currently capable of managing three categories:
- CPU faults
- Legacy IRQs
- LAPIC interrupts
The main dispatching is done inside `sh_irq_dispatch(sh_irq_INTERRUPT_FRAME *frame)`. The `frame` argument is built by pushing registers on the stack in the ASM stub and look like this:
``` C
typedef struct {
sh_uint64 r11,r10,r9,r8,rdx,rcx,rax;
sh_uint64 vector;
sh_uint64 error_code;
sh_uint64 rip;
sh_uint64 cs;
sh_uint64 rflags;
} sh_irq_INTERRUPT_FRAME;
#pragma pack()
```
This frame will probably be modified/expanded in futures updates.
When we say "block", we mean a simple infinite loop.
## CPU faults
The following table provide the informations related to CPU faults handling:
Name | Behaviour
-----|----------
Division by zero | Log and block
Debug | Log and block
NMI | Log and block
Breakpoints | Log and continue, usage of breakpoints isn't supported but it won't block the kernel
Overflow | Log and block
Bound range | Log and block
Device not available | Log and block
Double fault | Log and block
Co-processor segment overrun | Log and block
Invalid TSS | Log and block
Segment not present | Log and block
Stack segment fault | Log and block
General protection fault | Log and block
Page fault | Log and block
Alignement check | Log and block
Machine check | Log and block
SIMD floating point exception | Log and block
Virtualization exception | Log and block
Control protection exception | Log and block
Hypervisor exception exception | Log and block
VMM communication exception | Log and block
Security exception | Log and block
CPU faults are the only exception where the frame will be presented.
## Legacy IRQs
In Shelter, legacy IRQs are remapped on vectors 32 to 47. Depending on vector, two behavious are possibles:
- If vector is 32, the IRQ come from the PIT, starting a specific procedure depending on the context
- If vector is between 33 and 47, the IRQ is treated by registered IRQ handlers
Legacy IRQs have two main state:
- Managed by PIC: in this case, they are just logged and then ignored, except for the PIT IRQ0 depending on the context
- Managed by IOAPIC: in this case, PIT IRQ0 are logged and then ignored, and others IRQ are delegated to drivers handlers
The switch between PIC and IOAPIC is effectuated by `sh_irq_switch_irq_management()`. Only after this switch is correctly done, drivers handles can be registered using `sh_irq_legacy_register_handler(sh_uint8 legacy_irq,sh_irq_HANDLER_PTR handler)`. Handlers can't be registered for IRQ 0. Once a handler is set, it can't be removed or updated. The `sh_irq_legacy_register_handler()` function automatically unmask the GSI corresponding to the legacy IRQ.
The type `sh_irq_HANDLER_PTR` is defined like this:
``` C
typedef void (*sh_irq_HANDLER_PTR)();
```
No arguments are provided.
### PIT IRQ
The PIT is mainly used to estimate the CPU frequency. For this, each PIT IRQ is handled differently depending on the context.
The `sh_irq_start_tsc()` function signal to the IRQ handler that the CPU frequency estimation procedure has been triggered, setting the internal state of the procedure to 0, and reseting the TSC values (`tsc_start`, `tsc_end` and `tsc_delta`).
When a PIT IRQ arrives:
- If the legacy IRQs are managed by IOAPIC, simply log and continue
- If not, do the following:
- If the state of the estimation procedure is 0, set `tsc_start` to the current TSC register value and set the state of the procedure to 1
- Else if the state of the estimation procedure is 1, set `tsc_end` to the current TSC register value, compute `tsc_delta = tsc_end - tsc_start` and set the state of the procedure to 2
- Else, log and continue
The `sh_irq_get_tsc_delta()` return `tsc_delta`. A value of 0 indicating that the procedure isn't finished yet.
### Others legacy IRQs
Others legacy IRQs (vectors 33 to 47) are handled through a common loader that work like this:
- If the legacy IRQs are managed by IOAPIC:
- If a handler pointer is defined for this legacy IRQ, call it and continue once finished
- Else, just continue without logging
- Else, log and block
## LAPIC interrupts
LAPIC interrupts can be of two sorts:
- one shot timer: mapped on vector 254
- spurious vector: mapped on vector 255
### One shot timer
The purpose of this handler is to know when the one shot timer generated an interrupt.
Each CPU maintain in its per-CPU struct a boolean named `timer_state.`
The `sh_irq_start_timer()` function is executed by the function to arm the `timer_state` flag: it set this flag to `SH_TRUE`.
When an one shot timer interrupt arrives:
- If `timer_state` was `SH_FALSE`, log the interrupt for debug purposes, it doesn't block anything
- If `timer_state` was `SH_TRUE`, set it to `SH_FALSE` and then resume
The `sh_irq_get_timer_state()` function is a wrapper that return the local `timer_state` of any CPU.
### Spurious vector
In case of interrupts on the spurious vector, we just log it for debug purposes and continue.

11
docs/shelter/irq/tss.md Normal file
View File

@@ -0,0 +1,11 @@
# TSS structure generation and loading
The TSS management is defined in `shelter/lib/include/irq/tss.h` and implemented in `shelter/lib/include/irq/tss.c`.
This implementation provides the following types:
- `sh_tss_TSS`: a full structure of a 64 bits TSS
- `sh_tss_TR`: a structure used to load the TSS
The following functions are provided:
- `sh_tss_fill_tss(sh_tss_TSS *tss)`: fill a TSS structure with 4 stacks dedicated for double faults, NMI, page faults and stack-related faults. Heap must be available for using this function. All stacks are 4 pages long
- `sh_tss_load_tr(sh_tss_TSS* tss)`: load the provided TSS into the task register. Always assume that the TSS selector in the GDT is `0x28`

18
docs/shelter/kernel/efifb.md Normal file
View File

@@ -0,0 +1,18 @@
# EFI framebuffer
## Introduction
This subsystem aims to provide a simple way to indicate the boot process state using a progress bar displayed on the EFI GOP framebuffer. For the moment, only framebuffers using 32 bits pixel format are supported. It's defined in `shelter/lib/include/kernel/efifb.h` and implemented in `shelter/lib/src/kernel/efifb.c`.
## Overview
To avoid any complex colors encoding, pixel values for the colors white and gray must be provided through the framebuffer configuration.
The framebuffer parameters are loaded using the `sh_efifb_init_fb(sh_conf_FB_CONFIG *fb_conf,sh_page_PAGE_TABLE_POOL *ptp)` function. This function will:
1) Load framebuffers dimensions
2) Identity map the whole framebuffers
3) Erase the previous texts on the framebuffer, based on the information provided in the framebuffer configuration
The progress bar is initialized using `sh_efifb_init_bar()`. It will draw an empty progress bar.
Finally, to update the progress bar, the `sh_efifb_set_bar(sh_uint8 percent)` function is used.

1
docs/shelter/kernel/kernel.md
View File

@@ -9,3 +9,4 @@ The Shelter kernel define his own specific APIs like logging and configuration p
1) [Log API](log.md)
2) [Config parsing](config.md)
3) [Tests utilities](testutils.md)
4) [EFI framebuffer](efifb.md)

9
docs/shelter/kernel/log.md
View File

@@ -2,7 +2,7 @@
## Introduction
The logging API is one of the most important of the entire kernel: it allow it to output informations with various level of importance and various sources.
The logging API is one of the most important of the entire kernel: it allow it to output informations with various level of importance and various sources, during the early boot period. After the APs bootstrap, the log API is replaced by the print implementation provided by the standard library, because it is thread safe.
## Output methods
@@ -35,7 +35,7 @@ There are two mores functions:
## Output channels and log level
The Shelter logging API provide higher level functions to output on specific log channels. Here are all the log channels:
The Shelter logging API provides higher level functions to output on specific log channels. Here are all the log channels:
- `SH_LOG_DEBUG`: log level 0
- `SH_LOG_LOG`: log level 1
- `SH_LOG_WARNING`: log level 2
@@ -43,6 +43,7 @@ The Shelter logging API provide higher level functions to output on specific log
- `SH_LOG_CRITICAL`: log level 4
- `SH_LOG_FATAL`: log level 5
- `SH_LOG_TEST`: log level 6
- `SH_LOG_FAULT`: log level 7
Each log channel has a log level. The log level is set by the `log_level` boot configuration key. The log API will only output log messages with log level equal or higher to current log level. This restriction doesn't apply to log channel `SH_LOG_TEST`.
@@ -69,7 +70,7 @@ The `sh_log_OUTPUT_SOURCE` type is used to store log source. It's a wrapper of `
## Output payloads
In order to compile all of those informations into one payload, the logging API provide this structure :
In order to compile all of those informations into one payload, the logging API provides this structure :
``` C
typedef struct {
sh_log_OUTPUT_TYPE output_type;
@@ -115,7 +116,7 @@ They all need two arguments in the following order (except `sh_log_ltest` which
### Logging with formating
In order to allow for formatting log messages, the log API provide functions named like this:
In order to allow for formatting log messages, the log API provides functions named like this:
```
sh_log_f[test/debug/log/warning/error/critical/fatal]
```

11
docs/shelter/memory/index.md
View File

@@ -2,7 +2,7 @@
## Introduction
The memory subsystem is responsible for handling tasks like physical pages allocations, virtual memory management, initial memory map analysis, pages mapping and unmapping and kernel heap management.
The memory subsystem is responsible for handling tasks like physical pages allocations, virtual memory management, initial memory map analysis, pages mapping and unmapping and kernel heap management. Please note that, unless used with `sh_malloc` or `sh_free`, no object or function described in this subsystem is thread safe.
## Summary
@@ -14,3 +14,12 @@ The memory subsystem is responsible for handling tasks like physical pages alloc
6) [Radix trees subsystem](radix.md)
7) [Pez plane manager](pez.md)
8) [Kernel heap manager](heap.md)
## Memory subsystem
The memory subsystem is defined into `shelter/lib/include/memory/memory.h` and implemented into `shelter/lib/include/memory/memory.c`.
The entire memory subsystem is initialized with the `sh_memory_init_subsystem()` function. This function produces a `sh_memory_MEMORY_SERVICES` struct, which is stored in the main file and contains all the elements necessary for the memory subsystem initialization.
The memory subsystem also provides some higher level functions:
- `sh_memory_identity_map(sh_page_PHYSICAL_ADDRESS phys_start,sh_uint64 page_count,sh_uint64 flags)`: identity map a range of physical pages into the page table of the booting kernel. It is not thread safe.

5
docs/shelter/memory/page.md
View File

@@ -85,6 +85,9 @@ This role of memory allocations is lost once Pez is initialized.
The Page subsystem provide the following functions regarding memory statistics:
- `sh_uint64 sh_page_get_physical_memory_amount_pages()`: return amount of physical memory installed in pages
- `sh_uint64 sh_page_get_physical_memory_amount_bytes()`: return amount of physical memory installed in bytes
- `sh_uint64 sh_page_get_installed_memory_bytes()`: return the amount of physical memory in pages counted as free when parsing the memory map
- `SH_STATUS sh_page_get_memory_stats(sh_page_MEM_STATS *mem_stats)`: provide a extensive amount of statistics on physical memory.
For a more human-readable output, the function `sh_log_mem_stats` can be used.
For a more human-readable output, the function `sh_log_mem_stats` can be used for debugging purposes. It doesn't rely on Pez.
When Pez is ready, please uses instead the DevS memory subsystem endpoint, it's faster and refined.

2
docs/shelter/memory/ring.md
View File

@@ -6,6 +6,6 @@ The memory subsystem provide a simple ring buffer API, mainly used by the log AP
## Overview
The main structure for a ring buffer is `sh_ring_RING_BUFFER_HEADER`. The function provided by the API are volontary very simple for the moment:
The main structure for a ring buffer is `sh_ring_RING_BUFFER_HEADER`. The function provided by the API are volontarily very simple for the moment:
- `SH_STATUS sh_ring_write_byte(sh_ring_RING_BUFFER_HEADER *ring_buffer,sh_uint8 byte)`: write a byte inside provided ring buffer
- `SH_STATUS sh_ring_write_string(sh_ring_RING_BUFFER_HEADER *ring_buffer,char *string)`: write a null-terminated string inside provided ring buffer

4
docs/shelter/naming.md
View File

@@ -1,11 +1,13 @@
# Naming scheme
Shelter has a very specific way of organizing subsystems.
First, each part of the kernel (except `main.c` which is responsible for the boot process) is in his own folder. To this date, there is 4 main parts:
First, each part of the kernel (except `main.c` which is responsible for the boot process) is in his own folder. To this date, there is 6 main parts:
- `cpu`: CPU abstractions
- `kernel`: kernel services
- `std`: Shelter standard library
- `memory`: the whole memory subsystem
- `irq`: all thing IRQ-related (GDT, TSS, IDT, IRQ handler and GSI)
- `devs`: all things devices-related, including drivers, Device System API and ACPI parsers
Then, each of these parts have specific subsystem or API prefix. These prefixes always start by `sh_` and doesn't include the part each subsystem or API is in. For exemple, the serial outputing API prefix is `sh_serial_`.

4
docs/shelter/ptp.md
View File

@@ -2,7 +2,7 @@
## Introduction
In order to easely manage pages that are used for pages tables (PT), the Vystem project use pages tables pool (PTP). For details about mapping the PTP inside the kernel virtual memory space, please see [boot contract docs](bootcontract.md).
In order to easily manage pages that are used for pages tables (PT), the Vystem project use pages tables pool (PTP). For details about mapping the PTP inside the kernel virtual memory space, please see [boot contract docs](bootcontract.md).
## Overview
@@ -20,4 +20,4 @@ PTP, when created from the bootloader, are mapped within themself read-write and
The purpose of any PTP is to have a reserved amount of pages to permit the quick allocation of single page to allow any mapping. There is two kinds of allocator, depending on the needs of the situation:
- bumb/counter allocator: here, we simply increment a counter that represent an index inside the pages range of the PTP, allowing for very quick PTP initialization. Intended for PTP creation and mapping without unmapping by the program creating the PTP. This is the kind of allocator used into the Blastproof bootloader
- bitmap allocator: a bitmap for storing which pages of the pages range is allocated or not. Intended when heavy unmapping is made, to free pages more easely. This is the allocator used in the Shelter Kernel. It can be initialized from the counter value, passed by the bootloader for example. For the moment, automatic pages freeing from the page table isn't implemented.
- bitmap allocator: a bitmap for storing which pages of the pages range is allocated or not. Intended when heavy unmapping is made, to free pages more easily. This is the allocator used in the Shelter Kernel. It can be initialized from the counter value, passed by the bootloader for example. For the moment, automatic pages freeing from the page table isn't implemented.

4
docs/shelter/std/malloc.md
View File

@@ -1,7 +1,9 @@
# Heap memory allocations
The Shelter standard library provide a single API for memory allocations on the heap. The heap is only initialized at the end of the memory subsystem and can't be used before. It's not adapted for memory mapped I/O or big buffers allocations (larger than one hundrer pages). The heap internal documentation can be found inside the memory subsystem documentation.
The Shelter standard library provide a single API for memory allocations on the heap. The heap is only initialized at the end of the memory subsystem and can't be used before. It's not adapted for memory mapped I/O or big buffers allocations (roughly than one hundred pages). The heap internal documentation can be found inside the memory subsystem documentation.
The memory allocations API provide the two following functions (defined inside `shelter/lib/include/std/malloc.h` and implemented inside `shelter/lib/src/std/malloc.c`):
- `void* sh_malloc(sh_uint64 size)`: allocate `size` amount of bytes. Return `SH_NULLPTR` if an error occured. The heap internal will trigger a heap crash (essentially a `while (SH_TRUE)` loop) to prevent any further damage if something very bad happen.
- `void sh_free(void *ptr)`: free the memory allocated at `ptr`. The heap internal will trigger a heap crash (essentially a `while (SH_TRUE)` loop) to prevent any further damage if something very bad happen.
To prevent any race conditions, there is a global lock on the entry point of the heap in the standard library. After APs bootstrap, only those entry point should be used for basic memory allocation.

2
docs/shelter/std/mem.md
View File

@@ -1,6 +1,6 @@
# Basic memory operations
The Shelter standard library provide very basic memory operations primitives, defined in `shelter/lib/include/std/mem.h` and implemented inside `shelter/lib/src/std/mem.c`:
The Shelter standard library provides very basic memory operations primitives, defined in `shelter/lib/include/std/mem.h` and implemented inside `shelter/lib/src/std/mem.c`:
- `SH_STATUS sh_mem_compare(const void *a,const void *b,sh_uint64 size)`: compare two memory regions with the same size. Return `SH_STATUS_SUCCESS` if both regions are equal, or `SH_STATUS_MEM_NOT_EQUAL` if one byte is different.
- `SH_STATUS sh_mem_copy(const void *destination,const void *source,sh_uint64 size)`: copy one region of memory to another. Return `SH_STATUS_SUCCESS`
- `SH_STATUS sh_mem_set_8(sh_uint8 *ptr,const sh_uint8 byte,sh_uint64 count)`: set a provided amount of bytes to the value of one provided byte. Return `SH_STATUS_SUCCESS`

53
docs/shelter/std/print.md Normal file
View File

@@ -0,0 +1,53 @@
# Print primitives
## Introduction
The standard library provides his own abstraction to print strings. This abstraction can be used before the APs bootstrap and must be used after the APs bootstrap. The print implementation in the standard library is fully thread-safe. The print implementation is defined inside `shelter/lib/include/std/print.h` and implemented inside `shelter/lib/include/std/print.c`.
The format syntax is the same used in the kernel logging API. See [format syntax](../kernel/log.md).
## Print initialization
Before the initialization of the print implementation, all output using `sh_print` and `sh_printf` is redirected to the kernel logging API.
The initialization of the print implementation is done through `sh_print_setup_ring_buffers_per_ap()`. This function initializes a ring buffer for each CPU, including the bootstrap CPU. The size of each ring buffer is the amount of bytes for the global logging ring buffer divided by the amount of CPU.
Every single print primitives (except `sh_sprintf`, which doesn't print anything) respect the log level specified in the kernel boot configuration.
## Characteristics
This print implementation differ of the kernel logging API by these differences:
- It doesn't support displaying the name of the subsystem
- It shows the logical CPU id of the AP that called the print implementation
It also defines its own macros for the type of output: `SH_DEBUG`, `SH_LOG`, `SH_WARNING`, `SH_ERROR`, `SH_CRITICAL`, `SH_FATAL`, `SH_TEST`, `SH_FAULT`
## Normal printing
### API
The print implementation provide two functions for normal printing:
- `sh_print(sh_log_OUTPUT_TYPE output_type,char *text)`: print a normal string without formatting the text before hand
- `sh_printf(sh_log_OUTPUT_TYPE output_type,char* format,...)`: format and print the formatted string byte by byte
### Detailled processus
For normal printing, the outputed text goes through two main steps:
First, each byte is outputed into the local printing ring buffer. We keep track of the amount of bytes outputed using the attribute in the per-CPU struct of this CPU.
Then, extracting the outputed byte from the local printing ring buffer, all outputed bytes are sent to the serial port, no without locking a global serial spinlock before.
## Interrupts printing
The print system also provides a dedicated path for interrupt handlers. In this mode, output bypasses the local ring buffer and is written directly to the serial port after acquiring the serial spinlock. If the lock is already held by another CPU, the function will block until it becomes available.
The API is as follows:
- `sh_iprint(sh_log_OUTPUT_TYPE output_type,char *text)`: same as `sh_print` but for interrupt handlers
- `SH_STATUS sh_iprintf(sh_log_OUTPUT_TYPE output_type,char* format,...)`: same as `sh_iprintf` but for interrupt handlers
## String formatting
Finally, the print implementation provides a way to easely format strings using the same format syntax, using `sh_sprintf(char *output_string,sh_uint64 output_len,char *format,...)`.
This function writes into the provided buffer only. It doesn't print to the serial port, doesn't use ring buffers, and doesn't allocate memory via `sh_malloc`.

23
docs/shelter/std/queues.md Normal file
View File

@@ -0,0 +1,23 @@
# Queues
## Introduction
The standard library provides various queues implementations for various objects sizes. Queue objects aren't thread safe. It is defined inside `shelter/lib/include/std/queue.h` and implemented in `shelter/lib/src/std/queue.c`.
## Keyboard events queue
This queue is represented by the following object:
``` C
typedef struct {
sh_kbd_EVENT *buffer;
sh_uint32 capacity;
sh_uint32 write_index;
} sh_queue_KBD_EVENT;
```
These queues automatically overwrite the oldest object when the caller try to push an object when the queue is full.
The API is as follows:
- `sh_queue_event_init(sh_queue_KBD_EVENT *q,sh_uint32 capacity)`: initialize a keyboard events queue, allocate the queue buffer
- `sh_queue_event_push(sh_queue_KBD_EVENT *q,sh_kbd_EVENT ev)`: push an event into the provided queue
- `sh_queue_event_destroy(sh_queue_KBD_EVENT *q)`: destroy the provided queue, free the queue buffer

54
docs/shelter/std/smp.md Normal file
View File

@@ -0,0 +1,54 @@
# SMP services
## Introduction
The SMP services provide various abstractions related to SMP programming. It is defined inside `shelter/lib/include/std/smp.h` and implemented in `shelter/lib/src/std/smp.c`
## GS register
Regarding the GS register, two functions are provided:
- `sh_smp_write_gs_base(sh_uint64 value)`: Write the `IA32_GS_BASE` MSR. with a value, return nothing
- `sh_smp_gs_base()`: take no argument and return a `sh_ap_CPU_STRUCT*`
## Spinlocks
The SMP services provide a basic implementation for a spinlock, which look like this:
``` C
typedef struct {
volatile sh_uint32 spinlock;
sh_uint32 lapic_id;
} sh_SPIN_LOCK;
```
Two macros are provided to initialize an unlocked spinlock:
- `SH_LOCK()`: intended for spinlocks stored as global variables
- `SH_LOCK_LOCAL()`: intended for spinlocks stored as local variables or into structs
This spinlock implementation requires `sh_smp_gs_base()` to return a valid pointer to a CPU struct. It uses the atomics primitives provided by compilers. Spinlock operations provide full memory ordering guarantees.
Four functions are provided for spinlock manipulation:
- `sh_spin_lock(sh_SPIN_LOCK *lock)`: lock a spinlock, block until the lock is acquired. Return nothing
- `sh_spin_unlock(sh_SPIN_LOCK *lock)`: unlock a spinlock, return nothing
- `sh_spin_trylock(sh_SPIN_LOCK *lock)`: Attempt to acquire the lock without blocking., return `SH_TRUE` if successfull, `SH_FALSE` otherwise.
- `sh_spin_wholock(sh_SPIN_LOCK *lock)`: return the LAPIC id of the CPU locking the spinlock. Return `SH_UINT32_MAX` if the spinlock isn't owned by any CPU
Current implementation doesn't disable interrupts while holding locks.
## CPU count
SMP services can store the amount of CPU cores using the following functions:
- `sh_smp_set_cpu_count(sh_int16 cpu_count)`: set the count of CPU. It is only called one time by `sh_ap_prepare_for_smp_launch()`
- `sh_smp_get_cpu_count()`: return the count of CPU, return `-1` if value isn't initialized
## Memory barrier
The SMP services provide the following primitives:
Function name | Role
--------------|-----
`sh_mb()` | Full memory barrier
`sh_rmb()` | Read memory barrier
`sh_wmb()` | Write memory barrier
Current x86_64 implementation uses mfence/lfence/sfence.

4
docs/shelter/std/std.md
View File

@@ -10,5 +10,9 @@ In order to function properly, any kernel need a standard library. Shelter defin
2) [Return status](status.md)
3) [Basic memory operations](mem.md)
4) [Heap memory allocations](malloc.md)
5) [Print primitives](print.md)
6) [String operations](strings.md)
7) [SMP services](smp.md)
8) [Queues](queues.md)
You can include the file `shelter/lib/include/std/stdlib.h` to include all necessary headers to access the Shelter standard library.

9
docs/shelter/std/strings.md Normal file
View File

@@ -0,0 +1,9 @@
# Strings operations
The Shelter standard library provides basic strings operations primitives, defined in `shelter/lib/include/std/string.h` and implemented inside `shelter/lib/src/std/string.c`:
- `sh_string_len(char *str)`: return the length of a null terminated string
- `sh_string_compare(char *str1,char* str2,sh_uint64 length)`: compare two null terminated string, both strings must not exceed the provided length. return `SH_TRUE` if both strings are equal on the provided length
- `sh_string_find_char(char *str,char character)`: return the index of the first occurence of the provide character into the string, return `SH_UINT64_MAX` if not found or error
- `sh_string_find(char *str,char *substr)`: return the index of the start of the first occurence of the provided substring into the string, return `SH_UINT64_MAX` if not found or error
- `sh_string_substring(char *source_str,sh_uint64 start_index,sh_uint64 length,char *output)`: copy a substring from a string to another string, and return the pointer to the new null terminated string (aka the provided `output` pointer). Return `SH_NULLPTR` if error
- `sh_string_to_uint64(char *str)`: convert a string to a `sh_uint64` and return this `sh_uint64`

8
docs/shelter/tab.md
View File

@@ -43,7 +43,7 @@ The test will separate single page allocations and frees from multi-pages alloca
## Malloc subsystem test
These tests are design to test the reliability and performances of the Pez subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_malloc.h` and implemented inside `shelter/lib/src/kernel/tests/test_malloc.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_malloc_payload.h`.
These tests are designed to test the reliability and performances of the Pez subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_malloc.h` and implemented inside `shelter/lib/src/kernel/tests/test_malloc.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_malloc_payload.h`.
The payload contain the following datas:
- `test_malloc_small_size`: contain 10000 sizes for small allocations (under or equal to 1024 bytes)
@@ -51,3 +51,9 @@ The payload contain the following datas:
- `test_malloc_big_alloc`: contain 2000 values, serving the same role and generated like `test_pez_physical_alloc`
The malloc test combine two tests: 10000 allocations and deallocations of variously sized small objects using the generic slab allocators, and 1000 allocations and frees of pages on the heap, on the same logic of the Pez subsystem test, but adapted to the quantity used in this test.
## Queues tests
This test is mainly designed to stress test the queue implementation provided by the standard library.
It simply consists of 10000 objects pushed into a queue.

1070
docs/vybuild/actions.md Normal file
View File

File diff suppressed because it is too large Load Diff

14
docs/vybuild/cache.md Normal file
View File

@@ -0,0 +1,14 @@
# Cache
## Introduction
VyBuild integrate its own cache, stored in the `.vybuild_cache` folder in the root of the project folder, identified using `getcwd` at the start of the build pipeline execution. It is recommanded to put this folder in your `.gitignore` file if you use Git, or any other equivalent source control software.
## Smart cache
To ensure that the cache only contains what is actually needed for efficient compilation without taking too much storage, VyBuild integrate a unique technology called smart cache. The smart cache is enabled by default and ensure that the cache remain reasonably small on disk.
Instead of relying on metadata like file dates or complex dependency graphs, the smart cache acts at the root of the root of the cache subsystem, and it works even on custom build artifacts stored in cache by the user. It is built upon three simple rules:
- any new cache object that has been created during this build pipeline execution remains in cache
- any cache object that was already in the cache before this build pipeline execution, and that have been used/extracted from the cache during this build pipeline execution remains in cache
- any cache object that was already in the cache before this build pipeline execution, and that hasn't been used/extracted from the cache during this build pipeline execution will be deleted at the end of this build pipeline execution

30
docs/vybuild/index.md Normal file
View File

@@ -0,0 +1,30 @@
# VyBuild Docs
## Introduction
VyBuild is Vystem build system. It has been designed to considerably reduce the need for external tools and the pain of writing custom scripts. It is opinionated and optimized for OS development. While it was designed for Vystem, it can be used for others OS project, but feature such as Vyx/Vyld usage and VFTM aren't guaranteed to work. VyBuild is very integrated into the Vystem ecosystem and should not be considered complete for now.
## Features
VyBuild is capable of the following:
- Compiling C, C++ and Assembly file, with custom arguments, compiler, assembler, and even linker in certain cases.
- Automatically caching build artifacts and compilation objects files, with cached objects invalidation down to modification in the compiler/assembler binary and in C/CPP files included headers
- Automatically purging the cache to keep only newly created objects and objects of previous build that have been used during a build, preventing garbage accumulation
- Letting the user uses the cache how they want to store custom objects
- Running command in indicated folder, running command with auto-faillure if command return a certain status code, show output of commands live, on failure or never, save command output to provided file
- Giving the user complete control over how the build pipeline is parralelised, with flexible options at compilation, actions or modules granularity level
- Compiling Vyx executable with an upgraded and more powerful integration of Vyld
- Acting on the filesystem, like moving/copying files/folder, ensure folder existence or reset
- Letting the user uses system variables (like `%rootcallerfolder%`) or define their own to use in almost any actions field, as well as providing variable on command line argument of VyBuild
- Running differents actions or modules depending on differents condition, like folder existence, cache presence or variable value/existence
- Create fully valid FAT32 images (only support short file names) and fully valid disk images with GPT tables, custom type/unique partitions GUID and importing partitions from file
- Integrating Vystem FAT Trusted Manifest support for FAT32 images generation and interaction with `keygen`
## Index
This documentation is intended to be read in the following order:
1) [Modules](modules.md)
2) [Actions](actions.md)
3) [Variables](variables.md)
4) [Cache](cache.md)
5) [Usage](usage.md)

46
docs/vybuild/modules.md Normal file
View File

@@ -0,0 +1,46 @@
# Modules
## Introduction
Modules are the backbone of VyBuild. They allow you to precisely describe the build process of your project. They are JSON files with a very strict but clear syntax.
## Syntax
Modules can be executed as root module (called from command line) or submodule (called from another module).
A module is made of several fields indexed in a dictionnary. Here is an example:
``` json
{
"version":"0.1",
"root_folder":"...",
"run_context":"...",
"name":"...",
"global_uuid":[
...
],
"global_vars":{
...
},
"actions":[
...
]
}
```
### Mandatory fields
Each module has 5 mandatory fields:
- `version`: a string indicating the version of VyBuild JSON syntax. It must be set to the current version of the VyBuild JSON syntax, which is currently `0.1`
- `root_folder`: a string indicating the working folder of this module, all relative paths must be accessible in this folder. You can use variables in this field (see [variables docs](variables.md)). The path indicated in this field should always be absolute for maximum reliability
- `run_context`: a string indicating in which context this module can be executed. Possible values are:
- `both`: the module can be executed as root module and submodule
- `root`: the module can only be executed as root module
- `sub`: the module can only be executed as submodule
- `name`: a string indicating the name that will be displayed in VyBuild call tree
- `actions`: a list of actions (see [actions docs](actions.md)) to execute in the module
### Optional fields for root module
Modules called from the command line can include the following optional fields :
- `global_uuid`: a list of string giving variables names that will be filled with random UUIDs v4 as each build. They can be used as regular variables
- `global_vars`: a dictionnary of string to string entries, associating variables names to their contents. For more informations, see [variables docs](variables.md)

35
docs/vybuild/usage.md Normal file
View File

@@ -0,0 +1,35 @@
# Usage
## Build
In order to use VyBuild, you must compile it first. For this, use the `build.sh` script located in the `vybuild` folder. This script should only be used from the root folder of the Vystem project, just like this:
``` bash
vybuild/build.sh
```
The `vybuild` binary will be compiled inside the folder of same name.
## Usage
VyBuild can be used like this:
``` bash
vybuild/vybuild <path to root module> <build args>
```
The first argument must be the path to the root module. Arguments that comes after that will be sorted into two categories:
- VyBuild arguments: if VyBuild recognize an argument specific to the build system, it will remove it from the exposed arguments
- Build arguments: these arguments will be exposed to the build pipeline using system variables
Here is the list of VyBuild arguments to this day:
- `--purge-cache`: purge the cache before starting the build pipeline
- `--disable-smart-cache`: disable the smart cache feature for this execution of the build pipeline
## Licenses
External libraries used inside VyBuild:
- ELFIO, made by Serge Lamikhov-Center, sourced from [serge1/ELFIO](https://github.com/serge1/ELFIO), under the MIT license
- json, made by Niels Lohmann, sourced from [nlohmann/json](https://github.com/nlohmann/json), under the MIT license
Others libraries are used when loading the dynamic library `commoncrypto`, which can be found [here](../boottools/commoncrypto.md)

59
docs/vybuild/variables.md Normal file
View File

@@ -0,0 +1,59 @@
# Variables
## Introduction
Variables in VyBuild allow for rapid changes of the build pipeline parameters and can be used almost anywhere.
## Overview
In VyBuild, there are two types of variables:
- user-defined variables (also called global variables): these are defined as pairs of string in the `global_vars` field of every root module. They are usables in all parts of the build pipeline without restriction.
- systems variables: these are special variables defined by VyBuild containing specific values. Some are usable globally and some are only usables in some fields. For more information about these ones, see [actions docs](actions.md)
## Using a global variable
To define a global variable with a custom value, simply add a pair of string inside the `global_vars` field of the root module of your pipeline.
To define a global variable with an automated UUID v4 that changes at each build, simply add a string inside the `global_uuid` field of the root module of your pipeline.
You can use variables in almost every fields that are either a string or a list of strings. You know if a field support variables if there is a boolean field in the same action with the name `<field name>_variables`. These variables fields are mandatory and indicate if VyBuild will attempt to detect and replace variables in any field where the corresponding variable field is set to `true`.
To use global variables in a field that support it, just put your variable name between `$`.
Example with `SOME_VARIABLE` containing `somes informations`:
``` json
{
"field_variables":true,
"field":"hey$SOME_VARIABLE$hey"
}
```
In this example, the value that VyBuild sees after replacing the variable is `heysomes informationshey`.
Variable names are usually in uppercases letters to avoid any confusion with others parts of a field. This is to follow the same C macros style.
### Specials global variables
Some global variables are a way for some parts of VyBuild to let the user add features to them. Here is a exhaustive list of these special global variables
- `__VYBUILD_COMPILER_HASH_`: any global variables starting with this prefix will be used by actions that compile C/CPP code to include optional information on the used toolchain. These actions will consider the content of these variables as already-formated commands that they can execute and include the output of these commands into the cache keys of each cached objects. That way, any change to the toolchain can be detected and trigger a recompilation
## Using systems variables
Systems variables are variables created by VyBuild, usable in certain contexts. This file will only cover systems variables that aren't specific to certains actions.
Systems variables can be used in almost any fields with some limitations. When used, they are framed with `%`. Here is a complete list:
Name | Content | Behaviour
-----|---------|----------
`%rootcallerfolder%` | The full path of the parent folder of the root module of every build pipeline | Usable in every action fields that support it. Can also be used in the `root_folder` field of every module but alone (can't be combined in a path for this field).
`%filefolder%` | The absolute path of the directory containing the current module's JSON file | Can only be used alone (can't be combined in a path for this field) in the `root_folder` of every module.
Command line arguments that aren't VyBuild parameters will be exposed as systems variables under the index of the argument in the command line.
Example:
``` bash
vybuild <path to root module> img
```
In this example, the system variable `1` will be created by VyBuild with the value `img`, and can be used in any action fields that support variables using the syntax `%1%`.

5
docs/vyld/index.md
View File

@@ -1,5 +0,0 @@
# Vyld Docs
Vyld, standing for Vystem Linker, is a specialised linker designed to convert ELF binaries to the VYX executable format. This documentation is divided into two parts:
- [The VYX executable format](vyx.md)
- [The Vyld utility](vyld.md)

67
docs/vyld/vyld.md
View File

@@ -1,67 +0,0 @@
# Vyld Docs
## Introduction
Vyld, standing for Vystem Linker, is the utility responsible for creating VYX executable from ELF binaries. Vyld has been intended to be used only on a regular Linux system. For a better understanding, it's recommanded you read the [VYX format docs](vyx.md) first.
## Detailled informations
Folder: `vyld`
Source code file: `vyld.cpp`
For building the `vyld` utility, use the following command:
``` bash
g++ vyld.cpp -o vyld
```
External library:
- ELFIO, made by Serge Lamikhov-Center, sourced from [serge1/ELFIO](https://github.com/serge1/ELFIO), under the MIT license
## Usage
Requirement: having the `gcc` compiler and `ld` linker reachable in path. For the moment, only the `gcc` compiler is supported.
The `vyld` utility can be used like this:
``` bash
vyld <path to C file 1> <path to C file 2> ... <path to C file n> <path to output file>
```
Each path provided except the last one has to be C file, or else the compilation might fail. The last path can be any extension, but for standard respect, it should be a VYX file.
Vyld can accept custom compilation flags for provided C files if they are passed by using the `VYLD_COMPILATION_FLAGS` environnement variable. They are added to the mandatory compilation flags for provided C files but they will not replace them. The mandatory compilation flags for provided C files are:
``` bash
-fno-pic -m64 -mcmodel=large
```
The mandatory compilation flag for the `_vyx_start.c` entry point are:
``` bash
-ffreestanding -nostdlib -nostartfiles -Wall -Wextra -fno-stack-protector -fno-builtin -fno-pic -m64 -mcmodel=large
```
These flags aren't applied to provided C files unless passed inside `VYLD_COMPILATION_FLAGS`.
Vyld need to have access to the `_vyx_start.c` file in his executable directory in order to compile any VYX binary. This file is the entry point of any VYX executable.
Command example:
``` bash
vyld hello.c io.c binary.vyx
VYLD_COMPILATION_FLAGS="-Wall -Wextra -Werror -Wpedantic" vyld hello.c io.c binary.vyx
```
We plane to expand drastically the capabilities and versatility of the `vyld` utility and the VYX format in the future, but for the moment, it's sufficient for what we need (loading the Shelter kernel).
## Detailled generation processus
1) Obtaining his own executable path by reading `/proc/self/exe` in order to obtain a deterministic working directory
2) Searching for flags provided into the `VYLD_COMPILATION_FLAGS`
3) Checking for existence of `_vyx_start.c ` and compiling it into `_vyx_start.o` using mandatory compilation flags for entry point. The resulting object file is stored inside the `vyld` executable directory
4) Iterating for each provided C files: finding his location and generating his output name under the format `<file name without .c extension>.o`
5) Iterating for each provided C files: compiling each C file into an object file using mandatory compilation flags for provided C files and flags passed through `VYLD_COMPILATION_FLAGS`. The resulting objects files are stored inside the `vyld` executable directory
6) Combining all objects files (entry point and all provided files under their object form) into one object file named `combined.o` and stored inside the `vyld` executable directory. For that, we use the `gcc -r` feature of `gcc`. The entry point object file is specifically added in first position
7) Opening `combined.o` and obtaining sections size for `.text`, `.data`, `.rodata` and assimiled sections as well as `.bss` section. Padding these sizes to the nearest multiple of 4096 bytes superior to these sizes
8) Generating the `.text` section base virtual address (VA) and stack base VA, aligned on 4096 bytes pages
9) Generating a custom linker script to correctly link the `combined.o` file with the right VAs inside the code and putting in order all the sections into the final binary
10) Linking the `combined.o` file into an ELF executable named `compiled.elf` and stored inside the `vyld` executable folder
11) Opening the `compiled.elf` binary, extracting all sections, generating VYX header and executable
The range for generating `.text` section base is currently `0xFFFF800000000000` to `0xFFFF900000000000-1`, then padded to the nearest page boundary.
The range for generating stack base is currently `0xFFFFF00000000000` to `0xFFFFFF8000000000-0x100000`, then padded to the nearest page boundary.

26
docs/vyld/vyx.md → docs/vyx.md
View File

@@ -6,15 +6,16 @@ The VYX format, standing for Vystem Executable, is a very simple binary format m
## Header
Because VYX format has been designed to provide a simple way to load any kind of binary, we wanted to make it as easely updatable as possible. For the moment, it's just a very simple format built for loading the Shelter kernel, but in a few updates, it will be use for loading all kind of binaries. No spoiler on that. The header is designed like that:
Because VYX format has been designed to provide a simple way to load any kind of binary, we wanted to make it as easily updatable as possible. For the moment, it's just a very simple format built for loading the Shelter kernel, but in a few updates, it will be use for loading all kind of binaries. No spoiler on that. The header is designed like that:
- Signature: three bytes spelling `VyX` in ASCII
- VYX format version: the version of the VYX format used in the binary, represented as an uint16_t. Depending on the version number, the following structure will probably changed a lot. The current version is `0x0001`.
- VYX format version: the version of the VYX format used in the binary, represented as an uint16_t. Depending on the version number, the following structure will probably changed a lot. The current version is `0x0002`.
- `.text` base address: the virtual address (VA) at which the `.text` should be loaded. It should be aligned on the boundary of a 4 kilobytes page. Represented as an uint64_t
- stack base: the VA at which the stack base should be setuped. Be careful, it indicate the base, not the top. It should be aligned on the boundary of a 4 kilobytes page. Represented as an uint64_t
- stack base: the VA at which the stack base should be setup. Be careful, it indicates the base, not the top. It should be aligned on the boundary of a 4 kilobytes page. Represented as an uint64_t
- the size of the `.text` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the size of the `.data` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the size of the `.rodata` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the size of the `.bss` section: It should be aligned on a multiple of 4096 bytes. Represented as an uint64_t
- the number of payloads integrated into the VYX file. Represented as an uint64_t
All the multibytes integers should be written in little endian. The VYX executable format has been made for x86-64 only binary.
@@ -23,9 +24,26 @@ All the multibytes integers should be written in little endian. The VYX executab
The VYX format only support three sections inside the binary that should be in the following order: `.text`, `.data`, `.rodata`. The `.bss` should be initialized and zeroed by the loader just after the `.rodata` section. The space occupied by the three sections inside the binary should occupy a multiple of 4096 bytes, according to the size indicated in the header, padded with zeroes if necessary.
For example, if each sections take 2 pages, after being padded to the nearest multiple of 4096 bytes superior to their real size, and that the `.text` base is 0x1000, `.text` should be located at 0x1000, `.data` at 0x3000, `.rodata` at 0x5000 and `.bss` at 0x7000
## Payloads
The VYX format now support, as of `0x0002` version, the integration of payloads. There are additionnal ressources that the VYX loader need to discover.
This discovery process is started from the end of the `.rodata` section. The loader need to look for the signature `VPAYLOAD` in ASCII. Once localised, the loader know it has found a payload header, which is structured like this:
- Signature: 8 bytes
- Payload size in bytes: represented as an uint32_t
- Payload size in amount of 4 kilobytes pages: represented as an uint32_t
- Payload mapping flags: 1 byte specifying how the VYX loader should map the payload in the virtual address space of the program being loaded:
- Bit 0: if 1, the payload should be mapped as executable data
- Bit 1: if 1, the payload should be mapped as writable data
- Payload virtual address! the address at which the payload should be mapped by the VYX loader. Should be aligned on 4 kilobytes pages frontier and shouldn't be in reserved virtual address space for program image and stack
Then, for the indicated payload size in bytes, the VYX loader should extract the payload (that come immediately after the payload header) and map it accordingly. Restriction can be applied on mapping flags depending on the VYX loader context.
If the payload size in bytes is bigger that the remaining size in bytes into the VYX file, it indicates VYX file corruption. The discovery process continue as long as the amount of payloads discovered isn't the same as indicated into the header.
## Specifications
The VYX binaries use the SystemV ABI. They doesn't support rellocations nor dynamic linking. While relative addressing into the binary code is theorically working, all the Vystem binaries will be compiled 64 bits static addressing with position-dependent code (non-PIC), especially the kernel. That why respecting the indicated `.text` section base and order of sections loading is extremely important.
The VYX binaries use the SystemV ABI. They doesn't support rellocations nor dynamic linking. While relative addressing into the binary code is theoretically working, all the Vystem binaries will be compiled 64 bits static addressing with position-dependent code (non-PIC), especially the kernel. That why respecting the indicated `.text` section base and order of sections loading is extremely important.
Obviously, the right privileges should be applied in term of pagging: no execution unless `.text` pages, read only `.text` and `.rodata` sections, read-write `.data` and `.bss` sections.
For the moment, the entry point is located at the base of the `.text` section.
The stack size can be decided at the convenience of the loader, but the stack size used for the Shelter kernel is, for the moment, 256 kilobytes. The stack growing downward, the initial stack pointer should be set to stack base + stack size.