lolo859/vystem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First commit, Vystem v0.1

Browse Source
This commit is contained in:
lolo859
2026-03-31 22:15:00 +02:00
commit e15daed8c0
462 changed files with 134655 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
docs/shelter/bench.md Normal file
View File

@@ -0,0 +1,149 @@
# Benchmark results
In this file, you will be able to see examples of benchmark results. These are provided as indicative results. Depending on your configuration, you might get way differents results
## Tests environnement
- Host OS: EndeavourOS x86_64
- Host kernel: Linux 6.19.10-arch1-1
- Host CPU: Ryzen 7 9800x3d @ 5.27 GHz
- Environnement: VM running in qemu-system-x86_64 version 10.2.2
- VM parameters: KVM enabled, 4096M of RAM, cpu set to host, running in single thread
The following tests are shown in the order the kernel run them. Please keep in mind that Pez isn't available until the Pez physical plane benchmark.
## Results for region objects slab allocators
Physical regions object slab allocator:
```
Result for benchmark "allocations" :
Min: 141 | Med: 235 | Avg: 2736 | Max: 3194120 | Total : 27362977 (TSC)
[P00-P90] 141-282 : [##################--] 90%
[P90-P99] 282-282 : [#-------------------] 9%
[P99-Max] 282-3194120 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 141 | Avg: 132 | Max: 376 | Total : 1320700 (TSC)
[P00-P90] 94-141 : [##################--] 90%
[P90-P99] 141-141 : [#-------------------] 9%
[P99-Max] 141-376 : [--------------------] 1%
```
Virtual regions object slab allocator:
```
Result for benchmark "allocations" :
Min: 141 | Med: 235 | Avg: 2781 | Max: 2840445 | Total : 27816339 (TSC)
[P00-P90] 141-282 : [##################--] 90%
[P90-P99] 282-329 : [#-------------------] 9%
[P99-Max] 329-2840445 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 141 | Avg: 130 | Max: 517 | Total : 1304109 (TSC)
[P00-P90] 94-141 : [##################--] 90%
[P90-P99] 141-141 : [#-------------------] 9%
[P99-Max] 141-517 : [--------------------] 1%
```
## Results for radix node slab allocator (PBA based)
Radix node slab allocator (PBA based):
```
Result for benchmark "allocations" :
Min: 141 | Med: 188 | Avg: 441 | Max: 534531 | Total : 4410339 (TSC)
[P00-P90] 141-235 : [##################--] 90%
[P90-P99] 235-282 : [#-------------------] 9%
[P99-Max] 282-534531 : [--------------------] 1%
Result for benchmark "deallocations" :
Min: 94 | Med: 94 | Avg: 97 | Max: 329 | Total : 973370 (TSC)
[P00-P90] 94-94 : [##################--] 90%
[P90-P99] 94-141 : [#-------------------] 9%
[P99-Max] 141-329 : [--------------------] 1%
```
## Results for radix trees subsystem
Radix trees subsystem:
```
Result for benchmark "insertions into radix trees" :
Min: 2350 | Med: 3196 | Avg: 5194 | Max: 544448 | Total : 51944259 (TSC)
[P00-P90] 2350-3666 : [##################--] 90%
[P90-P99] 3666-134373 : [#-------------------] 9%
[P99-Max] 134373-544448 : [--------------------] 1%
Result for benchmark "reading into radix trees" :
Min: 470 | Med: 517 | Avg: 553 | Max: 31631 | Total : 5537399 (TSC)
[P00-P90] 470-564 : [##################--] 90%
[P90-P99] 564-987 : [#-------------------] 9%
[P99-Max] 987-31631 : [--------------------] 1%
Result for benchmark "searching value with lower bound key" :
Min: 423 | Med: 893 | Avg: 923 | Max: 30832 | Total : 9234184 (TSC)
[P00-P90] 423-1128 : [##################--] 90%
[P90-P99] 1128-1645 : [#-------------------] 9%
[P99-Max] 1645-30832 : [--------------------] 1%
Result for benchmark "deleting values" :
Min: 1645 | Med: 1880 | Avg: 1888 | Max: 35203 | Total : 18889770 (TSC)
[P00-P90] 1645-2021 : [##################--] 90%
[P90-P99] 2021-2162 : [#-------------------] 9%
[P99-Max] 2162-35203 : [--------------------] 1%
```
## Results for Pez physical plane
Pez physical plane:
```
Result for benchmark "allocations for single page" :
Min: 1833 | Med: 2444 | Avg: 3496 | Max: 8225 | Total : 3496518 (TSC)
[P00-P90] 1833-5922 : [##################--] 90%
[P90-P99] 5922-6815 : [#-------------------] 9%
[P99-Max] 6815-8225 : [--------------------] 1%
Result for benchmark "free for single page" :
Min: 1974 | Med: 2820 | Avg: 3807 | Max: 10105 | Total : 3807376 (TSC)
[P00-P90] 1974-6862 : [##################--] 90%
[P90-P99] 6862-8131 : [#-------------------] 9%
[P99-Max] 8131-10105 : [--------------------] 1%
Result for benchmark "allocations for multiple pages" :
Min: 2209 | Med: 5593 | Avg: 5716 | Max: 35861 | Total : 5716798 (TSC)
[P00-P90] 2209-7191 : [##################--] 90%
[P90-P99] 7191-8178 : [#-------------------] 9%
[P99-Max] 8178-35861 : [--------------------] 1%
Result for benchmark "free for multiple pages" :
Min: 1927 | Med: 4935 | Avg: 5095 | Max: 31772 | Total : 5095599 (TSC)
[P00-P90] 1927-7896 : [##################--] 90%
[P90-P99] 7896-10011 : [#-------------------] 9%
[P99-Max] 10011-31772 : [--------------------] 1%
```
## Results for malloc subsystem
Malloc subsystem:
```
Result for benchmark "sh_malloc for small size" :
Min: 188 | Med: 423 | Avg: 535 | Max: 380277 | Total : 5358752 (TSC)
[P00-P90] 188-799 : [##################--] 90%
[P90-P99] 799-1363 : [#-------------------] 9%
[P99-Max] 1363-380277 : [--------------------] 1%
Result for benchmark "sh_free for small size" :
Min: 141 | Med: 329 | Avg: 403 | Max: 27401 | Total : 4039556 (TSC)
[P00-P90] 141-658 : [##################--] 90%
[P90-P99] 658-1128 : [#-------------------] 9%
[P99-Max] 1128-27401 : [--------------------] 1%
Result for benchmark "sh_malloc for pages allocations" :
Min: 4935 | Med: 11468 | Avg: 11890 | Max: 47893 | Total : 11890013 (TSC)
[P00-P90] 4935-16450 : [##################--] 90%
[P90-P99] 16450-20257 : [#-------------------] 9%
[P99-Max] 20257-47893 : [--------------------] 1%
Result for benchmark "sh_free for pages allocations" :
Min: 5593 | Med: 11421 | Avg: 12005 | Max: 50760 | Total : 12005398 (TSC)
[P00-P90] 5593-17296 : [##################--] 90%
[P90-P99] 17296-21432 : [#-------------------] 9%
[P99-Max] 21432-50760 : [--------------------] 1%
```

64
docs/shelter/bootconfig.md Normal file
View File

@@ -0,0 +1,64 @@
# Shelter kernel boot configuration
## Introduction
The Shelter kernel requires a boot configuration in order to start. It doesn't come under the form of a command line but under the form of a C struct. For details on how to transmit the configuration to the kernel, please see [boot contract docs](bootcontract.md).
## Overview
Shelter boot configuration come under the following layout (extracted from `shelter/lib/include/kernel/conf.h`) :
``` C
typedef struct __attribute__((aligned(8))) {
sh_uint8 sig_start[8];
sh_uint8 log_level;
sh_uint16 page_table_allocator_level;
sh_page_PHYSICAL_ADDRESS page_table_pool_pa;
sh_page_VIRTUAL_ADDRESS page_table_pool_va;
sh_bool test_benchmark;
sh_uint64 bench_iterations;
sh_bool log_disable_serial_port;
sh_bool disable_serial_port;
sh_uint16 log_ring_size;
sh_uint8 sig_end[8];
} sh_conf_BOOT_CONFIG;
```
The starting signature must be `ShCfgBeg` in ASCII and the starting signature must be `ShCfgEnd` in ASCII. The alignment to respect for the boot config is 8 bytes.
## List of all keys
**log_level:**
- Type: 1 byte unsigned integer
- Description: represent the minimal (inclusive) log level for any log payload to be logged, can range from 0 to 6. The value of this key is provided by the `kernel_log_level` from the Blastproof boot config
**page_table_allocator_level:**
- Type: 2 bytes unsigned integer
- Description: the value of the bumb allocator after all mapping inside the kernel boot PTP
**page_table_pool_pa:**
- Type: physical address, aka 8 bytes unsigned integer
- Description: the physical address of the first page of the kernel boot PTP
**page_table_pool_va:**
- Type: virtual address, aka 8 bytes unsigned integer
- Description: the virtual address of the first page of the kernel boot PTP, inside kernel virtual memory layout
**test_benchmark:**
- Type: boolean, aka 1 byte unsigned integer
- Description: define if the kernel should test and benchmark his subsystems. The value of this key is provided by the `kernel_test_benchmark` from the Blastproof boot config
**bench_iterations:**
- Type: 8 bytes unsigned integer
- Description: define the amount of iterations to apply for compatible benchmarks. Not all benchmarks will follow this value. The value of this key is provided by the `kernel_bench_iterations` from the Blastproof boot config
**log_disable_serial_port:**
- Type: boolean, aka 1 byte unsigned integer
- Description: if true, the kernel will not output his logs on the serial port. The value of this key is provided by the `kernel_log_disable_serial_port` from the Blastproof boot config
**disable_serial_port:**
- Type: boolean, aka 1 byte unsigned integer
- Description: define if the kernel should allow serial port usage. The value of this key is provided by the `kernel_disable_serial_port` from the Blastproof boot config
**log_ring_size:**
- Type: 2 bytes unsigned integer
- Description: define the amount of pages used for logging ring buffer

66
docs/shelter/bootcontract.md Normal file
View File

@@ -0,0 +1,66 @@
# Shelter boot contract
## Introduction
The Shelter boot contract is the list of steps necessary in order to successfully boot Shelter.
## Prerequisites
We assume the following:
- you understand the concept of pages table pool, Shelter boot configuration and Shelter memory map
- you have a bootloader capable of mapping things into into pages table pool
- your bootloader is capable of generating Shelter boot configuration and Shelter memory map
- your bootloader is capable of allocating or call the UEFI firmware to allocates pages to copy data into them
- your bootloader is capable of parsing VYX executable
- your bootloader is capable of accessing the content of the Shelter VYX executable and keycard binary
While this documentation is redacted like a tutorial, this isn't a tutorial nor intended to be understand as one.
## Step 1: allocate pages
Your bootloader should be allocating 4KB physical pages ranges and store their physical start address for the following elements with the following type if you use the `AllocatePages` from the EFI firmware:
- pages for `.text` section should be `EfiLoaderCode` type
- pages for `.data`, `.rodata`, `.bss` sections should be `EfiLoaderData` type
- pages for stack should be `EfiLoaderCoe` type. The standard amount of stack pages is 64
- page(s) for keycard should be `EfiLoaderCode` type. Keycard should fit into one page, but you are free to allocate more if you want
- page(s) for Shelter boot configuration, memory map and pages table pool should be `EfiLoaderCode`. Shelter boot configuration should fit into one page, but you are free to allocate more if you want
- pages for Shelter logging ring should be `EfiLoaderCode` type
Note: even if the logging ring size specified in the configuration is less than 4 pages, at least 4 pages should be allocated, because the kernel logs some things before loading the boot configuration that specify that log in ring buffer is disabled
## Step 2: prepare pages
The following steps should be completed before mapping the pages:
- zeroing the `.bss` section pages range
- zeroing the stack pages range
- zeroing the entire memory map pages range
- zeroing the entire logging ring buffer
- copying the content of the `.text`, `.data` and `.rodata` section inside their respectives pages range
- copying the content of Keycard inside his pages range
- copying the content of the boot configuration inside his pages range
## Step 3: mapping pages
The prerequisites to this step is having a PTP-compatible page mapping function. A working implementation can be found in the `Blastproof/src/libs/src/vyx.c` file.
The following pages ranges will be mapped at the following virtual address in the following order:
- the pages table pool pages range should be mapped first at the virtual address computed like this `text_base + text_size + data_size + rodata_size + bss_size + 0x200000` where `text_base` is the base VA of the `.text` section and the differents sizes are the size of each section, padded to the nearest 4096 bytes (normally it's already the case in the VYX executable header). It should be mapped in read-write with no execution permissions
- the `.text` section should be mapped at the VA indicated by `text_base`, with execution but read-only permissions
- the `.data` section should be mapped at the VA indicated by `text_base + text_size`, with no execution and read write permissions
- the `.rodata` section should be mapped at the VA indicated by `text_base + text_size + data_size` with no execution and read-only permissions
- the `.bss` section should be mapped at the VA indicated by `text_base + text_size + data_size + rodata_size`, with no execution and read-write permissions
- the 64 pages of the stack should be mapped at the VA indicated by `stack_base`, with no execution and read-write permissions
- Keycard pages ranges should be identity mapped to the same virtual address as his physical address, with execution but read-only permissions
- Shelter boot configuration pages should be mapped at VA `0x00180000`, with no execution and read-write permissions
- memory map pages should be mapped at VA `0x00190000`, with no execution and read-only permissions
- logging ring pages should be mapped at VA `0xFFFFFFFFF0000000`, with no execution and read-write permissions
## Step 4: completing configuration
The boot configuration should be completed with the PTP physical and virtual address as well as the level of the counter allocator of the PTP. Don't forget to update the boot configuration already copied inside the boot configuration pages range.
## Step 5: generating memory map
After obtaining the EFI memory map, the memory map should be generated inside her dedicated pages range. The complete syntax of the Shelter memory map can be found [here](memmap.md).
## Step 6: jump
Once all of this is done, the bootloader can exit the EFI boot services, prepare the register and jump to Keycard. The details about completing the registers is provided [here](keycard.md).

89
docs/shelter/bootprocess.md Normal file
View File

@@ -0,0 +1,89 @@
# Shelter Boot Process
## Introduction
Shelter has a very precise boot process to ensure everything goes to plan. In this file, we detail everything that is done for the kernel boot but it doesn't include how to use the kernel API, algorithms, etc. For that, please dig through the documentation.
## 1) Basic kernel initialization
1.1) Print kernel version
1.2) Detect and parse boot config
1.3) Load basic settings
1.3.1) Load logging ring size
1.3.2) Load serial port settings
1.3.3) Load PTP virtual address
1.3.4) Load kernel log level
---
Starting here, all logs go through the standard log API.
---
## 2) Memory subsystem initialization
### 2.1) Page subsystem initialization
2.1.1) Copying memory map to a secured buffer
2.1.2) Initializing current PTP using boot configuration information
2.1.3) Parsing and validating memory map, initializing physical pages bitmap
### 2.2) Pez dependencies subsystems initialization
2.2.1) Initializing physical region slab allocator
2.2.1.a) Initializing slab allocator structure
2.2.1.b) Adding first slab and verifying slab metadata
2.2.2) Initializing virtual region slab allocator
2.2.2.a) Initializing slab allocator structure
2.2.2.b) Adding first slab and verifying slab metadata
2.2.3) Initializing radix tree node slab allocator
2.2.3.a) Initializing page block allocator structure for slab allocator
2.2.3.b) Initializing slab allocator structure
2.2.3.c) Adding first slab and verifying slab metadata
2.2.4) Benchmarking and testing all slabs allocator
2.2.5) Benchmarking and testing radix tree subsystem
## 2.3) Pez physical subsystem initialization
2.3.1) Creating Pez physical allocation plane
2.3.2) Benchmarking and testing Pez physical allocation plane
## 2.4) Pez virtual subsystem initialization
2.4.1) Creating Pez virtual allocation plane for kernel heap
(Pez virtual plane for kernel heap isn't benchmarked and tested because it's exactly the same algorithm and metadatas under the hood)
## 2.5) Kernel heap initialization
2.5.1) Initializing kernel heap structure
2.5.2) For each slab allocator inside kernel heap:
2.5.2.a) Initializing corresponding page block allocator structure
2.5.2.b) Initializing corresponding slab allocator structure
2.5.2.c) Adding first slab and verifying slab metadata
2.5.3) Benchmarking and testing kernel heap

13
docs/shelter/cpu/asmint.md Normal file
View File

@@ -0,0 +1,13 @@
# ASM instructions
## Introduction
While the Shelter kernel is made in C, somes basic CPU functions require using ASM instructions. For that, we define wrappers that can be used anywhere in the kernel. These wrappers are defined inside the header file `shelter/lib/include/cpu/asm.h` which act as the only place in the entire kernel where the `__asm__` should be used. The subsystem prefix is `sh_asm_`.
## Overview
Here is a list of all the wrappers available. They are all defined as `static inline` function:
- `inb`: take a `sh_uint16` in input for specifying the port and return the inputed byte
- `outb`: take a `sh_uint16` for specifying the port and a `sh_uint8` for specifying the byte to output
- `rdtsc`: take no arguments and return the current TSC as a `sh_uint64`
- `invlpg`: take a `void *` as an address and return nothing

11
docs/shelter/cpu/cpuabstract.md Normal file
View File

@@ -0,0 +1,11 @@
# CPU Abstraction
## Introduction
This component of the Shelter kernel allow for basic CPU functions abstractions, like ASM instruction, serial output and TSC utilities.
## Summary
1) [ASM instructions](asmint.md)
2) [Serial outputing API](serial.md)
3) [TSC API](tsc.md)

12
docs/shelter/cpu/serial.md Normal file
View File

@@ -0,0 +1,12 @@
# Serial outputing API
## Introduction
Shelter provide an abstraction around the serial port for outputing on it. This should act as the central point for outputing on the serial. The API is defined in `shelter/lib/include/cpu/serial.h` and implemented in `shelter/lib/src/cpu/serial.c`. This API obey the `serial_port_disabled` killswitch defined in kernel boot configuration. The subsystem prefix is `sh_serial_`.
## Overview
The API define the following elements:
- `sh_serial_load_serial_port_setting`: should be used only once at the start of the boot process to load the serial port setting using the argument `is_disabled` which is a `sh_bool`
- `sh_serial_send_byte`: safely (wait for previous byte to finish being send) send a byte to the serial port
The default and only serial port available is COM1.

20
docs/shelter/cpu/tsc.md Normal file
View File

@@ -0,0 +1,20 @@
# TSC API
## Introduction
In order to be able to measure and approximate time as soon as the kernel start the boot process, the TSC API is implemented in a volontary minimal way. The TSC API is defined inside `shelter/lib/include/cpu/tsc.h` and implemented inside `shelter/lib/src/cpu/tsc.c`. The API prefix is `sh_tsc_`.
## Overview
The TSC API being intented for measuring time during the boot process, the provided features are extremely basic. In order for any TSC value to start at 0, we define two concept:
- `kernel_init_tsc`: a TSC value initialized at the very start of the kernel boot process
- `kernel_current_tsc`: a TSC value which is the result of `kernel_init_tsc` substracted to the value returned by `sh_asm_rdtsc()`
## API content
The API define the following elements:
- `sh_tsc_TSC_VALUE`: a value of the TSC register, which is a wrapper of a `sh_uint64`
- `sh_tsc_read_value()`: an inline function reading the TSC register using `sh_asm_rdtsc()`, return a `sh_tsc_TSC_VALUE`
- `sh_tsc_init_tsc()`: a function initializing `kernel_init_tsc`. This function should only be called once as soon as the kernel start. Return a `SH_STATUS`
- `sh_tsc_get_kernel_init_tsc()`: return `kernel_init_tsc` under a `sh_tsc_TSC_VALUE`
- `sh_tsc_get_kernel_current_tsc()`: return the result of `sh_asm_rdtsc()` less `kernel_init_tsc`, under a `sh_tsc_TSC_VALUE`

35
docs/shelter/index.md Normal file
View File

@@ -0,0 +1,35 @@
# Shelter Docs
## Introduction
Shelter is an hybrid kernel developped alongside and for the Vystem project.
## Principles
Shelter stricly follow these principles:
- absolutely no external code inside it. All the code that run in ring 0 should be sovereign
- absolutely no UNIX dependency on design philosophy. Shelter follow his own rules for kernel design (but doesn't forbid itself to reuse somes UNIX concepts), even if it mean spending years designing, refining and standardizing them
- a very strict boot process that ensure that everything goes well during the system boot
- fully made in C
- focuses on auditability, self-testing and benchmarking, and determinism
- use his own algorithms for various very important components of every kernel (mainly memory management, scheduling, etc), except on cryptographic algorithms, which follow a very strict integration processus
The Shelter kernel is currently in very early developpement stage and isn't suitable at all for every-day usage.
## Summary
- Key principles to know in order to boot Shelter
- [Kernel bootstrapping using keycard](keycard.md)
- [Introduction to Pages Table Pool](ptp.md)
- [Shelter kernel boot configuration](bootconfig.md)
- [Shelter kernel memory map layout](memmap.md)
- [Shelter boot contract](bootcontract.md)
- [Shelter Boot Process](bootprocess.md)
- [Naming scheme](naming.md)
- Detailled components
- [CPU abstraction](cpu/cpuabstract.md)
- [Shelter standard library](std/std.md)
- [Kernel-specific APIs](kernel/kernel.md)
- [Memory subsystem](memory/index.md)
- [Test-and-benchmark framework](tab.md)
- [Benchmark results](bench.md)

3
docs/shelter/kernel/config.md Normal file
View File

@@ -0,0 +1,3 @@
# Config parsing
In Shelter, the parsing of the config is made by the `SH_STATUS sh_conf_get_boot_config(sh_conf_BOOT_CONFIG **config)` function. It allow for quick and secure configuration parsing. It create a `sh_conf_BOOT_CONFIG` object. These elements are defined inside `shelter/lib/include/kernel/conf.h` and implemented inside `shelter/lib/src/kernel/conf.c`. For the full configuration documentation, please see [boot configuration documentation](../bootconfig.md).

11
docs/shelter/kernel/kernel.md Normal file
View File

@@ -0,0 +1,11 @@
# Kernel-specific APIs
## Introduction
The Shelter kernel define his own specific APIs like logging and configuration parsing.
## Summary
1) [Log API](log.md)
2) [Config parsing](config.md)
3) [Tests utilities](testutils.md)

137
docs/shelter/kernel/log.md Normal file
View File

@@ -0,0 +1,137 @@
# Log API
## Introduction
The logging API is one of the most important of the entire kernel: it allow it to output informations with various level of importance and various sources.
## Output methods
The logging API can log informations with two differents methods:
- serial output: use the serial API to output in real time all the provided informations. Can be disabled by setting it to true using the `sh_log_load_serial_setting(sh_bool is_disabled)` function. The serial API being disabled, this method can also be disabled by that.
- ring buffer output: use the ring buffer API to store all the provided informations into a ring buffer, mapped by the bootloader. Can be disabled by setting the `logging_ring_size` boot configuration key to 0
The following functions provide way to set and get those settings:
- `void sh_log_load_serial_setting(sh_bool is_disabled)`: specify if serial log outputting should be disabled
- `sh_bool sh_log_get_serial_setting()` return the activation status of serial log outputting (`SH_TRUE` mean disabled)
- `void sh_log_load_logging_ring_size(sh_uint16 pages_count)`: set logging ring size
- `sh_uint32 sh_log_get_logging_ring_size()`: return logging ring size
- `sh_uint64 sh_log_get_total_bytes_written()`: return total amount of bytes written inside logging ring
## Output primitives
The following functions doesn't care about log level or log source and are specialized into specific data type:
- `void sh_log_byte(sh_uint8 byte)`: log a byte, basic entry point of the entire logging system
- `SH_STATUS sh_log_string(const char* str)`: log a table of characters
Here are the numbers logging functions under their naming scheme:
```
sh_log_(u)int[8/16/32/64](_hex)
```
Basically, there is 12 functions that can output numbers, in various sizes (8 to 64 bits), signed or unsigned, in decimal or hexadecimal. The hexadecimal variant doesn't exist for signed integers. Examples: `sh_log_uint8`, `sh_log_int16`, `sh_log_uint64_hex`...
There are two mores functions:
- `SH_STATUS sh_log_uint64_hex_fixed(sh_uint64 n)`: log a `sh_uint64` in hexadecimal without removing useless zeros, useful for logging pointers and address
- `SH_STATUS sh_log_double(double value)`: log a double encoded in decimal
## Output channels and log level
The Shelter logging API provide higher level functions to output on specific log channels. Here are all the log channels:
- `SH_LOG_DEBUG`: log level 0
- `SH_LOG_LOG`: log level 1
- `SH_LOG_WARNING`: log level 2
- `SH_LOG_ERROR`: log level 3
- `SH_LOG_CRITICAL`: log level 4
- `SH_LOG_FATAL`: log level 5
- `SH_LOG_TEST`: log level 6
Each log channel has a log level. The log level is set by the `log_level` boot configuration key. The log API will only output log messages with log level equal or higher to current log level. This restriction doesn't apply to log channel `SH_LOG_TEST`.
The `sh_log_OUTPUT_TYPE` type is used to store output type by log level. It's a wrapper of `sh_uint8`. The `sh_log_output_type_valid(sh_log_OUTPUT_TYPE t)` can be used to check if a log level is valid.
Two functions allow to set and get the current log level:
- `SH_STATUS sh_log_load_log_level(sh_uint8 log_level)`
- `sh_uint8 sh_log_get_log_level()`
## Output sources
The Shelter logging API also require a source in order to log informations. Here are all the possible sources:
- `SH_LOG_SOURCE_MAIN`
- `SH_LOG_SOURCE_CONF`
- `SH_LOG_SOURCE_PAGE`
- `SH_LOG_SOURCE_SLAB`
- `SH_LOG_SOURCE_TEST`
- `SH_LOG_SOURCE_PEZ`
- `SH_LOG_SOURCE_PBA`
- `SH_LOG_SOURCE_HEAP`
- `SH_LOG_SOURCE_STD`
The `sh_log_OUTPUT_SOURCE` type is used to store log source. It's a wrapper of `sh_uint16`. The `sh_log_output_source_valid(sh_log_OUTPUT_SOURCE s)` can be used to check if a log level is valid.
## Output payloads
In order to compile all of those informations into one payload, the logging API provide this structure :
``` C
typedef struct {
sh_log_OUTPUT_TYPE output_type;
sh_log_OUTPUT_SOURCE output_source;
sh_tsc_TSC_VALUE tsc_value;
const char* message_pointer;
} sh_log_OUTPUT_PAYLOAD;
```
This allow to contain the message content, the output channel, the output source and the TSC value of any log message.
There is two functions that can manipulate output payloads:
- `SH_STATUS sh_log_payload(sh_log_OUTPUT_PAYLOAD *payload)`: output a payload
- `SH_STATUS sh_log_payload_format(sh_log_OUTPUT_PAYLOAD *payload,va_list args)`: format then output a payload, not directly callable, see below for more informations
## Higher levels logging functions
The following functions will automatically create the payload depending on the parameters they receive.
### Logging with automatic line break
These functions are named like this:
```
sh_log_[test/debug/log/warning/error/critical/fatal]
```
They all need two arguments in the following order (except `sh_log_test` which only need the first one):
- `const char* str`: the text to log
- `sh_log_OUTPUT_SOURCE source`: the source of the log message
These functions will automatically add the `\n` characters at the end of each log message.
### Logging without automatic line break
These functions are named like this:
```
sh_log_l[test/debug/log/warning/error/critical/fatal]
```
They all need two arguments in the following order (except `sh_log_ltest` which only need the first one):
- `const char* str`: the text to log
- `sh_log_OUTPUT_SOURCE source`: the source of the log message
### Logging with formating
In order to allow for formatting log messages, the log API provide functions named like this:
```
sh_log_f[test/debug/log/warning/error/critical/fatal]
```
They all need at least two arguments in the following order (except `sh_log_ltest` which only need the second one):
- `sh_log_OUTPUT_SOURCE source`: the source of the log message
- `const char* format`: the format text
- all the others argument after that are used as content for formatting the log message
All of these functions use `sh_log_payload_format()` under the hood which itself use `SH_STATUS sh_log_format(const char* format,va_list args)` to format log messages.
The format syntax is as follow:
- `%s`: insert a string made of `char`, passed as `char*`
- `%d`: insert a double
- `%x`: insert a integer that need to be provided as a `sh_uint64`, format it as hexadecimal without removing useless zeros. Used to log addresses
- `%c`: insert a `char`
- `%%`: insert a `%`
There is also a specific syntax for formatting integers. It's start with the `%` symbol, then need a amount of bytes (can be 1, 2, 4 or 8) and a output format: `u` for unsigned decimal, `s` for signed decimal, `U` for unsigned decimal. Example: `%1s` output a `sh_int8` in decimal, `%2u` output a `sh_uint16` in decimal, `%4U` output a unsigned `sh_uint32` in hexadecimal.

23
docs/shelter/kernel/testutils.md Normal file
View File

@@ -0,0 +1,23 @@
# Tests utilities
## Introduction
To support the test-and-benchmark framework, the Shelter kernel include a common base of utilities that provide to all tests a basic way to store and expose their results. This file only cover the basic utilities usable by all tests, not the detail and typical results of each test. These utilities are defined inside `shelter/lib/include/kernel/tests/test_utils.h` and implemented inside `shelter/lib/src/kernel/tests/test_utils.c`.
## Overview
The test-and-benchmark (TAB) framework is responsible for testing differents subsystems to ensure their stability and performances over differents devices. But it also provides a common base for showing results and others little things.
Firsty, it define internally a buffer of 10000 `sh_tsc_TSC_VALUE` for storing cycles count and compute statistics. The pointer to this buffer is obtainable through the `sh_tsc_TSC_VALUE* sh_test_get_tsc_values_buffer_ptr()` function.
Then, the TAB framework also define the `void sh_test_load_iterations_count(sh_uint64 iterations_num)` function. This function load the iterations count defined by the `bench_iterations` boot configuration key. Some tests doesn't support this feature, so this function will only set iterations count for tests that support custom iterations count.
Finally, each test uses the `SH_STATUS sh_test_compute_print_stats(char* benchname,sh_tsc_TSC_VALUE *tsc_value_array,sh_uint64 array_size)` function. This function will compute various statistics and print them like this:
```
Result for benchmark "<benchmark name>" :
Min: <min time> | Med: <median time> | Avg: <average time> | Max: <max time> | Total : <total time> (TSC)
[P00-P90] <min time>-P90 : [##################--] 90%
[P90-P99] P90-P99 : [#-------------------] 9%
[P99-Max] P99-<max time> : [--------------------] 1%
```
Here, `PXX` where `XX` is a percentage mean a percentile.

21
docs/shelter/keycard.md Normal file
View File

@@ -0,0 +1,21 @@
# Kernel bootstrapping using keycard
## Introduction
Shelter being a kernel that need to start with pagging enabled, it need to have a intermediary program to safely jump to it: Keycard. Keycard and Shelter have been designed and thinked for x86-64 only.
## Overview
Keycard is a very simple assembly program with the following role in the following order:
- loading the kernel page table and enabling pagging
- loading and aligning the kernel stack
- jumping to kernel entry point
In order to achieve that, it need to be identity mapped into virtual lower half into the kernel page table, using a physical address provided by UEFI firmware. The bootloader map it into a page (keycard compiled code fit into a single page) with allowed execution but read-only. It also have a kind of 'ABI' which specify which value to put into which register before jumping to Keycard:
- `rax`: should contain the stack top virtual address (VA)
- `rbx`: should contain the physical adrdess of the page table root
- `rcx`: should contain the VA of the kernel entry point
Due to security concerns, the `keycard.bin` file contain the compiled code of Keycard and is stored inside the InitFS with the kernel VYX executable.
The Keycard source code is stored inside `Blastproof/src/keycard.asm`.

37
docs/shelter/memmap.md Normal file
View File

@@ -0,0 +1,37 @@
# Shelter kernel memory map layout
## Introduction
In order to transmit efficiently the EFI memory map to the kernel, Shelter have his own memory map layout that need to be created from the EFI memory map in order for Shelter to understand it. For details about mapping the Shelter memory map inside the kernel virtual memory space, please see [boot contract docs](bootcontract.md).
## Header
The Shelter memory map is first assigned a buffer of 64 kilobytes or 16 pages mapped with non-execution and read only permissions. Each memory map has an header described by the following C structure (sourced from `shelter/lib/include/memory/page.h`)
``` C
#pragma pack(1)
typedef struct {
sh_uint8 sig_start[8];
sh_uint64 entry_count;
sh_uint64 entry_size;
sh_uint8 mmap_syntax_version;
} sh_page_MEMORY_MAP_HEADER;
#pragma pack()
```
Each field is packed inside the struct. The signature must be `SheMmapB`. The number of entry is indicated inside the `entry_count` field. The size of an entry is indicated inside the `entry_size` field. Both of these fields are little endian 8 bytes unsigned integer. The `mmap_syntax_version` is always set to `0x01`, as the current version of the Shelter memory map syntax. The kernel also compute the total size of the memory map (header + all entries) and raise an error if it's above the size of the memory map buffer.
## Entries
After the header, there are entries that describe which part of the physical memory is currently unusable (due to various factors like ACPI table, etc). A entry is described by the following C structure:
``` C
#pragma pack(1)
typedef struct {
sh_uint32 type;
sh_uint64 physical_start;
sh_uint64 pages_count;
sh_uint64 attributes;
} sh_page_MEMORY_MAP_ENTRY;
#pragma pack()
```
Each field is packed inside the struct. The `type` field is a 4 bytes unsigned integer that give the type of the region, as the EFI specification specify it. The `physical_start` field is a 8 bytes unsigned integer that contain the physical address of the start of the region. The `pages_count` field is a 8 bytes unsigned integer that contain the number of 4 kilobytes pages that constitute the region. The `attributes` field contain any other attributes provided by the UEFI firmware. All fields are in little endian.

45
docs/shelter/memory/heap.md Normal file
View File

@@ -0,0 +1,45 @@
# Kernel heap manager
## Introduction
In order to allocate small-sized object or medium-sized buffers (approximately between one and one hundred pages), the lower half have been designed for hosting the heap of the kernel. This is an early design subject to changes for perfomances and/or robustess in the coming updates. It's defined inside `shelter/lib/include/memory/heap.h` and implemented inside `shelter/lib/src/memory/heap.c`.
## Overview
The kernel heap manager is based on a dual allocation strategy:
- small objects, less or equal to 1024 bytes goes into slab allocators
- medium buffers, greater than 1024 bytes, are allocated using the Pez allocator for physical and virtual planes management
The heap metadatas are stored inside the `sh_heap_KERNEL_HEAP` structure. The `alloc_size_tree` field is a radix tree mapping virtual pages offset of any pages mapping with their size in pages. This allow for quick retrieval of any pages allocations on the heap.
## Slab allocators
These are described into [generic slab allocator documentation](slabs.md#generic-slab-allocators). Each one is setted up with a PBA containing 12 terabytes of virtual memory
Here are a table describing the virtual pages ranges of all allocators:
Object size range (bytes) | Allocator level | Virtual pages range start address
--- | --- | ---
1-8 | 0 | `0x0000200000000000`
9-16 | 1 | `0x00002C0000000000`
17-32 | 2 | `0x0000380000000000`
33-64 | 3 | `0x0000440000000000`
65-128 | 4 | `0x0000500000000000`
129-256 | 5 | `0x00005C0000000000`
257-512 | 6 | `0x0000680000000000`
513-1024 | 7 | `0x0000740000000000`
## Pages allocations
If the needed amount of bytes is greater than 1024 bytes, pages are allocated on the heap. The number of pages allocated is the nearest amount that can satisfy the demand. The pages are allocated inside a virtual pages range starting at `0x0000100000000000` and contain 16 terabytes of virtual memory.
## API
The heap manager provide the following API, that should only be called by the abstraction provided by `sh_malloc`, except the first three one:
- `sh_heap_KERNEL_HEAP *sh_heap_get_default_heap()`: return a pointer to the kernel heap structure
- `void sh_heap_load_default_heap(sh_heap_KERNEL_HEAP *heap)`: set the pointer to the kernel heap structure
- `SH_STATUS sh_heap_init_heap(sh_pez_PHYSICAL_PLANE *phys_plane,sh_pez_VIRTUAL_PLANE *virt_plane,sh_page_PAGE_TABLE_POOL *kernel_ptp,sh_heap_KERNEL_HEAP *kernel_heap)`: initialize the heap structure, the calling entity need to setup and place the generic slab allocators pointers manually inside the returned structure
- `SH_STATUS sh_heap_allocate_pages(sh_uint32 pages_count,sh_page_VIRTUAL_ADDRESS *address)`: allocate a certain amount of pages, provided by the Pez physical backend and mapped on the heap pages allocation range
- `SH_STATUS sh_heap_free_pages(sh_page_VIRTUAL_ADDRESS va)`: free an allocated region of pages from the heap. Return the pages to Pez physical backend
- `SH_STATUS sh_heap_allocate_object(sh_uint32 size_bytes,sh_page_VIRTUAL_ADDRESS *address)`: allocate a certain object based on his size, automatically uses the most adapted generic slab allocator. Will refuse any size superior to 1024 bytes
- `SH_STATUS sh_heap_free_object(sh_page_VIRTUAL_ADDRESS va)`: free a certain object by providing his VA. Automatically free from the previously used slab allocator. Will refuse any VA not in the range of the object allocation range

16
docs/shelter/memory/index.md Normal file
View File

@@ -0,0 +1,16 @@
# Memory subsystem
## Introduction
The memory subsystem is responsible for handling tasks like physical pages allocations, virtual memory management, initial memory map analysis, pages mapping and unmapping and kernel heap management.
## Summary
1) [Page subsystem](page.md)
2) [Virtual memory layout](vmemlayout.md)
3) [Ring buffer](ring.md)
4) [Pages block allocator](pba.md)
5) [Slabs allocator](slabs.md)
6) [Radix trees subsystem](radix.md)
7) [Pez plane manager](pez.md)
8) [Kernel heap manager](heap.md)

90
docs/shelter/memory/page.md Normal file
View File

@@ -0,0 +1,90 @@
# Page subsystem
## Introduction
The Page subsystem is a part of the memory subsystem responsible for managing initial memory map analysis, physical pages bitmap, pages tables pool management, mapping and unmapping pages as well as memory allocations before Pez initialization and memory statistics. It's defined inside `shelter/lib/include/memory/page.h` and implemented inside `shelter/lib/src/memory/page.c`
## Macros and types
The Page header defines a few usefuls macros like:
- the start and end of the kernel big allocations virtual range
- the size of a page
- the max memory count in bytes as well as the max physical pages count
- the standard size of a pages table pool pages range
- all the types of EFI memory type
- the null value of physical and virtual addresses
- the flags for pages table entry
The Page header also defines the following types:
- `sh_page_MEMORY_TYPE`: a wrapper of `sh_uint32`
- `sh_page_PHYSICAL_ADDRESS` and `sh_page_VIRTUAL_ADDRESS`: wrappers of `sh_uint64`
## Structures
The Page header defines the following structure:
- `sh_page_MEMORY_MAP_ENTRY` and `sh_page_MEMORY_MAP_HEADER`: internal structures used for memory map analysis
- `sh_page_PAGE_TABLE_POOL`: structure for pages table pool, containing the physical and virtual address of a pages table pool as well as the internal bitmap for the internal allocator
- `sh_page_MEM_STATS`: a structure filled by a specific function to get statistics on physical memory using physical pages bitmap
## Pages table pool
It's recommended to read [pages table pool documentation](../ptp.md) before reading this.
The Page subsystem provide the following functions regarding PTP management:
- `SH_STATUS sh_page_load_boot_ptp_va(sh_page_VIRTUAL_ADDRESS pt_pool_va)`: load the virtual address of the boot PTP provided by bootloader. This VA is the address of the root PML4, not the address of the PTP structure. Should only be used once
- `sh_page_VIRTUAL_ADDRESS sh_page_get_boot_ptp_va()`: return the pointer to root PM4 stored inside boot PTP
- `SH_STATUS sh_page_init_ptp(sh_page_PHYSICAL_ADDRESS ptp_pa,sh_page_VIRTUAL_ADDRESS ptp_va,sh_uint64 initial_fill_level,sh_page_PAGE_TABLE_POOL *page_table_pool)`: initialize the boot PTP structure with the fill level of the PTP, provided by bootloader. Doesn't allocate pages for a new PTP but it use the pages already allocated for the boot PTP. Should only be used once for boot PTP
- `SH_STATUS sh_page_dump_ptp_bitmap(sh_page_PAGE_TABLE_POOL *ptp)`: dump the PTP bitmap on the log output, intended for debug purposes
- `sh_page_PHYSICAL_ADDRESS sh_page_ptp_alloc_one_page(sh_page_PAGE_TABLE_POOL *pt_pool)`: allocate one page from the PTP internal bitmap allocator
- `static inline sh_uint64 *sh_page_ptp_pa_to_va(sh_page_PAGE_TABLE_POOL *ptp,sh_uint64 pa)`: convert a PA from the PTP pages range to a usable VA using the current PTP
- `SH_STATUS sh_page_ptp_va_to_pa(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_page_PHYSICAL_ADDRESS *pa)`: return equivalent physical address by searching inside provided PTP
## Memory map analysis
The Page subsystem is responsible for the initial memory map analysis. It provide the following functions:
- `SH_STATUS sh_page_copy_memory_map()`: copy the memory map from her original VA to a dedicated buffer
- `SH_STATUS sh_page_check_memory_map()`: check for memory map signature and detect eventual buffers overflow
- `SH_STATUS sh_page_analyse_memory_map(sh_page_PAGE_TABLE_POOL *ptp)`: analyse memory map, initialize physical pages bitmap by allocating into a free area of the physical memory layout. This allow for dynamic sizing for physical pages bitmap.
All three functions should be run in order and only once.
## Physical pages bitmap
In order to ensure a reliable source of thruth and early allocations, the Page subsystem manage a physical pages bitmap where one bit on mean one page occupied.
The Page subsystem provides the following functions regarding physical pages bitmap:
- `SH_STATUS sh_page_set_pages_range_bitmap(sh_uint8 *bitmap,sh_uint64 page_count_in_bitmap,sh_uint64 page_index,sh_uint64 page_count,sh_bool state)`: allow for manipulation of any bitmap, but should only be used for physical pages bitmap
- `static inline sh_bool sh_page_is_allocated(sh_uint8 *bitmap,sh_uint64 page_index)`: return the status of a specific page inside the bitmap
- `sh_page_VIRTUAL_ADDRESS sh_page_get_physical_bitmap_ptr()`: return physical pages bitmap pointer
## Pages mapping and unmapping
The Page subsystem provide the following functions regarding pages mapping:
- `SH_STATUS sh_page_map_one_page_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_page_PHYSICAL_ADDRESS pa,sh_uint64 flags)`: map one page inside the pages table contained in the provided PTP
- `SH_STATUS sh_page_is_va_mapped_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va)`: return `SH_STATUS_VA_NOT_MAPPED` or `SH_STATUS_VA_MAPPED`, which aren't errors, according to page mapping state
- `SH_STATUS sh_page_is_va_range_mapped_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_uint64 size_bytes)`: return `SH_STATUS_VA_NOT_MAPPED`, `SH_STATUS_VA_FULLY_MAPPED` or `SH_STATUS_VA_PARTIALLY_MAPPED` according to pages range mapping state
- `SH_STATUS sh_page_map_contiguous_pages_range_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_page_PHYSICAL_ADDRESS pa,sh_uint64 flags,sh_uint64 size_bytes)`: map a pages range that has to be contiguous virtually and physically. VAs availability is checked, not physicals pages availability
- `SH_STATUS sh_page_unmap_one_page_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va)`: unmap one page from the pages table contained inside provided PTP
- `SH_STATUS sh_page_unmap_contiguous_pages_range_ptp(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_uint64 size_bytes)`: unmap a pages range from the pages table inside provided PTP. Both physical and virtual ranges have to be contiguous. Virtual pages range is checked, not physical ranges occupation
## Basic memory allocations
The Page subsystem implement basic memory allocation, available before Pez initialization, using a first fit algorithm.
The Page subsystem provides the following functions regarding basic memory allocations:
- `sh_uint64 sh_page_get_one_page_na()`: return the index of the first available page inside physical pages bitmap
- `SH_STATUS sh_page_search_available_va_range(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS range_base,sh_page_VIRTUAL_ADDRESS range_size_bytes,sh_uint64 size_bytes,sh_page_VIRTUAL_ADDRESS *address_found)`: search a free region inside PTP using provided size and search area boundaries provided
- `SH_STATUS sh_page_search_physical_contiguous_block_na(sh_uint64 pages_needed,sh_page_PHYSICAL_ADDRESS *pa)`: search a free region of provided size inside physical pages bitmap
- `SH_STATUS sh_page_alloc_contiguous(sh_page_PAGE_TABLE_POOL *ptp,sh_uint64 size_bytes,sh_page_VIRTUAL_ADDRESS* va)`: allocate a specified amount of pages and return the start VA of the allocated region
- `SH_STATUS sh_page_alloc_contiguous_extended(sh_page_PAGE_TABLE_POOL *ptp,sh_uint64 size_bytes,sh_page_VIRTUAL_ADDRESS* va,DEFAULT sh_uint64 flags,DEFAULT sh_page_VIRTUAL_ADDRESS va_range_start,DEFAULT sh_uint64 va_range_size_bytes)`: allow for extensive allocations parameters for selecting mapping flags and VAs search range
- `SH_STATUS sh_page_unalloc_one_page(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va)`: free one page. Check if page is mapped before hand. PA is calculated from searching into PTP
- `SH_STATUS sh_page_unalloc_contiguous(sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS va,sh_uint64 size_bytes)`: free a contiguous memory region. Check if the entire region is allocated before hand
This role of memory allocations is lost once Pez is initialized.
## Memory statistics
The Page subsystem provide the following functions regarding memory statistics:
- `sh_uint64 sh_page_get_physical_memory_amount_pages()`: return amount of physical memory installed in pages
- `sh_uint64 sh_page_get_physical_memory_amount_bytes()`: return amount of physical memory installed in bytes
- `SH_STATUS sh_page_get_memory_stats(sh_page_MEM_STATS *mem_stats)`: provide a extensive amount of statistics on physical memory.
For a more human-readable output, the function `sh_log_mem_stats` can be used.

19
docs/shelter/memory/pba.md Normal file
View File

@@ -0,0 +1,19 @@
# Pages block allocator
## Introduction
The pages block allocator is a simple configurable bumb allocator designed to be the pages source of slabs allocator inside the kernel. It's defined inside `shelter/lib/include/memory/pba.h` and implemented inside `shelter/lib/src/memory/pba.c`.
## Overview
A pages block allocator is conifigurable with various parameters.
Firstly, it need to know the size of each block that are allocated. This is specified inside the `block_pages` field of the PBA structure, under the `sh_uint64` type.
The `start_va` field, a `sh_page_VIRTUAL_ADDRESS`, specify the starting VA of the virtual pages ranges dedicated to the allocator. It need to be aligned on a boundary of a region `block_pages` pages. The `total_pages` field, a `sh_uint64`, specify the size of the decicated virtual pages range, in pages. It need to be a multiple of `block_pages`.
Finally, the `block_count` field, a `sh_uint64`, store the amount of blocks already allocated. The `max_blocks` field, a `sh_uint64`, is computed at initialization and allow to know instantly if the PBA is full, which happen when the dedicated virtual pages range is fully mapped.
All those fields are stored inside the `sh_pba_PAGE_BLOCK_ALLOCATOR` structure. This structure can be manipulated by the following functions:
- `SH_STATUS sh_pba_init(sh_pba_PAGE_BLOCK_ALLOCATOR *pba,sh_page_VIRTUAL_ADDRESS start_va,sh_uint64 area_pages_amount,sh_uint64 block_pages)`: initialize a PBA, using provided parameters. It ensure the previously specified constraints are met with the provided parameters
- `SH_STATUS sh_pba_alloc(sh_pba_PAGE_BLOCK_ALLOCATOR *pba,sh_page_PAGE_TABLE_POOL *ptp,sh_page_VIRTUAL_ADDRESS *ptr)`: allocate a block from the PBA. Take care of the physical region search (using Page subsystem or Pez if it's initialized), VA computation (using provided virtual pages range) and pages mapping, as well as physical pages bitmap update if Pez isn't initialized.

86
docs/shelter/memory/pez.md Normal file
View File

@@ -0,0 +1,86 @@
# Pez plane manager
## Introduction
Pez is the allocator that manages physical pages allocation and virtual pages ranges allocation. It's a custom design, (normally) never seen before. It's defined into `shelter/lib/include/memory/pez/pez.h` and implemented into `shelter/lib/src/memory/pez/pez.c`.
## Planes
Pez manages memory using planes. It can be as granular as a 4KB pages. Since Pez uses `sh_uint32` for representating pages index and regions sizes, a plane can technically manages up to 16 terabytes of memory.
In the implementation, Pez separate the physical plane from virtual planes. They work the same in their logic and algorithm but use differents source of truth and the virtual plane can provide an offset to each address it allocate.
The physical plane uses the physical pages bitmap that the Page subsystem maintain as source of truth. While the source of truth for virtual planes are the corresponding pages mapping into the virtual pages range the virtual plane manages, for the moment, only empty (not allocated at all) virtual planes can be created.
## Metadatas
### Regions objects
The Pez allocator uses regions objects allocated from their dedicated (physical or virtual) slab allocators.
The physical version is:
``` C
#pragma pack(1)
typedef struct {
sh_uint32 start_page_index;
sh_uint32 region_size_pages;
sh_uint8 next_region_index[3];
sh_uint8 flags;
} sh_pez_REGION_PHYSICAL_OBJECT;
#pragma pack()
```
The virtual version is:
``` C
#pragma pack(1)
typedef struct {
sh_uint32 start_page_index;
sh_uint32 region_size_pages;
sh_uint32 next_region_index;
} sh_pez_REGION_VIRTUAL_OBJECT;
#pragma pack()
```
The `start_page_index` field is the index of the first page of the region, starting at 0 from the start of the plane. The `region_size_pages` field is the size of the region in pages. The `next_region_index` field is the index of the next region in the size list. It's one byte longer on virtual region object to account for the 29 bits index. The `flags` field, while not being used in this implementation, can be used to store flags.
These regions objects represente free regions that are ready to be allocated.
### Size lists radix tree
Each plane maintain lists of free regions by their sizes. These are call size lists. They are linked through the `next_region_index` field, using their object index inside their respective slab allocator. These lists only contain regions of exact size. To signal the end of a size list, the last element in the list have their `next_region_index` field set to 0. This prevent the first slot of any of the slab inside the slab allocator to be allocated.
Each plane maintain a radix tree that link sizes of free regions (in page count) to indexes of the first region in this size list. This radix tree is called the size lists radix tree and is 8 levels deep.
This architecture allow for two allocations methods that are stricly time-constant or nearly time-constant:
Each time the allocator need to allocate a region, it search for this size into the corresponding size list. Two possibles results:
- if a free region of exactly this size is found, it's popped of the size list and allocated
- if no free region of exactly this size is found, the algorithm search for another region immediatly above in size of the requested size using the backtracking algorithm. This free region is popped of the size list and splitted in two. The first part is allocated and the second part is inserted into the corresponding size list.
This allocation scheme allow for zero internal fragmentation at the granularity of 4KB pages.
### Boundary radix tree
Each plane maintain what we call a boundary radix tree. This radix tree map the start and end pages index in the plane of each free regions to a boundary entry. For one-page free region, there is only one boundary. The two boundaries for free regions with more than one page are strictly identical.
A boundary is structured like this:
- The lower 32 bits of the boundary are the region object index of the region to which the boundary belong
- The upper 32 bits of the boundary are the previous region object index in the size list of the region to which the boundary belong
This radix tree allow for two things:
- constant-time passive fusion of free regions with deallocated occupied regions, at each free of occupied region. The external fragmentation is repaired as soon as the allocated region that cause it is deallocated.
- instant element suppression of any free region in any size lists
## API
The Pez allocator implementation inside the Shelter kernel provide the following functions:
- `void sh_pez_set_available()`: executed once to signal that Pez has taken over the role of Page for allocating pages
- `sh_bool sh_pez_is_available()`: return the current status of availability of Pez
- `sh_pez_PHYSICAL_PLANE* sh_pez_get_reference_phys_plane()`: return the reference physical plane that is used to manage physical memory
- `SH_STATUS sh_pez_init_physical_plane(sh_uint8 *physical_bitmap,sh_uint64 physical_page_count,sh_slab_reg_phys_SLAB_ALLOCATOR *slab_reg_phys,struct sh_slab_radix_node_SLAB_ALLOCATOR *slab_radix_node,sh_page_PAGE_TABLE_POOL *kernel_ptp,sh_pez_PHYSICAL_PLANE *phys_plane)`: initialize the physical plane by scanning the physical pages bitmap
- `SH_STATUS sh_pez_alloc_physical_pages(sh_pez_PHYSICAL_PLANE *phys_plane,sh_uint32 pages_count,sh_page_PHYSICAL_ADDRESS *address)`: allocate pages range from the physical plane. Doesn't map the pages at all
- `SH_STATUS sh_pez_free_physical_pages(sh_pez_PHYSICAL_PLANE *phys_plane,sh_page_PHYSICAL_ADDRESS *address,sh_uint32 pages_count)`: free a pages range and return them to the physical plane. Doesn't unmap the pages at all
- `SH_STATUS sh_pez_debug_physical(sh_pez_PHYSICAL_PLANE *phys_plane)`: print debugging information about Pez physical plane, intented for debug
- `SH_STATUS sh_pez_init_virtual_plane(sh_page_VIRTUAL_ADDRESS plane_offset,sh_slab_reg_virt_SLAB_ALLOCATOR *slab_reg_virt,struct sh_slab_radix_node_SLAB_ALLOCATOR *slab_radix_node,sh_page_PAGE_TABLE_POOL *kernel_ptp,sh_page_PAGE_TABLE_POOL *reference_ptp,sh_pez_VIRTUAL_PLANE *virt_plane)`: initialize an empty virtual plane
- `SH_STATUS sh_pez_alloc_virtual_pages(sh_pez_VIRTUAL_PLANE *virt_plane,sh_uint32 pages_count,sh_page_VIRTUAL_ADDRESS *address)`: allocate virtual pages range from the provided virtual plane
- `SH_STATUS sh_pez_free_virtual_pages(sh_pez_VIRTUAL_PLANE *virt_plane,sh_page_VIRTUAL_ADDRESS *address,sh_uint32 pages_count)`: free virtual pages range and return them to the provided virtual plane

37
docs/shelter/memory/radix.md Normal file
View File

@@ -0,0 +1,37 @@
# Radix trees subsystem
## Introduction
The memory subsystem require a capable radix trees to unleash the full power of the Pez allocator. It's defined inside `shelter/lib/include/memory/pez/radix.h` and implemented inside `shelter/lib/src/memory/pez/radix.c`.
## Overview
In Shelter, the radix nodes are 128 bytes wide, capable of storing 16 `sh_uint64` values. This mean all radix trees have a 16 fanout, they filter 4 bits per level, and require 16 levels for 64 bits keys and 8 levels for 32 bits keys.
The node is defined by this sructure:
``` C
typedef struct {
sh_page_VIRTUAL_ADDRESS ptr[16];
} sh_radix_NODE;
```
A radix tree is defined into the following structure:
``` C
typedef struct {
sh_radix_NODE *root_node;
sh_uint8 depth;
} sh_radix_TREE;
```
The API for modifying nodes is as follow:
- `SH_STATUS sh_radix_node_read_value(sh_radix_NODE *node,sh_uint8 index,sh_page_VIRTUAL_ADDRESS* value)`: read a value from the node at the following index
- `SH_STATUS sh_radix_node_set_value(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_radix_NODE *node,sh_uint8 index,sh_page_VIRTUAL_ADDRESS value)`: set the value inside the node at the provided node
The API for manipulating radix trees is as follow:
- `SH_STATUS sh_radix_tree_init(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_radix_TREE *tree,sh_uint8 depth)`: initialize a radix tree, fail if depth is greater than 16. Allocate the root node
- `SH_STATUS sh_radix_tree_get_value(sh_radix_TREE *tree,sh_uint64 key,sh_page_VIRTUAL_ADDRESS *value)`: search in a straight line inside the tree to provide the associated value of a key. Stop and return `SH_STATUS_NOT_FOUND` as soon as it hit an empty pointer where there should be one
- `SH_STATUS sh_radix_tree_insert_value(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_radix_TREE *tree,sh_uint64 key,sh_page_VIRTUAL_ADDRESS value)`: insert a value associated with a key inside the tree. Can allocates new nodes and overwrite previously setted value.
- `SH_STATUS sh_radix_tree_delete_value(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_radix_TREE *tree,sh_uint64 key)`: delete a vale and deallocates all nodes (including leaf) that form a path to the leaf if the deletion make them empty
- `SH_STATUS sh_radix_tree_search_smallest_min_bound(struct sh_slab_radix_node_SLAB_ALLOCATOR *alloc,sh_radix_TREE *tree,sh_uint64 lower_bound_key,sh_page_VIRTUAL_ADDRESS *value)`: backtracking algorithm used to quickly search the value with a key equal or greater than the provided key. Can't allocate new nodes
The radix trees subsystem uses the same slab allocator for all radix trees.

11
docs/shelter/memory/ring.md Normal file
View File

@@ -0,0 +1,11 @@
# Ring buffer
## Introduction
The memory subsystem provide a simple ring buffer API, mainly used by the log API. This implementation doesn't allow for automatic structure initialization with allocation. The structure has to be manually created for it to work. This ring buffer implementation is defined inside `shelter/lib/include/memory/ring.h` and implemented inside `shelter/lib/src/memory/ring.c`
## Overview
The main structure for a ring buffer is `sh_ring_RING_BUFFER_HEADER`. The function provided by the API are volontary very simple for the moment:
- `SH_STATUS sh_ring_write_byte(sh_ring_RING_BUFFER_HEADER *ring_buffer,sh_uint8 byte)`: write a byte inside provided ring buffer
- `SH_STATUS sh_ring_write_string(sh_ring_RING_BUFFER_HEADER *ring_buffer,char *string)`: write a null-terminated string inside provided ring buffer

91
docs/shelter/memory/slabs.md Normal file
View File

@@ -0,0 +1,91 @@
# Slabs allocator
## Introduction
The memory subsystem provides 4 independant but similars slab allocators dedicated to various purposes.
## Regions objects slabs allocator
These slabs allocator are dedicated to the storage of region objects. These objects are 12 bytes in size. There is two slab allocators that store region objects:
- `slab_reg_phys`: the slab allocator for physical region objets. It uses 24 bits index, making it able to store up to 16 millions objects. Defined in `shelter/lib/include/memory/slabs/slab_reg_phys.h` and implemented in `shelter/lib/src/memory/slabs/slab_reg_phys.c`
- `slab_reg_virt`: the slab allocator for virtual region objets. It uses 29 bits index, making it able to store up to 536 millions objects. Defined in `shelter/lib/include/memory/slabs/slab_reg_virt.h` and implemented in `shelter/lib/src/memory/slabs/slab_reg_virt.c`
They uses 3 pages per slab, capable of storing 1024 objects per slabs. They store the slab header outside, into a dedicated, pre allocated area. The slabs are allocated, reused, but never deallocated.
The slab allocator uses indexes to identify objects slots. These indexes are what the user is given upon allocation and need to be provided back in order to free objects slots. The indexes are composed of two parts:
- index in the slab: 10 lower bits
- slab index: 14 or 19 upper bits
The structures `sh_slab_reg_phys_SLAB_STRUCT` and `sh_slab_reg_virt_SLAB_STRUCT` define the headers of each slab. The structures `sh_slab_reg_phys_SLAB_ALLOCATOR` and `sh_slab_reg_virt_SLAB_ALLOCATOR` are used to store allocators metadata. The `sh_slab_reg_phys_OBJECT_INDEX` and `sh_slab_reg_virt_OBJECT_INDEX` are both wrapper of `sh_uint32`.
The only difference between these two are the amount of slabs they can store and their dedicated virtual pages ranges.
The API for physical region objects slab allocator is as follow:
- `SH_STATUS sh_slab_reg_phys_alloc_init(sh_slab_reg_phys_SLAB_ALLOCATOR* slab_alloc,sh_page_PAGE_TABLE_POOL *ptp)`: initialize the allocator, allocate all the headers but does not add the first slab
- `SH_STATUS sh_slab_reg_phys_add_slab(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_phys_SLAB_STRUCT** out_slab)`: add a new slab, can allocate from Page or Pez subsystem, do all the mapping itself
- `void* sh_slab_reg_phys_ref_to_ptr(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_slab_reg_phys_OBJECT_INDEX ref)`: transform an index into a pointer to the referenced object
- `SH_STATUS sh_slab_reg_phys_alloc(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_phys_OBJECT_INDEX* out_index)`: allocate a new object and return his index. Can allocate new slabs
- `SH_STATUS sh_slab_reg_phys_dealloc(sh_slab_reg_phys_SLAB_ALLOCATOR* alloc,sh_slab_reg_phys_OBJECT_INDEX index)`: deallocate a object using the provided index. Doesn't delete slabs that are made empty by objects deallocation
The API for virtual region objects slab allocator is as follow:
- `SH_STATUS sh_slab_reg_virt_alloc_init(sh_slab_reg_virt_SLAB_ALLOCATOR* slab_alloc,sh_page_PAGE_TABLE_POOL *ptp)`: initialize the allocator, allocate all the headers but does not add the first slab
- `SH_STATUS sh_slab_reg_virt_add_slab(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_virt_SLAB_STRUCT** out_slab)`: add a new slab, can allocate from Page or Pez subsystem, do all the mapping itself
- `void* sh_slab_reg_virt_ref_to_ptr(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_slab_reg_virt_OBJECT_INDEX ref)`: transform an index into a pointer to the referenced object
- `SH_STATUS sh_slab_reg_virt_alloc(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_reg_virt_OBJECT_INDEX* out_index)`: allocate a new object and return his index. Can allocate new slabs
- `SH_STATUS sh_slab_reg_virt_dealloc(sh_slab_reg_virt_SLAB_ALLOCATOR* alloc,sh_slab_reg_virt_OBJECT_INDEX index)`: deallocate a object using the provided index. Doesn't delete slabs that are made empty by objects deallocation
## PBA-based slab allocators
These slab allocators relies on a PBA to automatically allocate and map slab pages inside their dedicated virtual pages ranges. They doesn't create their own PBA. While this responsability fall on the user, this allow the user to set personnalized virtual pages range, even if they aren't intented to have multiples instances of them at the same time.
These slabs allocator also store the header of each slabs inside the slab, making it possible to allocate slabs headers on the fly. The slab header contain pointers that can form a partial slabs linked list.
Finally, these slab allocators provide infinite slabs without any software limits. They reuse but doesn't deallocate slabs. They also don't use indexes but direct pointers.
### Radix node slab allocator
This allocator use 32 pages slabs, and manage 128 bytes object. The slab contain 1024 objects slots. But due to the space taken by the header, each slab only contain 1006 objects due to the header taking space.
Inside the slab header, the slab allocator also maintain a 16 bit extra object per radix node. It serve at a bitmap for radix nodes.
The `sh_slab_radix_node_SLAB` structure describe the header of each slab, as well as each object slot. The `sh_slab_radix_node_SLAB_ALLOCATOR` structure contain the metadatas of the allocator. The header is 256 bytes wide.
The API for radix nodes slab allocator is as follow:
- `SH_STATUS sh_slab_radix_node_alloc_init(struct sh_slab_radix_node_SLAB_ALLOCATOR* slab_alloc,sh_pba_PAGE_BLOCK_ALLOCATOR *pba)`: initialize the slab allocator, doesn't allocate any slab
- `SH_STATUS sh_slab_radix_node_add_slab(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_radix_node_SLAB** out_slab)`: add a new slab, allocate from the PBA
- `SH_STATUS sh_slab_radix_node_alloc(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_radix_NODE** out_obj)`: allocate a new object and return his pointer. Can allocate new slabs
- `SH_STATUS sh_slab_radix_node_dealloc(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_radix_NODE *object_ptr)`: deallocate a object using the provided pointer. Doesn't delete slabs that are made empty by objects deallocation
- `sh_uint16 *sh_slab_radix_node_get_node_bitmap(struct sh_slab_radix_node_SLAB_ALLOCATOR* alloc,sh_radix_NODE *object_ptr)`: return a pointer to the bitmap of the node
It's defined inside `shelter/lib/include/memory/slabs/slab_radix_node.h` and implemented inside `shelter/lib/src/memory/slabs/slab_radix_node.c`.
### Generic slab allocators
These slab allocators are used to store objects allocated on the heap. The header is always 128 bytes. Each slab contain 512 objects but the actual size of each slab depend on the level of the allocator:
Level | Object size in bytes | Amount of pages per slab | Amount of actual objects per slab
--- | --- | --- | ---
0 | 8 | 1 | 496
1 | 16 | 2 | 504
2 | 32 | 4 | 508
3 | 64 | 8 | 510
4 | 128 | 16 | 511
5 | 256 | 32 | 511
6 | 512 | 64 | 511
7 | 1024 | 128 | 511
The `sh_slab_generic_SLAB` structure describe the header of each slab, as well as each object slot. The `sh_slab_generic_SLAB_ALLOCATOR` structure contain the metadatas of the allocator.
The API for generic slab allocators is as follow:
- `SH_STATUS sh_slab_generic_alloc_init(sh_uint8 level,struct sh_slab_generic_SLAB_ALLOCATOR* slab_alloc,sh_pba_PAGE_BLOCK_ALLOCATOR *pba)`: initialize the slab allocator, doesn't allocate any slab
- `SH_STATUS sh_slab_generic_add_slab(struct sh_slab_generic_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,sh_slab_generic_SLAB** out_slab)`: add a new slab, allocate from the PBA
- `SH_STATUS sh_slab_generic_alloc(struct sh_slab_generic_SLAB_ALLOCATOR* alloc,sh_page_PAGE_TABLE_POOL *ptp,void** out_obj)`: allocate a new object and return his pointer. Can allocate new slabs
- `SH_STATUS sh_slab_generic_dealloc(struct sh_slab_generic_SLAB_ALLOCATOR* alloc,void *object_ptr)`: deallocate a object using the provided pointer. Doesn't delete slabs that are made empty by objects deallocation
It's defined inside `shelter/lib/include/memory/slabs/slab_generic.h` and implemented inside `shelter/lib/src/memory/slabs/slab_generic.c`.
## Notes
The current implementation of all slabs allocator uses a bitmap stored inside the headers to find the first available object slot. It's planned the next update to test the freelist approch. If it's concluant, it will be used as the default approch in the next update.
It's also planned that all slabs allocators uses PBAs in the next update. This switch will not changes the indexes-based approch for region objects slab allocators.

9
docs/shelter/memory/vmemlayout.md Normal file
View File

@@ -0,0 +1,9 @@
# Virtual memory layout
## Introduction
The virtual memory layout is a file `shelter/lib/include/memory/vmem_layout.h` defining which area of virtual memory serve which purposes. It's automatically checked for no overlaps by a python script.
## Overview
Virtual memory regions are defined with macros. Any macro ending with `_VA` will create a new virtual region for the script. The script will then look for the size of this virtual region in another macro that start with the same prefix and end with `_SIZE_BYTES`. If a macro ending with `_VA` doesn't have a corresponding macro ending with `_SIZE_BYTES`, the script will trigger an error and the kernel compilation will fail. If a macro ending with `_SIZE_BYTES` doesn't have a corresponding macro ending with `_VA`, the script will ignore it. The start of each virtual region must be aligned to 4096 bytes and the size must be provided in bytes. Any overlapping virtual region will trigger a compilation error. Consider using this file as the source of trust for everything related to static virtual regions. Any macro that doesn't end with `_VA` or `_SIZE_BYTES` or that doesn't correspong to the behaviour described above will be ignored.

21
docs/shelter/naming.md Normal file
View File

@@ -0,0 +1,21 @@
# Naming scheme
Shelter has a very specific way of organizing subsystems.
First, each part of the kernel (except `main.c` which is responsible for the boot process) is in his own folder. To this date, there is 4 main parts:
- `cpu`: CPU abstractions
- `kernel`: kernel services
- `std`: Shelter standard library
- `memory`: the whole memory subsystem
Then, each of these parts have specific subsystem or API prefix. These prefixes always start by `sh_` and doesn't include the part each subsystem or API is in. For exemple, the serial outputing API prefix is `sh_serial_`.
The `std` part is the only part where, when not precised, the prefix will be shortened to only `sh_`, like `sh_malloc` or `SH_STATUS`.
In all the subsystems and API described in this documentation, the prefix will always be indicated.
These prefixes extand to names of functions, macros and type/structures. Here is the naming style for each of these entity:
- functions: all letters should be lowercase, starting with the prefix in lowercase. Example: `sh_serial_send_byte`
- macros: all letters should be uppercase, starting with the prefix in uppercase. Example: `SH_PAGE_RW`
- types/structure: the name should start with the prefix in lowercase and continue with the type/structure name in uppercase. Example: `sh_page_PAGE_TABLE_POOL`.
Please avoid abreviations/acronyms in the type name, instead define the abreviation/acronym inside the short comment preceding the declaration. Abreviations/acronyms anywhere else is tollerated as long as it's defined somewhere obvious. Don't hesitate to ask if you are not sure.

23
docs/shelter/ptp.md Normal file
View File

@@ -0,0 +1,23 @@
# Introduction to Pages Table Pool
## Introduction
In order to easely manage pages that are used for pages tables (PT), the Vystem project use pages tables pool (PTP). For details about mapping the PTP inside the kernel virtual memory space, please see [boot contract docs](bootcontract.md).
## Overview
In order to map any page into a pages table on x86-64, you need to descend all four layers of the PT. But what happen when you need a new page for an intermediate layer or directly a pages table ? To solve this problem, we have decided to introduce Pages Table Pool (PTP), a very simple way to allocate new pages for pages table.
Each pages table pool is made from a range of pages, contiguous both physically (this is what allow us to instantely know which value to put into a PTE for a intermediate layer) and virtually. The standard size for this range is currently 4096 pages, giving us plenty of rooms for various allocations accross the virtual memory. This size is evolutive and can changed with versions. The PML4 root page is by standard the first page of the PTP, making it suitable to use the starting VA of any PTP (as long as it's aligned on 4096 bytes) as value for the CR3 register.
There is two kinds of PTP:
- modifiable PTP: this is any PTP that is modifiable by the code that run into the virtual layout the PTP describe. To accomplish that, we map the entire pages range of the PTP into itself before putting it into CR3. We use this kind of PTP for the Shelter kernel
- unmodifiable PTP: here, we just don't do the recursive mapping of the pages range of the PTP into itself. This is the kind of PTP that will be used for any userland program
PTP, when created from the bootloader, are mapped within themself read-write and non-execution permissions. There is no primitive to create PTP from the kernel for the moment.
## Allocator
The purpose of any PTP is to have a reserved amount of pages to permit the quick allocation of single page to allow any mapping. There is two kinds of allocator, depending on the needs of the situation:
- bumb/counter allocator: here, we simply increment a counter that represent an index inside the pages range of the PTP, allowing for very quick PTP initialization. Intended for PTP creation and mapping without unmapping by the program creating the PTP. This is the kind of allocator used into the Blastproof bootloader
- bitmap allocator: a bitmap for storing which pages of the pages range is allocated or not. Intended when heavy unmapping is made, to free pages more easely. This is the allocator used in the Shelter Kernel. It can be initialized from the counter value, passed by the bootloader for example. For the moment, automatic pages freeing from the page table isn't implemented.

7
docs/shelter/std/malloc.md Normal file
View File

@@ -0,0 +1,7 @@
# Heap memory allocations
The Shelter standard library provide a single API for memory allocations on the heap. The heap is only initialized at the end of the memory subsystem and can't be used before. It's not adapted for memory mapped I/O or big buffers allocations (larger than one hundrer pages). The heap internal documentation can be found inside the memory subsystem documentation.
The memory allocations API provide the two following functions (defined inside `shelter/lib/include/std/malloc.h` and implemented inside `shelter/lib/src/std/malloc.c`):
- `void* sh_malloc(sh_uint64 size)`: allocate `size` amount of bytes. Return `SH_NULLPTR` if an error occured. The heap internal will trigger a heap crash (essentially a `while (SH_TRUE)` loop) to prevent any further damage if something very bad happen.
- `void sh_free(void *ptr)`: free the memory allocated at `ptr`. The heap internal will trigger a heap crash (essentially a `while (SH_TRUE)` loop) to prevent any further damage if something very bad happen.

6
docs/shelter/std/mem.md Normal file
View File

@@ -0,0 +1,6 @@
# Basic memory operations
The Shelter standard library provide very basic memory operations primitives, defined in `shelter/lib/include/std/mem.h` and implemented inside `shelter/lib/src/std/mem.c`:
- `SH_STATUS sh_mem_compare(const void *a,const void *b,sh_uint64 size)`: compare two memory regions with the same size. Return `SH_STATUS_SUCCESS` if both regions are equal, or `SH_STATUS_MEM_NOT_EQUAL` if one byte is different.
- `SH_STATUS sh_mem_copy(const void *destination,const void *source,sh_uint64 size)`: copy one region of memory to another. Return `SH_STATUS_SUCCESS`
- `SH_STATUS sh_mem_set_8(sh_uint8 *ptr,const sh_uint8 byte,sh_uint64 count)`: set a provided amount of bytes to the value of one provided byte. Return `SH_STATUS_SUCCESS`

20
docs/shelter/std/status.md Normal file
View File

@@ -0,0 +1,20 @@
# Return status
## Introduction
The vast majority of functions in Shelter return status code know as the `SH_STATUS` type. All the status codes are defined inside `shelter/lib/include/std/status.h`.
## Overview
Shelter define a status code with the `SH_STATUS` type. It's a wrapper of `sh_int64`.
The sign of a status play a big role:
- If the sign is negative, the status code isn't an error but rather an indication of something not that bad that happened during the function call. It can also serve the purpose of indicating a more detailled result.
- If the status code is 0, it's equivalent to a success, defined as `SH_STATUS_SUCCESS`
- If the sign is positive, the status code signify an error. You can use the `sh_status_error(SH_STATUS status)` function to know if a status code is an error. Return `SH_TRUE` if so.
Please check `shelter/lib/include/std/status.h` for the list of all status codes.
In Shelter, most function will returned to the caller the status code of all the functions calls they made if an error happened. Sometimes, it's necessary to dig through the code to understand from where an error come from.
If at any moment, a function that return a `SH_STATUS` call a function that return a pointer and the pointer is `SH_NULLPTR`, the status code `SH_STATUS_ERROR_NULLPTR_RETURNED` (not all error start with `SH_STATUS_ERROR`) will be returned.

14
docs/shelter/std/std.md Normal file
View File

@@ -0,0 +1,14 @@
# Shelter standard library
## Introduction
In order to function properly, any kernel need a standard library. Shelter define his own minimal standard library with his own particularities.
## Summary
1) [Basic types](types.md)
2) [Return status](status.md)
3) [Basic memory operations](mem.md)
4) [Heap memory allocations](malloc.md)
You can include the file `shelter/lib/include/std/stdlib.h` to include all necessary headers to access the Shelter standard library.

35
docs/shelter/std/types.md Normal file
View File

@@ -0,0 +1,35 @@
# Basic types
The Shelter standard library define the following type in `shelter/lib/include/std/type.h`:
- `sh_int8`: 1 byte signed integer
- `sh_uint8`: 1 byte unsigned integer
- `sh_int16`: 2 bytes signed integer
- `sh_uint16`: 2 bytes unsigned integer
- `sh_int32`: 4 bytes signed integer
- `sh_uint32`: 4 bytes unsigned integer
- `sh_int64`: 8 bytes signed integer
- `sh_uint64`: 8 bytes unsigned integer
- `sh_bool`: wrapper of `sh_uint8`
- `sh_iter64`: wrapper of `sh_uint64`, used for loops
- `va_list`: wrapper of `__builtin_va_list`
The Shelter standard library the following macros and values in `shelter/lib/include/std/type.h`:
- `SH_INT8_MIN`: the minimum possible value of `sh_int8`
- `SH_INT8_MAX`: the maximum possible value of `sh_int8`
- `SH_UINT8_MAX`: the maximum possible value of `sh_uint8`
- `SH_INT16_MIN`: the minimum possible value of `sh_int16`
- `SH_INT16_MAX`: the maximum possible value of `sh_int16`
- `SH_UINT16_MAX`: the maximum possible value of `sh_uint16`
- `SH_INT32_MIN`: the minimum possible value of `sh_int32`
- `SH_INT32_MAX`: the maximum possible value of `sh_int32`
- `SH_UINT32_MAX`: the maximum possible value of `sh_uint32`
- `SH_INT64_MIN`: the minimum possible value of `sh_int64`
- `SH_INT64_MAX`: the maximum possible value of `sh_int64`
- `SH_UINT64_MAX`: the maximum possible value of `sh_uint64`
- `SH_TRUE`: a wrapper of `(sh_bool)1`
- `SH_FALSE`: a wrapper of `(sh_bool)0`
- `SH_DEFVALUE`: the default value for any argument marked by `DEFAULT`
- `SH_NULLPTR`: wrapper of `(void*)0`
- `va_start(v,l)`: wrapper of `__builtin_va_start(v,l)`
- `va_end(v)`: wrapper of `__builtin_va_end(v)`
- `va_arg(v,l)`: wrapper of `__builtin_va_arg(v,l)`

53
docs/shelter/tab.md Normal file
View File

@@ -0,0 +1,53 @@
# Test-and-benchmark framework
## Introduction
The Shelter kernel includes his own test-and-benchmark (TAB) framework to support the measurements of performances and reliability of differents subsystems. This file describe the logic behind all the tests included inside the Shelter kernel and the embarqued payload, not the common API base described in [tests utilities](kernel/testutils.md) or the benchmark results.
A payload is a file generated at compilation, mostly random, used for accurate testing in specific situations.
## Slab allocators tests
These tests are designed to test the reliability and performances of the three main slab allocators behind the Pez subsystem: physical and virtual region objects slab allocators as well as the radix tree nodes slab allocators. They are defined inside `shelter/lib/include/kernel/tests/test_slabs.h` and implemented inside `shelter/lib/src/kernel/tests/test_slabs.c`. This test doesn't require a payload.
They consist in allocating 10000 objects from these three slabs and deallocating them afterward. This allocate a few slabs for each allocator, meaning they will consume a few pages at the start but nothing too expensive for the systems. This also allow for quick allocations of the first few thousands objects of these allocators since the slabs are already allocated.
## Radix tree subsystem tests
These tests are designed to test the reliability and performances of the radix tree subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_radix.h` and implemented inside `shelter/lib/src/kernel/tests/test_radix.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_radix_payload.h`.
The payload contain the following datas:
- `test_keys`: an array of 10000 random `sh_uint64` serving as keys for the pairs
- `test_values`: an array of 10000 random `sh_uint64` serving as values for the pairs
- `test_search`: an array of 10000 random `sh_uint64` serving as keys for the lower bound keys for the backtracking algorithm
All tests are run on a 16-deep unique radix tree. The test are in the following order:
- Insertion: trying to insert 10000 pairs of keys-values defined inside `test_keys` and `test_values`
- Reading: trying to obtain the associated value of each of the 10000 pairs by providing the values of `test_keys`, checking if the correct value is returned by comparing to the values of `test_values`
- Searching: trying to search using the backtracking algorithm with lower bound key using the values of `test_search`
- Deleting: trying to delete all pairs using the keys contained inside `test_keys`
## Pez subsystem tests
This test is designed to test the reliability and performances of the Pez subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_pez.h` and implemented inside `shelter/lib/src/kernel/tests/test_pez.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_pez_payload.h`.
This specific test has been designed to see how Pez react under high pressure. It's only apply it on Pez's physical plane, as the virtual plane uses the same logic.
The payload contain the following datas:
- `test_pez_physical_size`: an array of 2000 `sh_uint64` serving as sizes in pages for the allocation. Half of them are 1. The other half is between 2 and 1000.
- `test_pez_physical_alloc`: an array of 4000 `sh_uint64`, serving as reference for when to allocate and when to deallocate each allocated region. These numbers range from 1 to 2000 and appear exactly 2 times for each.
The test work like this: we iterate on all the value of `test_pez_physical_alloc`. When we hit for the first time a number, we allocate the corresponding size inside `test_pez_physical_size` by using the allocation number as index. When we hit for the second time a number, we deallocate this allocation.
The test will separate single page allocations and frees from multi-pages allocations and frees when showing the results.
## Malloc subsystem test
These tests are design to test the reliability and performances of the Pez subsystem. They are defined inside `shelter/lib/include/kernel/tests/test_malloc.h` and implemented inside `shelter/lib/src/kernel/tests/test_malloc.c`. They use the `shelter/lib/src/kernel/tests/payloads/test_malloc_payload.h`.
The payload contain the following datas:
- `test_malloc_small_size`: contain 10000 sizes for small allocations (under or equal to 1024 bytes)
- `test_malloc_big_size`: contain 1000 sizes in pages, ranging from 1 to 25, serving the same role as `test_pez_physical_size`
- `test_malloc_big_alloc`: contain 2000 values, serving the same role and generated like `test_pez_physical_alloc`
The malloc test combine two tests: 10000 allocations and deallocations of variously sized small objects using the generic slab allocators, and 1000 allocations and frees of pages on the heap, on the same logic of the Pez subsystem test, but adapted to the quantity used in this test.