vystem/docs/shelter/cpu/ap.md

Application Processors subsystem

Introduction

The Application Processors (AP) subsystem is responsible for managing SMP-related CPU abstractions, including CPU descriptors and per-CPU data structures. It also handles the preparation and bootstrap sequence required to bring up secondary processors. This subsystem is defined inside shelter/lib/include/cpu/ap.h and implemented inside shelter/lib/include/cpu/ap.c.

Overview

This subsystem is responsible for three things:

• defining the structures used to describe CPUs and storing per-CPU metadatas
• preparing all necessary structures in order to start the APs
• starting the APs

CPU structures

The AP subsystem defines two structures related to CPU metadatas:

sh_ap_CPU_STRUCT:

#pragma pack(1)
typedef struct {
  sh_page_VIRTUAL_ADDRESS c_entry_point_stack_top_va;
  sh_uint32 lapic_id;
  sh_uint16 tss_selector;
  sh_ap_PER_CPU *per_cpu;
} sh_ap_CPU_STRUCT;
#pragma pack()

The CPU struct is mainly used for the AP trampoline: it contain informations specific to each AP that will allow them to reach the C entry point. It also contains a pointer to the per-CPU struct of each CPU.

sh_ap_PER_CPU:

#pragma pack(1)
typedef struct {
  sh_uint64 cpu_id;
  sh_uint32 bytes_outputed;
  sh_bool timer_state;
  char *temp_buffer;
  sh_uint64 temp_buffer_size;
} sh_ap_PER_CPU;
#pragma pack()

The per-CPU struct is used for per-CPU metadatas like LAPIC timer state, per-CPU temporary buffer, logical CPU id and various per-CPU metadatas for differents kernel subsystems.

The logical CPU id is an id assigned to each processor. The bootstrap processor always obtain the id 0, and there is no gap in the logical CPU id allocation space. It allow for an alternative to LAPIC ids and ACPI processors id, which can contain gaps in their allocation space.

The GS base register of each AP contain a pointer to the CPU struct of each AP. The CPU and per-CPU struct of the bootstrap processor is created at the start of the kernel entry point and then elaborated during the AP bootstrap procedure. It isn't stored inside the AP's CPU struct array.

Preparing all necessary structures for AP bootstrap

Since all APs start in 16 bits mode, Shelter uses the following physical address to put importants compoments there:

• 0x7000: the physical address of the AP trampoline, shared by all APs
• 0x8000: the physical address of the 32 bits GDT, necessary for the 32 bits far jump
• 0x9000: the physical address of the AP bootstrap struct

All those addresses are identity mapped in the bootstrap CPU page table. Additionally, the AP trampoline loaded from the VYX binary of Shelter is mapped at the virtual address 0xA000.

The AP bootstrap struct is a structure that contains all commons addresses used by all APs:

#pragma pack(1)
typedef struct {
  sh_page_PHYSICAL_ADDRESS lapic_base_pa;
  sh_page_VIRTUAL_ADDRESS c_entry_point_va;
  sh_page_PHYSICAL_ADDRESS page_table_pa;
  sh_page_VIRTUAL_ADDRESS cpu_struct_base_area;
  sh_page_VIRTUAL_ADDRESS shared_gdt_64_va;
  sh_gdt_GDTR gdt_64_gdtr;
  sh_idt_IDT *idt;
} sh_ap_AP_BOOTSTRAP;
#pragma pack()

It contain things such as the LAPIC memory mapped area PA, the C entry point VA, the page table PA, the CPU struct array VA, the shared GDT VA for the 64 bits far jump, the GDTR for the said GDT and the IDT VA.

During the kernel boot process, all APs share the same page table as the bootstrap processor for simplicity.

First, the 32 bits GDT is generated and placed at PA 0x8000 by sh_ap_load_gdt_32(). Then the rest of the AP bootstrap preparation is handled by sh_ap_prepare_for_smp_launch()

Here is the detailled processus:

1. Identity-mapping the page at PA 0x9000 to initialize obvious values like C entry point VA, page table PA and LAPIC base PA.
2. Allocating the array of CPU struct on the heap (indexed by LAPIC id), and initializing the corresponding value in the AP bootstrap struct.
3. Allocating the array of TSS on the heap.
4. Looping on all availables LAPIC to fill all CPU struct: allocating the stack for each AP, filling the LAPIC id, filling the TSS of each AP, allocating the multiple stacks for each TSS, filling the per-CPU struct of each AP.
5. Filling the global GDT for all APs with all TSS, completing the AP bootstrap struct, initializing the AP start state.

Starting the APs

Once everything is ready, sh_ap_start_ap_boot_procedure() is called once to start all APs. The procedure is as follow:

1. Copying the AP trampoline from VA 0xA000 to VA/PA 0x7000.
2. Filling CPU struct and per-CPU struct of the bootstrap CPU.
3. Sending INIT IPI and waiting 10 millisecond.
4. Sending SIPI and waiting for all APs to start. At this point, the sh_print standard library feature is ready.

All APs will start in 16 bits mode and execute the following sequence of action:

1. Disabling interrupts
2. Load 32 bits GDT, enabling 32 bits mode in CR0 and executing the far jump to get to 32 bits mode
3. Reading LAPIC base, extracting LAPIC id and keeping it in a register
4. Enabling PAE, loading page table PA into CR3, enabling LME and NXE bits
5. Enabling pagging, and executing the far jump to get to 64 bits mode
6. Locating the CPU struct for this CPU using the CPU LAPIC id
7. Loading the local CPU stack
8. Loading 64 bits shared GDT, reloading CS register
9. Loading task register and GS register
10. Jumping to C entry point

The C entry point is the function named sh_ap_entry_point(). It does the following things:

1. Initialize the LAPIC of the AP
2. Load the shared IDT
3. Confirm to the bootstrap CPU that everything is ready for this AP by updating the corresponding AP start state and goes into an infinite loop

If the bootstrap CPU doesn't receives any confirmation from one of the AP, the boot process is halted.