essence-os/help/Build System.md

# Build system

Building an operating system is tricky: there are many different components to be built that all support varying configurations, and there are lots of dependencies (including compilers and assemblers) that need to be prepared. The Essence build system has been designed to make this as much of a streamlined process as is possible. And hopefully it is a lot easier than building other operating systems from scratch (for a comparison, see the wonderful "Linux From Scratch" books).

The build system is divided into three components: `util/start.script`, `util/build.c` and `util/build_core.c` ("build core"). 

- `util/start.script` first verifies the environment is acceptable for building Essence. It then compiles `util/build.c`. It then enters a command prompt interface where the developer can issue commands to build and test project, and additionally manage configurations, emulators, ports and the source tree. Most commands are implemented by calling out to `util/build.c`.
- `util/build.c` implements most of the high-level operations of commands called in `util/start.sh`. However, its functionality is being slowly moved into `util/start.script`, so that it can be removed.
- Build core is responsible for actually building Essence components, and producing a finished drive image. It expects all dependencies and the toolchain to be prepared before it is executed. It is able to run within Essence itself (where dependencies and a toolchain are already provided). It has minimal interaction with the platform it is running on. It is typically invoked automatically by `util/build.c`, which passes a description of the requested build in `bin/build.ini`. This includes information of the toolchain to use, enabled options, applications and drivers to compile, fonts to bundle, installation settings (such as how large the drive image should be), and other general settings (like how many threads to compile with, and whether optimisations are enabled).

`util/build_common.h` contains common definitions shared between these components.

## util/build.c

The following commands are available in the interactive prompt:

- `help` Print a list of the most commonly used commands, with a short description.
- `exit`, `x`, `quit`, `q` Exit the interactive prompt.
- `compile`, `c` Compile the operating system components with optimisations disabled. This does not produce a drive image.
- `build`, `b` Build the operating system to `bin/drive` with optimisations disabled.
- `build-optimised`, `opt` Build the operating system to `bin/drive` with optimisations enabled.
- `debug`, `d` Build the operating system to `bin/drive` with optimisations disabled, and then launch Qemu configured to wait for GDB to attach to it before starting.
- `vbox`, `v` Build the operating system to `bin/vbox.vdi` with optimisations enabled, and then launch VBox. To use this, you need to make a 128MB drive called `vbox.vdi` in the `bin` folder, and attach it to a virtual machine called "Essence" (choose "Windows 7 64-bit" as the OS).
- `vbox-without-opt`, `v2` Same as `v` but with optimisations disabled.
- `vbox-without-compile`, `v3` Same as `v` but compilation is skipped.
- `qemu-with-opt`, `t` Build the operating system to `bin/drive` with optimisations enabled, and then launch Qemu.
- `test`, `t2` Same as `t` but with optimisations disabled.
- `qemu-without-compile`, `t3` Same as `t` but compilation is skipped.
- `qemu-with-kvm`, `k` Same as `t` but Qemu is launched with the `-enable-kvm` flag. This results in significantly faster emulation. The command also disables the `Flag.DEBUG_BUILD` option; see below for a description of this.
- `build-cross` Force a rebuild of the cross compiler toolchain.
- `build-utilities`, `u` Build the utility applications only, which will run on the host development system. This is done automatically as needed by the other commands.
- `make-installer-archive` Compress the contents of `root/` to make a archive for use by the system installer application. This outputs the files `bin/installer_archive.dat` and `bin/installer_metadata.dat`.
- `make-installer-root` Copy all the needed files to make an installer image to `root/`. You need to run `make-installer-archive` first.
- `font-editor` Compile and launch the bitmap font editor utility application. This requires X11.
- `config` Compile and launch the config editor utility application. This requires X11.
- `designer2` Compile and launch the theme editor utility application. This requires X11.
- `replace-many` Replace a list of strings in the source code. A file containing the strings to replace and their replacements is passed. Each line should be of the form `<thing to replace> <what to replace it with>`.
- `replace` Replace a string in the source code. Only full words are matched.
- `replace-in` Same as `replace`, except it only replaces in a single folder.
- `find` Find a string in the source code. Ignores binary files.
- `find-word` Same as `find`, except only the full word is matched. For example, `at` would not match `cat`.
- `find-in` Same as `find`, except it looks in a single folder.
- `fix-replaced-field-name` After renaming a structure field, run this to change all references to it. It parses the compiler errors to do this automatically.
- `ascii <string>` Converts a string to ASCII codepoints.
- `build-optional-ports` Runs the `port.sh` script in each of the folders in `ports/`. If there is no script, the folder is skipped. Note that to use many of the ports the POSIX subsystem must be enabled.
- `do <string>` Run a command through `system()`.
- `live` Build a live CD or USB image. Work in progress.
- `line-count` Calculate the number of lines of code in the project.
- `a2l <symbols file>` Run the a2l utility. See the "a2l" section.
- `build-port` Build a single port. A list of availble ports are listed.
- `get-source <folder name> <url>` The file at the URL is downloaded and cached in `bin/cache`. The download is skipped if the file was already cached. It is then extracted and untar'd. The folder of the given name is then moved to `bin/source`.
- `get-source-checked <sha256> <folder name> <url>` Same as `get-source` except the downloaded file has its SHA-256 checksum compared against the given value.
- `make-crash-report` Copies various system files and logs into a `.tar.gz` which can be used to report a crash.
- `setup-pre-built-toolchain` Setup the pre-built toolchain for use by the build system. You can download and prepare it by running `./start.sh get-source prefix https://github.com/nakst/build-gcc/releases/download/gcc-11.1.0/gcc-x86_64-essence.tar.xz` followed by `./start.sh setup-pre-built-toolchain`.
- `run-tests` Run the API tests. `desktop/api_tests.ini` must be added to `bin/extra_applications.ini`, and `Emulator.SerialToFile` must be enabled.

## Levels of optimisation

Commands in the build system provide two levels of optimisation, either with optimisations enabled or disabled. However, there is an additional configuration flag, `Flag.DEBUG_BUILD` which can be toggled in the configuration editor. This determines whether the `DEBUG_BUILD` flag will be defined in the source code or not. When this flag is defined, many expensive checks are performed to constantly validate the operating system's data structures. To make a "fully optimised" build of Essence, you should first disable this flag, and then build with optimisations enabled. 

Note that some commands will force disable the `Flag.DEBUG_BUILD` option. Most notably, the `k` command.

## Build core

Build core can run in three modes. The mode is passed as the first argument.

- `headers <language> <path-to-output-file>` Generates the API header for `c` (also works for C++), `odin` or `zig`, and saves it to the provided output path.
- `application <configuration>` Builds a single application, given its configuration file. This will not install the application. See the "Application configuration options" section below.
- `standard <configuration>` Builds the system, given the configuration file. See the "Standard build configuration options" section below.

### Application configuration options

In the `[build]` section:

- `source` Path to the source file. This can be repeated, if the application has multiple translation units.
- `compile_flags` Additional flags to be passed to the compiler.
- `link_flags` Additional flags to be passed to the linker.
- `with_cstdlib` Set to 1 to link to the C standard library (currently Musl). This requires enabling the POSIX subsystem. Defaults to 0.
- `require` Optional. Set to the path of a file that must exist for the application to be built. If the file is not found, then the application will not be built.
- `custom_compile_command` A custom build command for the application. If this is set, then the default calls to the compiler and linker will be skipped. This command will be passed directly to `system()`. It should output an ELF executable to `bin/<application name>`. The application name is taken from the `[general]` section.

In the `[general]` section:

- `name` The name of the application.
- `disabled` Set to 1, and the application will not be built.
- `needs_native_toolchain` Set to 1 if the application requires a native toolchain to build. TODO Merge this with `with_cstdlib` from `[build]`?
- `icon` The name of the icon from `desktop/icons.header` to use for the application. This method of setting the application's icon should only be used for builtin application; third party application can set the icon of their application in the `[embed]` section. See below.
- `use_single_process` Set to 1 if the application should use a single process, shared between all its instances. Default is 0.
- `use_single_instance` Set to 1 if the application can only have a single instance open at a time. If the application is already open and the user tries to open it again, then the tab containing the loaded instance will be switched to. Default is 0.
- `hidden` Set to 1 if the application should not be listed on the "New Tab" screen.
- `permission_<permission>` Set to 1 to grant the application a specific application permission.
- `is_file_manager` Set to 1 if the application is the file manager. For system use only!
- `is_installer` Set to 1 if the application is the installer. For system use only!
- `background_service` Set to 1 if the application should be loaded at startup as a background service. For system use only!

In the `[embed]` section there is a list of files that should be embedded into the application bundle. These embedded file can be accessed at runtime using the `EsBundleFind` API. The entries in this section are of the form `<embedded name>=<path>`. Embedded name prefixed with `$` are reserved for definition by the system. The ELF executable files are automatically embedded into the application's bundle, with names of the form `$Executables/<target>`. The application's icon should be embedded as PNG images with names of the form `$Icons/<size>`. You must, as a minimum, provide sizes `16` (16x16 pixels) and `32` (32x32 pixels).

Each `[file_type]` section provides information about a file type.

- `match` Gives the file name extensions to match for the file type.
- `name` Gives the readable name of the file type, which will be shown to the user. TODO Translations.
- `icon` Gives the name of the icon from `desktop/icons.header` to show for files of this type. TODO Bundled icons.
- `has_thumbnail_generator` Set to 1 if the file type has a thumbnail generator. Only images are supported at the moment. TODO Custom thumbnail generators.
- `uuid` The EsUniqueIdentifier of the file type.
- `textual` Set to 1 if the file type can be read by plain text editors.
- `actions` The actions the application supports with this file type, e.g. `open`.

### Standard build configuration options

In the `[toolchain]` section:

- `path` Path to the `bin` folder of the toolchain.
- `tmpdir` A path to use to store temporary files. Passed to the toolchain in the `TMPDIR` environment variable.
- `ar`, `cc`, `cxx`, `ld`, `nm`, `strip`, `nasm`, `convert_svg` Paths to the toolchain executables.
- `linker_scripts` Path to the linker scripts. This should be set to the `util/` folder of the source tree.
- `compiler_objects` The path containing the `crt*.o` files provided by the toolchain.
- `crt_objects` The path where `crt*.o` files will be output.

In the `[general]` section:

- `system_build` Set to 1 to build the bootloader, Desktop and Kernel.
- `minimal_rebuild` Set to 1 to enable minimal rebuilds. Dependency information is stored in `bin/dependencies.ini`. Only components where the dependent source files have been modified will be rebuilt. All components are rebuilt if any of the options in this configuration file are modified.
- `colored_output` Set to 1 to enable ANSI color codes in the output.
- `thread_count` Set to the number of threads to use for calling the toolchain.
- `target` Set the name of the target platform, e.g. `x86_64`.
- `skip_header_generation` Set to 1 if the C/C++ API header does not need to be regenerated.
- `verbose` Set to 1 to list every toolchain invocation. See `thread_count=1` if you are using this.
- `common_compile_flags` Additional flags to pass to the C/C++ compiler invocations for all components.
- `for_emulator` Set to 1 to enable a few features that improve the experience of running the system on an emulator.
- `optimise` Enable compiler optimisations.
- `skip_compile` Skip compiling components and only build a drive image from the previously built executables.

In the `[install]` section:

- `file` The output file to put the drive image in.
- `partition_size` The size of the drive image in megabytes.
- `partition_label` The label of the partition. This can be set to anything you like, as long as it doesn't exceed `ESFS_MAXIMUM_VOLUME_NAME_LENGTH`.

In the `[options]` section there is a copy of the options in "Configuration options", from `bin/config.ini`. Note that many of these options are not handled by build core, but rather by `util/build.c`. See the "Configuration options" section for more details.

Each `[application]` section should contain a single key, `manifest`, giving the path to the application configuration options (see the above section).

Each `[driver]` section should contain:

- `name` The name of the driver. This must be found in `kernel/config.ini`.
- `source` The source file of the driver. This must be C/C++.
- `builtin` Set to 1 if the driver should be linked directly into the kernel. Set to 0 if the driver should be built as a loadable module.

`[font]` sections specify the fonts that are to be bundled in the desktop executable. Each section should contain:

- `name` The name of the font.
- `category` One of `Sans`, `Serif` or `Mono`. More categories will likely be added in the future.
- `scripts` The scripts supported by the font. See `hb_script_t` in `bin/harfbuzz/src/hb-common.h` for a list of scripts. Separate each script with a `,`.
- `license` The license file to bundle. This is a path relative to `res/Fonts/`.
- `.<digit>` and `.<digit>i` The font file for a weight 1-9, with `i` for italics. More possibilities will likely be added in the future. This is a path relative to `res/Fonts/`.

## Configuration editor

You can start the configuration editor by typing `config` at the build system prompt. Click `Save` to save your changes. Click `Defaults` to load the defaults. Left click an option in the table to toggle it or otherwise modify it. You will be presented with a warning if modifying the option is not recommended. Right click to reset a specific option. The changes are saved to `bin/config.ini`, which will not be uploaded to source control.

See the section "Configuration options" for a description of what each option does.

## Configuration options

Options starting with `Driver.` are used to enable and disable drivers. The defaults will be configured to work optimally for development work in emulators.

### Emulator options

- `Emulator.AHCI` Use AHCI/SATA for the drive in Qemu.
- `Emulator.ATA` Use ATA/IDE for the drive in Qemu.
- `Emulator.NVMe` Use NVMe for the drive in Qemu.
- `Emulator.CDROMImage` A path to a CD-ROM image that will be passed to Qemu.
- `Emulator.USBImage` A path to a drive image that will be passed to Qemu to appear as a USB mass storage device.
- `Emulator.USBHostVendor`, `Emulator.USBHostProduct` The vendor and product of a USB device to pass-through to Qemu. You will probably need to set `Emulator.RunWithSudo` for this to work.
- `Emulator.RunWithSudo` Run Qemu with `sudo`.
- `Emulator.Audio` Enable audio output in Qemu, saved to `bin/audio.wav`.
- `Emulator.MemoryMB` The amount of memory for Qemu. Probably needs to be at least 32 MB.
- `Emulator.Cores` The number of CPU cores to use in the emulator. Use 1 for a better debugging experience. Note that the build system command "k" automatically uses the maximum number of CPU cores possible.
- `Emulator.PrimaryDriveMB` The size of the primary drive image.
- `Emulator.SecondaryDriveMB` The size of a secondary drive image that will be connected to the emulator.
- `Emulator.VBoxEFI` Use UEFI boot in VBox.
- `Emulator.QemuEFI` Use Qemu boot in VBox.
- `Emulator.SerialToFile` Save the serial output from Qemu to `bin/Logs/qemu_serial1.txt`. If disabled, then the output will be available as a separate display in Qemu, but will not be saved after Qemu exits! You can view the serial output saved to a file as it is output using the shell command `tail -f bin/Logs/qemu_serial1.txt`.

### Build core options

- `BuildCore.Verbose` Set to enable `verbose` flag passed to build core. See the `[general]` section of the build core options.
- `BuildCore.NoImportPOSIX` Do not import the `root/Applications/POSIX` folder onto the drive image. This will allow building much smaller drive images.
- `BuildCore.RequiredFontsOnly` Only import the minimal number of fonts onto the drive image. This will allow building much smaller drive images.

### Flag options

- `Flag.ENABLE_POSIX_SUBSYSTEM` Enable the POSIX subsystem. Needed to use most ports and the POSIX Launcher.
- `Flag.DEBUG_BUILD` Set to 1 to define the `DEBUG_BUILD` flag. See the "Levels of optimisation" section above.
- `Flag.USE_SMP` Enable symmetric multiprocessing. You may wish to disable this for easier debugging.
- `Flag.PROFILE_DESKTOP_FUNCTIONS` Set to 1 to enable gf profiling integration for the Desktop.
- `Flag.BGA_RESOLUTION_WIDTH` The default resolution width to use for BGA, the graphics adapter used by Qemu. Must be a multiple of 4.
- `Flag.BGA_RESOLUTION_HEIGHT` The default resolution height to use for BGA, the graphics adapter used by Qemu. Must be a multiple of 4.
- `Flag.VGA_TEXT_MODE` Use VGA text mode at startup. Use with `Flag.START_DEBUG_OUTPUT` for early debugging.
- `Flag.CHECK_FOR_NOT_RESPONDING` Set to enable the Desktop's feature of checking whether the foreground application is responding. Disabling this will reduce the number of thread switches, making debugging slightly easier.
- `Flag._ALWAYS_USE_VBE` Use VBE mode setting. Needed for real hardware, if using the MBR bootloader.
- `Flag.COM_OUTPUT` Set to enable serial output. Should be disabled before making a build to run on real hardware.
- `Flag.POST_PANIC_DEBUGGING` Set to enable the special debugger that runs after a `KernelPanic`. Requires a PS/2 keyboard.
- `Flag.START_DEBUG_OUTPUT` Log kernel messages to the screen immediately during startup.
- `Flag.PAUSE_ON_USERLAND_CRASH` Send a userland application into a spin loop when it crashes. This makes it possible to attach a debugger (see `Debugging.md`) and get a stack trace.

### General options

- `General.first_application` The name of application to launch at startup. Useful if you don't want to have to manually open it from the "New Tab" page every time you restart the emulator.
- `General.wallpaper` The path to the wallpaper file. This is an Essence path; paths beginning with `0:/` will correspond to the files in the `root/` folder of the source tree.
- `General.window_color` The default window color.
- `General.installation_state` The installation state. If set to `1`, then the installer will be invoked. `make-installer-root` should have been run before doing this.
- `General.ui_scale` The default UI scale, as a percentage.
- `General.keyboard_layout` The default keyboard layout. See `Building.md` for instructions on how to set this.

### Dependency options

You can disable optional dependencies using these options.

- `Dependency.ACPICA` ACPICA is a kernel driver that provides power management through ACPI. Without it, the system will not be able to turn off the computer or provide thermal management.
- `Dependency.stb_image` stb\_image provides loading images of various formats. Without it, the system will only be able to load PNG images.
- `Dependency.stb_image_write` stb\_image\_write provides writing images of various formats. Without it, the system will not be able to save images. Note that the `Image Editor` application includes this library regardless of this setting.
- `Dependency.stb_sprintf` stb\_sprintf provides the printf-like API calls. Without it, these calls will be redirected to `EsUnimplemented`. Other optional dependencies may require this to be enabled.
- `Dependency.FreeTypeAndHarfBuzz` FreeType and HarfBuzz provide support for TrueType font rendering. Without it, the system will only be able to load bitmap fonts.

Additionally, there is the MD4C library which is included by the `Markdown Viewer` application.

## a2l

This utility is used to convert hexadecimal code addresses into line numbers. Start it with `./start.sh a2l <path to symbol file>`. Recall that symbol files are saved to `bin/<application name>`. Paste in any text containing addresses and they will be converted to line numbers, if possible. Usually you will want to do this for the crash information logged in `bin/Logs/qemu_serial1.txt`.