Sortix cross-volatile manual

This manual documents Sortix cross-volatile. You can instead view this document in the latest official manual.

NAME

following-development — instructions for following development

DESCRIPTION

This document is a list of changes that affect the ability to build and run the latest Sortix source code, when doing native development(7) or cross-development(7). The list is in reverse chronological order. The system has policies that handle incompatible changes gracefully.

If you are updating to new source code, be sure you are reading the new version of this document, rather than an old copy installed on the system or on an online manual page viewer. To view the new version from the command line:

    cd /src  # system source code location 
    man share/man/man7/following-development.7

The sysmerge(8) and sysupgrade(8) programs automatically handle compatibility issues when upgrading an existing system to a new build. This is done using upgrade hooks built into the programs that are run if the installation does not have the appropriate files in /share/sysinstall/hooks. They can handle incompatible ABI changes and automatically fix configuration files and other issues.

When building the system from source code, you must have the latest build tools installed as detailed in development(7) and cross-development(7). After upgrading to newer source code, you must first install the new build tools before building. If this is needed, there will be a notice in this document. Build tools will likely complain about missing options or invalid parameters if they are not up to date. Automatic build scripts should use the install-build-tools target provided by the root makefile.

Changes to the source code may rely on the newest build tools, but the build tools must retain the ability to build all the way back to the latest stable release. The build tools of release N+1 must be able to build release N. The build tools must be portable to the latest stable release and the supported cross-development(7) systems. If a build tool needs an incompatible change, there must be added a way to opt into the new behavior, and the build system must use that option and not use the old behavior any more. This forward compatibility will allow the removal of the old behavior after the next release. Temporary compatibility must be mentioned with a condition for when it can be removed and it must have instructions for what needs to be done. Use a comment in the format "TODO: After releasing Sortix x.y, foo." to allow the maintainer to easily grep(1) for it after a release.

CHANGES

Move system and ports from /etc/upgrade.conf to /tix/collection.conf

The system and ports variables in /etc/upgrade.conf (upgrade.conf(5)) has been moved to the /tix/collection.conf (collection.conf(5)) configuration file in the tix-vars(8) format and renamed to SYSTEM and PORTS.

An upgrade hook will move the configuration to the new file.

Add tix-repository(8)

The new tix-repository(8) program is used to generate repository metadata in releases. The new EDITION and RENAMES variables in port(5) are used to rename and split packages and requires a new tix-build(8).

Tix must be upgraded:

    cd /src/tix && 
    make clean && 
    make install

Add tix-metabuild(8)

The new tix-metabuild(8) program is used to build and clean ports.

Tix must be upgraded:

    cd /src/tix && 
    make clean && 
    make install

Add init groups

The setinit(2) and getinit(2) system calls have been added for nested init groups. The TIOCSCTTY ioctl(2) for acquiring a controlling terminal has gained a force flag essential to transferring terminals into nested sessions, and the new TIOCUCTTY ioctl(2) releases a controlling terminal. This is a minor compatible ABI change.

Update to gcc-14.2.0 and binutils-2.43.1

The toolchain has been updated to gcc 14.2.0 and binutils 2.43.1 and must be rebuilt before building the operating system per development(7) and cross-development(7):

    cd /src && 
    make clean-cross-toolchain && 
    make PREFIX="$CROSS_PREFIX" TARGET=$SORTIX_PLATFORM install-cross-toolchain

For native development(7), the TARGET variable can be omitted. Use a temporary non-empty PREFIX to avoid clobbering the system toolchain and add $PREFIX/bin to the PATH environment variable afterwards.

The new toolchain is not able to build versions before this commit due to bugs in the base system and ports that are now compile-time errors. These issues are fixed as of this commit.

The old toolchain with gcc 5.2.0 and binutils 2.24 remain supported to build the base system and selected ports, such that the new version of the operating system can be bootstrapped. However, ports may now require the new toolchain and it must be upgraded to build all ports. Backwards compatibility for the old toolchain will be dropped after releasing the next version of this operating system.

Support combining CONFIGURE_USE_BUILD_DIRECTORY and SUBDIR

tix-build(8) now supports combining CONFIGURE_USE_BUILD_DIRECTORY and SUBDIR port(5) variables to do an out-of-directory build in a subdirectory.

Tix must be upgraded:

    cd /src/tix && 
    make clean && 
    make install

Support system upgrades and configuration in GRUB

The GRUB configuration now implements the /etc and /etc/default configuration split per hier(7) and sysmerge(8) now collaborates with system upgrades to boot them correctly. Installations with manually configured bootloaders may need updating.

Notably /etc/default/grub has been moved to /etc/grub since it is owned by the system administrator and there is no counterpart with operating system provided defaults.

Additionally /etc/grub.d has been moved into /etc/default/grub.d since it provides operating system defaults. The system administrator can now create the /etc/grub.d directory manually to override files in /etc/default/grub.d, although overriding 10_sortix is unsupported.

10_sortix now supports the GRUB_CMDLINE_SORTIX variable in /etc/grub to specify kernel command line options.

sysmerge(8) now colloborates with the update-initrd(8) and 10_sortix scripts in the new system to prepare the upgrade system's kernel files in /boot/sysmerge/. If the GRUB bootloader is accepted, then /boot/grub/grub.cfg is regenerated with a menu entry to perform the upgrade. The /boot/sortix.bin and /boot/sortix.initrd files are no longer replaced with the new versions and continue to boot the old system without upgrading. Any manual bootloader configuration must be configured to instead load the kernel and initrd from /boot/sysmerge/ (if they exist). The /etc/default/grub.d/10_sortix.cache file will demonstrate how to boot the new system using GRUB after starting a sysmerge(8) (if the grub port is installed).

An upgrade hook will delete the obsolete stale /etc/grub.d/10_sortix.cache file.

Third generation Tix

The tix binary package format has upgraded from generation 2 to 3 and has a new internal layout that can be directly extracted into the filesystem. The metatdata in /tix is now using the tix-vars(8) format in the style of port(5).

Tix must be upgraded to build the new ports:

    cd /src/tix && 
    make clean && 
    make install

An upgrade hook will migrate the /tix metadata to tix generation 3.

Add include and comment support to passwd(5) and group(5)

The passwd(5) and group(5) files have gained support for include statements via libc support. Installations must now include /etc/default/passwd.d/* and /etc/default/group.d/* respectively in order to have system users and groups.

An upgrade hook will add the inclusions to /etc/passwd and /etc/group. Applications accessing passwd and group databases must be recompiled with the latest libc.

Add memory statistics to struct psctl_stat

The psctl(2) now provides per-process memory statistics in struct psctl_stat via the new members pss, rss, uss, and vms.

This is an incompatible ABI change.

Add memusage(2)

The memusage(2) system call has been added, which provides detailed system memory statistics. This is a minor compatible ABI change.

Add networking stack

The network stack has been implemented in the kernel and exposed through additions to the system call interface.

This is a compatible ABI change that adds features to socket(2) (AF_INET, IPPROTO_TCP, IPPROTO_UDP, IPPROTO_PING), the ioctls for if(4), socket options, and the lo(4) loopback interface.

Add daemon support to init(8)

init(8) has gained daemon(7) support with the new init(5) configuration format.

The old /etc/init/target configuration file is replaced by the default daemon in /etc/init/default. An upgrade hook will migrate the configuration.

Add ports to the Sortix repository

The ports have been moved from the porttix/srctix repositories into the ports/ subdirectory in the main repository. The upstream releases are downloaded from the appropriate mirror when built and patched with the patches in the ports/ subdirectory.

Any existing manually created ports/ directory should be removed before checking out the new source code.

Tix must be upgraded to build the new ports:

    cd /src/tix && 
    make clean && 
    make install

If not developing natively, set PREFIX to the desired location.

All ports are built by default and the PACKAGES environment variable configures which ports are built.

Sortix 1.0 has a broken patch(1) and the extract-ports makefile target should be invoked on another system in order to bootstrap the new version.

Implement file descriptor passing

The SCM_RIGHTS control message have been implemented, allowing file descriptors to be passed over AF_UNIX sockets. This is a minor compatible ABI change.

Implement threading primitives that truly sleep

The futex(2) system system call for efficient thread waiting has been added. The exit_thread(2) system call has gained a EXIT_THREAD_FUTEX_WAKE flag for waking a single waiter on a futex. This is a minor compatible ABI change.

Fix system upgrade leaking files

sysupgrade(8) and sysmerge(8) will now delete files that no longer exist in the new system and ports. However, files may already have leaked if a 1.0 installation was upgraded to a development build prior to this change. An upgrade hook will delete any well known leaked files.

Note: You must use the --wait option to do a two-stage upgrade if doing a sysmerge(8) upgrade from an installation prior to this change to a version after this change. This requirement is because the old sysmerge(8) will leak files and the upgrade hook only deal with well known files as of this change, and doesn't handle future changes.

Fix /tix/manifest permissions in installations

The /tix/manifest directory was accidentally installed by sysinstall(8) as mode 6603 instead of 7555. This problem is fixed with an upgrade hook.

Add socket(2)

The /dev/net virtual filesystem for socket creation has been removed in favor of adding an actual socket(2) system call. This is a major incompatible ABI change. Only Unix sockets were exposed this way. In the base system, sf(1) is the only program with Unix socket capabilities. Ports using Unix sockets must be updated. Otherwise the system is compatible except accessing Unix sockets fails with ENOENT.

Add split packages and cross-bootstrapping support to tix-build(8)

tix-build(8) has gained a number of features that will soon be required in order to build certain ports. In particular, it now supports the pkg.use-bootstrap, pkg.source-package, and pkg.alias-of variables; bugs in the pkg.subdir variable have been fixed; and a --source-directory option has been added. tix-build(8) must be upgraded before building ports using any of those features.

    cd /src/tix && 
    make clean && 
    make install

If not developing natively, set PREFIX to the desired location.

If the new program isn't used, ports may fail to build due to local software not being the exact same version (pkg.use-bootstrap), clean or post-install in the wrong subdirectory (pkg.subdir), the --source-directory option not being recognized or failing to locate the source code (pkg.source-package), stopping because pkg.build-system isn't set (pkg.alias-of), or other mysterious circumstances.

Add German keyboard layout

The kblayout-compiler(1) build tool has gained support for modifier combinations used by the German keyboard layout, and the new German keyboard layout relies on this. kblayout-compiler(1) must be upgraded before building the system.

    cd /src/kblayout-compiler && 
    make clean && 
    make install

If not developing natively, set PREFIX to the desired location.

If the new program isn't used, the build will still complete successfully but an incorrect German keyboard layout will be installed.

Seed kernel entropy with randomness from the previous boot

Entropy from the previous boot is now stored in /boot/random.seed. The bootloader is supposed to load this file as a multiboot module with the command line option --random-seed. The kernel will issue a security warning if it was booted without a random seed, unless the kernel command line contains --no-random-seed. The GRUB port has been updated with an improved /etc/default/grub.d/10_sortix script that will automatically emit the appropriate GRUB commands.

Users using the included GRUB will need to update to the latest GRUB port and then run update-initrd(8) to regenerate /etc/grub/grub.cfg. All of this will be handled automatically if upgrading with sysupgrade(8) and the new build contains the new GRUB, or if upgrading with sysmerge(8) and the source system root contains the new GRUB.

Users not using the included GRUB, but still using GRUB from another installation, will need to reconfigure that bootloader installation. In the boot commands of this system, add after the initrd load:

    module /boot/random.seed --random-seed

If the GRUB port is installed, but not used, then if that port is updated with sysupgrade(8) or sysmerge(8) or manually, the /etc/default/grub.d/10_sortix script can be invoked, which will generate a /etc/default/grub.d/10_sortix.cache fragment that can be spliced into the configuration of another GRUB installation.

Users not using GRUB will need to configure their bootloader to load /boot/random.seed appropriately.

sysmerge(8) and sysupgrade(8) will automatically create /boot/random.seed if it doesn't exist. If using neither to upgrade, manually create that file owned by user root and group root with mode 600 containing 256 bytes of entropy.

Modernize carray(1) and fix missing allocation checks

The carray(1) build tool has gained the -EGHot options and the build system now relies on this. carray(1) must be upgraded before building the system.

    cd /src/carray && 
    make clean && 
    make install

If not developing natively, set PREFIX to the desired location.

Remove compatibility with Sortix 0.9

Sortix 1.0 has been released. The build tools are no longer capable of building anything prior to the Sortix 1.0 release.