.Dd October 10, 2016
.Dt FOLLOWING-DEVELOPMENT 7
.Os
.Sh NAME
.Nm following-development
.Nd instructions for following development
.Sh DESCRIPTION
This document is a list of changes that affect the ability to build and run the
latest Sortix source code, when doing native
.Xr development 7
or
.Xr cross-development 7 .
The list is in reverse chronological order.
The system has policies that handle incompatible changes gracefully.
.Pp
If you are updating to new source code, be sure you are reading the new version
of this document, rather than an old copy installed on the system or on an
online manual page viewer.
To view the new version from the command line:
.Bd -literal
    cd /src  # system source code location
    man share/man/man7/following-development.7
.Ed
.Pp
The
.Xr sysmerge 8
and
.Xr sysupgrade 8
programs automatically handle compatibility issues when upgrading an existing
system to a new build.
This is done using upgrade hooks built into the programs that are run if the
installation does not have the appropriate files in
.Pa /share/sysinstall/hooks .
They can handle incompatible ABI changes and automatically fix configuration
files and other issues.
.Pp
When building the system from source code, you must have the latest build tools
installed as detailed in
.Xr development 7
and
.Xr cross-development 7 .
After upgrading to newer source code, you must first install the new build tools
before building.
If this is needed, there will be a notice in this document.
Build tools will likely complain about missing options or invalid parameters if
they are not up to date.
Automatic build scripts should use the
.Sy install-build-tools
target provided by the root makefile.
.Pp
Changes to the source code may rely on the newest build tools, but the build
tools must retain the ability to build all the way back to the latest stable
release.
The build tools of release N+1 must be able to build release N.
The build tools must be portable to the latest stable release and the supported
.Xr cross-development 7
systems.
If a build tool needs an incompatible change, there must be added a way to opt
into the new behavior, and the build system must use that option and not use the
old behavior any more.
This forward compatibility will allow the removal of the old behavior after the
next release.
Temporary compatibility must be mentioned with a condition for when it can be
removed and it must have instructions for what needs to be done.
Use a comment in the format
"TODO: After
.\" Line break so this occurrence doesn't make a false positive when I grep.
releasing Sortix x.y, foo." to allow the maintainer to easily
.Xr grep 1
for it after a release.
.Sh CHANGES
.Ss Add init groups
The
.Xr setinit 2
and
.Xr getinit 2
system calls have been added for nested init groups.
The
.Dv TIOCSCTTY
.Xr ioctl 2
for acquiring a controlling terminal has gained a force flag essential to
transferring terminals into nested sessions, and the new
.Dv TIOCUCTTY
.Xr ioctl 2
releases a controlling terminal.
This is a minor compatible ABI change.
.Ss Move system and ports from /etc/upgrade.conf to /tix/collection.conf
The
.Sy system
and
.Sy ports
variables in
.Pa /etc/upgrade.conf
.Xr ( upgrade.conf 5 )
has been moved to the
.Pa /tix/collection.conf
.Xr ( collection.conf 5 )
configuration file in the
.Xr tix-vars 8
format and renamed to
.Sy SYSTEM
and
.Sy PORTS .
.Pp
An upgrade hook will move the configuration to the new file.
.Ss Add tix-repository(8)
The new
.Xr tix-repository 8
program is used to generate repository metadata in releases.
The new
.Sy EDITION
and
.Sy RENAMES
variables in
.Xr port 5
are used to rename and split packages and requires a new
.Xr tix-build 8 .
.Pp
Tix must be upgraded:
.Bd -literal
    cd /src/tix &&
    make clean &&
    make install
.Ed
.Ss Support system upgrades and configuration in GRUB
The GRUB configuration now implements the
.Pa /etc
and
.Pa /etc/default
configuration split per
.Xr hier 7
and
.Xr sysmerge 8
now collaborates with system upgrades to boot them correctly.
Installations with manually configured bootloaders may need updating.
.Pp
Notably
.Pa /etc/default/grub
has been moved to
.Pa /etc/grub
since it is owned by the system administrator and there is no counterpart with
operating system provided defaults.
.Pp
Additionally
.Pa /etc/grub.d
has been moved into
.Pa /etc/default/grub.d
since it provides operating system defaults.
The system administrator can now create the
.Pa /etc/grub.d
directory manually to override files in
.Pa /etc/default/grub.d ,
although overriding
.Pa 10_sortix
is unsupported.
.Pp
.Pa 10_sortix
now supports the
.Pa GRUB_CMDLINE_SORTIX
variable in
.Pa /etc/grub
to specify kernel command line options.
.Pp
.Xr sysmerge 8
now colloborates with the
.Xr update-initrd 8
and
.Pa 10_sortix
scripts in the new system to prepare the upgrade system's kernel files in
.Pa /boot/sysmerge/ .
If the GRUB bootloader is accepted, then
.Pa /boot/grub/grub.cfg
is regenerated with a menu entry to perform the upgrade.
The
.Pa /boot/sortix.bin
and
.Pa /boot/sortix.initrd
files are no longer replaced with the new versions and continue to boot the old
system without upgrading.
Any manual bootloader configuration must be configured to instead load the
kernel and initrd from
.Pa /boot/sysmerge/
(if they exist).
The
.Pa /etc/default/grub.d/10_sortix.cache
file will demonstrate how to boot the new system using GRUB after starting a
.Xr sysmerge 8
(if the grub port is installed).
.Pp
An upgrade hook will delete the obsolete stale
.Pa /etc/grub.d/10_sortix.cache
file.
.Ss Third generation Tix
The tix binary package format has upgraded from generation 2 to 3 and has a new
internal layout that can be directly extracted into the filesystem.
The metatdata in
.Pa /tix
is now using the
.Xr tix-vars 8
format in the style of
.Xr port 5 .
.Pp
Tix must be upgraded to build the new ports:
.Bd -literal
    cd /src/tix &&
    make clean &&
    make install
.Ed
.Pp
An upgrade hook will migrate the
.Pa /tix
metadata to tix generation 3.
.Ss Add include and comment support to passwd(5) and group(5)
The
.Xr passwd 5
and
.Xr group 5
files have gained support for include statements via libc support.
Installations must now include
.Pa /etc/default/passwd.d/*
and
.Pa /etc/default/group.d/*
respectively in order to have system users and groups.
.Pp
An upgrade hook will add the inclusions to
.Pa /etc/passwd
and
.Pa /etc/group .
Applications accessing passwd and group databases must be recompiled with the
latest libc.
.Ss Add memory statistics to struct psctl_stat
The
.Xr psctl 2
now provides per-process memory statistics in
.Vt struct psctl_stat
via the new members
.Fa pss ,
.Fa rss ,
.Fa uss ,
and
.Fa vms .
.Pp
This is an incompatible ABI change.
.Ss Add memusage(2)
The
.Xr memusage 2
system call has been added, which provides detailed system memory statistics.
This is a minor compatible ABI change.
.Ss Add networking stack
The network stack has been implemented in the kernel and exposed through
additions to the system call interface.
.Pp
This is a compatible ABI change that adds features to
.Xr socket 2
.Sy ( AF_INET , IPPROTO_TCP , IPPROTO_UDP , IPPROTO_PING ) ,
the ioctls for
.Xr if 4 ,
socket options, and the
.Xr lo 4
loopback interface.
.Ss Add daemon support to init(8)
.Xr init 8
has gained
.Xr daemon 7
support with the new
.Xr init 5
configuration format.
.Pp
The old
.Pa /etc/init/target
configuration file is replaced by the
.Sy default
daemon in
.Pa /etc/init/default .
An upgrade hook will migrate the configuration.
.Ss Add ports to the Sortix repository
The ports have been moved from the porttix/srctix repositories into the
.Pa ports/
subdirectory in the main repository.
The upstream releases are downloaded from the appropriate mirror when built and
patched with the patches in the
.Pa ports/
subdirectory.
.Pp
Any existing manually created
.Pa ports/
directory should be removed before checking out the new source code.
.Pp
Tix must be upgraded to build the new ports:
.Bd -literal
    cd /src/tix &&
    make clean &&
    make install
.Ed
.Pp
If not developing natively, set
.Ev PREFIX
to the desired location.
.Pp
All ports are built by default and the
.Ev PACKAGES
environment variable configures which ports are built.
.Pp
Sortix 1.0 has a broken
.Xr patch 1
and the
.Sy extract-ports
makefile target should be invoked on another system in order to bootstrap the
new version.
.Ss Implement file descriptor passing
The
.Dv SCM_RIGHTS
control message have been implemented, allowing file descriptors to be passed
over
.Dv AF_UNIX
sockets.
This is a minor compatible ABI change.
.Ss Implement threading primitives that truly sleep
The
.Xr futex 2
system system call for efficient thread waiting has been added.
The
.Xr exit_thread 2
system call has gained a
.Dv EXIT_THREAD_FUTEX_WAKE
flag for waking a single waiter on a futex.
This is a minor compatible ABI change.
.Ss Fix system upgrade leaking files
.Xr sysupgrade 8
and
.Xr sysmerge 8
will now delete files that no longer exist in the new system and ports.
However, files may already have leaked if a 1.0 installation was upgraded to
a development build prior to this change.
An upgrade hook will delete any well known leaked files.
.Pp
Note:
You must use the
.Fl \-wait
option to do a two-stage upgrade if doing a
.Xr sysmerge 8
upgrade from an installation prior to this change to a version after this
change.
This requirement is because the old
.Xr sysmerge 8
will leak files and the upgrade hook only deal with well known files as of this
change, and doesn't handle future changes.
.Ss Fix /tix/manifest permissions in installations
The
.Pa /tix/manifest
directory was accidentally installed by
.Xr sysinstall 8
as mode 6603 instead of 7555.
This problem is fixed with an upgrade hook.
.Ss Add socket(2)
The
.Pa /dev/net
virtual filesystem for socket creation has been removed in favor of adding an
actual
.Xr socket 2
system call.
This is a major incompatible ABI change.
Only Unix sockets were exposed this way.
In the base system,
.Xr sf 1
is the only program with Unix socket capabilities.
Ports using Unix sockets must be updated.
Otherwise the system is compatible except accessing Unix sockets fails with
.Er ENOENT .
.Ss Add split packages and cross-bootstrapping support to tix-build(8)
.Xr tix-build 8
has gained a number of features that will soon be required in order to build
certain ports.
In particular, it now supports the
.Sy pkg.use-bootstrap ,
.Sy pkg.source-package ,
and
.Sy pkg.alias-of
variables; bugs in the
.Sy pkg.subdir
variable have been fixed; and a
.Fl \-source-directory
option has been added.
.Xr tix-build 8
must be upgraded before building ports using any of those features.
.Bd -literal
    cd /src/tix &&
    make clean &&
    make install
.Ed
.Pp
If not developing natively, set
.Ev PREFIX
to the desired location.
.Pp
If the new program isn't used, ports may fail to build due to local software not
being the exact same version
.Sy ( pkg.use-bootstrap ) ,
clean or post-install in the wrong subdirectory
.Sy ( pkg.subdir) ,
the
.Fl \-source-directory
option not being recognized or failing to locate the source code
.Sy ( pkg.source-package) ,
stopping because
.Sy pkg.build-system
isn't set
.Sy ( pkg.alias-of) ,
or other mysterious circumstances.
.Ss Add German keyboard layout
The
.Xr kblayout-compiler 1
build tool has gained support for modifier combinations used by the German
keyboard layout, and the new German keyboard layout relies on this.
.Xr kblayout-compiler 1
must be upgraded before building the system.
.Bd -literal
    cd /src/kblayout-compiler &&
    make clean &&
    make install
.Ed
.Pp
If not developing natively, set
.Ev PREFIX
to the desired location.
.Pp
If the new program isn't used, the build will still complete successfully but an
incorrect German keyboard layout will be installed.
.Ss Seed kernel entropy with randomness from the previous boot
Entropy from the previous boot is now stored in
.Pa /boot/random.seed .
The bootloader is supposed to load this file as a multiboot module with the
command line option
.Fl \-random-seed .
The kernel will issue a security warning if it was booted without a random seed,
unless the kernel command line contains
.Fl \-no-random-seed .
The GRUB port has been updated with an improved
.Pa /etc/default/grub.d/10_sortix
script that will automatically emit the appropriate GRUB commands.
.Pp
Users using the included GRUB will need to update to the latest GRUB port
and then run
.Xr update-initrd 8
to regenerate
.Pa /etc/grub/grub.cfg .
All of this will be handled automatically if upgrading with
.Xr sysupgrade 8
and the new build contains the new GRUB, or if upgrading with
.Xr sysmerge 8
and the source system root contains the new GRUB.
.Pp
Users not using the included GRUB, but still using GRUB from another
installation, will need to reconfigure that bootloader installation.
In the boot commands of this system, add after the initrd load:
.Bd -literal
    module /boot/random.seed --random-seed
.Ed
.Pp
If the GRUB port is installed, but not used, then if that port is updated with
.Xr sysupgrade 8
or
.Xr sysmerge 8
or manually, the
.Pa /etc/default/grub.d/10_sortix
script can be invoked, which will generate a
.Pa /etc/default/grub.d/10_sortix.cache
fragment that can be spliced into the configuration of another GRUB
installation.
.Pp
Users not using GRUB will need to configure their bootloader
to load
.Pa /boot/random.seed
appropriately.
.Pp
.Xr sysmerge 8
and
.Xr sysupgrade 8
will automatically create
.Pa /boot/random.seed
if it doesn't exist.
If using neither to upgrade, manually create that file owned by user root and
group root with mode 600 containing 256 bytes of entropy.
.Ss Modernize carray(1) and fix missing allocation checks
The
.Xr carray 1
build tool has gained the
.Fl EGHot
options and the build system now relies on this.
.Xr carray 1
must be upgraded before building the system.
.Bd -literal
    cd /src/carray &&
    make clean &&
    make install
.Ed
.Pp
If not developing natively, set
.Ev PREFIX
to the desired location.
.Ss Remove compatibility with Sortix 0.9
Sortix 1.0 has been released.
The build tools are no longer capable of building anything prior to the Sortix
1.0 release.
.Sh SEE ALSO
.Xr cross-development 7 ,
.Xr development 7 ,
.Xr sysmerge 8 ,
.Xr sysupgrade 8