Porting Guide ============= This guide documents how to port software to the Sortix operating system and how to release your port to the Sortix community. Before you get started, you should get the system source code and the current core set of existing ports and be comfortable with building the system (with ports) entirely from scratch. Overview -------- ### `$SORTIX_TOPLEVEL` ### This is the top-level source directory containing the system source code. This is actually not a real environmental variable, but just something used in this document to denote the top-level system source code directory. This is where you have cloned the Sortix source repository. ### Source Tix ### A source tix is simply a directory, whose name is the name of a package, and which contains the source code for the package and a special `tixbuildinfo` file, which contains instructions for building the source tix into a binary tix. ### `$SORTIX_PORTS_DIR` (default `$SORTIX_TOPLEVEL/ports`) ### This directory is where the build system will search for source tixes that will automatically be built along with the rest of the system. Any other files and sub-directories will be ignored. You can use symbolic links to source tixes if you wish. You integrate new packages into the build simply by simply putting the source tix inside this directory as a sub-directory. It will automatically be built along with the rest of the system during the next system build. ### `.srctix.tar.xz` (Archived Source Tix) ### Since you cannot publish raw directories, published source tixes are put into a compressed archive and given the extension `.srctix.tar.xz`. You can easily install an archived source tix by extracting it using standard tools: cd $SORTIX_PORTS_DIR && tar --extract --file libfoo.srctix.tar.xz This will install the source tix into the source repository and it will automatically be built the next time you compile the system and ports. The archived source tix is simply a tarball that contains a single directory, which is a source tix as described above. ### `.porttix.tar.xz` (Archived Port Tix) ### Upstream releases of software cannot be expected to contain a `tixbuildinfo` file and sometimes they need to be patched. When maintaining a port, it is often useful to have a copy of the upstream release and the patches applied to it. The `.porttix.tar.xz` archives contain a copy of the upstream release (as a compressed archive) and all the patches applied to it. This can automatically be converted into an archived source tix using the `srctix-create` program. Users will normally not use this format unless they wish to inspect how a package was ported or confirm that no malicious alternations were made to the upstream release. Can the package be ported? -------------------------- It's a very good idea to examine the package closely before attempting to port it, or you might end up wasting a lot of time on something that isn't even possible in the first place. There's a number of red flags that can prevent the successful porting of a package, at least not without enhancements to the tix package management system or to Sortix itself. The first thing to verify is whether we want such a package on Sortix. Not all packages are good, some have bad licenses, some have security problems, or bad code quality -- and so on. Perhaps there is a similar package already ported, which is technically superior, and it's better to have just one such package on the system for the sake of purity. Perhaps it's just better to leave this package behind and focus on the pure Sortix future? Does the philosophy of the package developers contradict that of the Sortix project? The second thing to verify is the build system. If it uses GNU autoconf or a compatible ./configure script things are looking good. If the package has a custom hand-written ./configure script you will likely have trouble, as the authors of these tend to be ignorant about many useful ./configure options and likely don't properly support-cross-compilation. If the package uses some exotic build system, you might need to write wrapper scripts or teach Tix how to deal with such build systems. If the package just has a makefile, you will likely run into trouble if it doesn't follow good makefile conventions and you might end up heavily patching it or rolling your own makefile. In these bad cases, consider whether we even want to port such a package to Sortix. The third thing to check is whether it cross-compiles. A lot of software have considerable problems with this, and while we are able to work around some of these problems, sometimes it's just not feasible. It's probably worth searching around the net to see if other people have cross-compiled it. Perhaps they ran into similar problems - as you are about to have - but have a fix? The fourth thing to verify is whether Sortix is ready. Perhaps it uses some API that Sortix doesn't have yet or depend on kernel features that are missing? Perhaps the package has dependencies that are not yet satisfies, because nobody has ported all the dependencies yet. You can often check guides such as "Beyond Linux from Scratch" which contains a lot of building instructions and dependency information. Try ask around in the operating system development community if anyone has ported it before and what their experience was. Authoring a Source Tix ---------------------- The first step in porting software to Sortix is creating a working source tix and building it along with the rest of the system. You will need to get a copy of the software you wish to install. You will need to save a copy of this original compressed archive in a safe place, as you will need it later when you create an archived port tix for publishing. In this example we will pretend to port a fictitious piece of software called `libfoo`. The latest release of libfoo is released as a compressed tarball with the filename `libfoo-0.42.tar.xz`. You will save a copy of this original tarball in a safe location and then proceed to extract it into the `$SORTIX_PORTS_DIR` directory. This will usually create a `libfoo-0.42` sub-directory, but you need to rename it to simply `libfoo`. The next step is to author a `libfoo/tixbuildinfo` file, if libfoo does not ship with support for Sortix. You will need to examine the package and deduce what it's build system is and write the file accordingly. As a minimum, you will need to put this in the file (with no leading spaces): tix.version=1 tix.class=srctix pkg.name=libfoo pkg.build-libraries=libbar libbaz libqux This is a simple key-value format. You will need to set the `pkg.name` variable to the exact name of the source tix. The `pkg.build-libraries` variable contains the build-time dependencies of the package as a space-delimited set. This is the common key-values that all `tixbuildinfo` files *must* contain. In addition, they must also contain the `pkg.build-system` variable. GNU configure scripts --------------------- The Tix package management system is designed such that it is normally easy to port a package using autoconf generated `./configure` scripts (or compatible). In the best case, it will simply suffice to write: pkg.build-system=configure ### config.sub ### Most packages contain a copy of the config.sub shell script that recognizes the platform triplets. There is a list of operating system names and the Sortix operating system isn't in the upstream config.sub file yet, so we will need to add it ourselves. You can simply search the line that contains the `-aos*` operating system entry and append `-sortix*` to it: - | -aos* | -aros* \ + | -aos* | -aros* | -sortix* \ If you don't do this, you will receive mysterious errors about that the package doesn't know what Sortix is. ### Passing options and variables to configure and make ### However, some packages are known to be silly and requires further tricks to be ported. Fortunately, the `tixbuildinfo` file provides control of what arguments and environmental variables are given to `./configure` and to make: pkg.configure.args=--enable-bar --without-x pkg.configure.vars=gt_cv_locale_fr=false gt_cv_locale_ja=false pkg.make.vars=V=1 It is generally recommended to set V=1 when running Make, if the package is silly and doesn't actually report the exact executed commands to the terminal during the compilation. This often happens with packages that use libtool. The exact used commands are valuable for debugging purposes, if you run into a compile warning or a compile error. ### Building Out-Of-Directory ### If the package requires being built outside of the main source directory, you can easily enable this behaviour: pkg.configure.use-build-directory=true ### Changing Make Targets ### By default the make command will be run twice as `make all` and `make install`. However, in some cases you would want to customize which targets are invoked: pkg.make.build-target=all-libfoo pkg.make.install-target=install-libfoo ### Post-Install Command ### After the make install target has been run, you have the option of running a custom command, with the working directory being the temporary staging directory, before the package is fully packaged up. If you need to run multiple commands at this point, you will need to wrap them in a shell script: pkg.post-install.cmd=./srctix-post-install.sh ### `.la` files ### Packages that use libtool have a nasty habit of installing `.la` files into the system library directory. However, this format doesn't properly support cross-compilation and have a number of other problems. The better choice is simply to eradicate such files using a post build command: pkg.post-install.cmd=tix-eradicate-libtool-la You should check whether your port produces such `.la` files and use this post build command if you see any. ### `--with-sysroot`, `--with-build-sysroot` ### If the package supports the --with-sysroot configure option and it works correctly for cross-compilation, then you can take advantage of it: pkg.configure.with-sysroot=true Otherwise the build system will simply communicate the correct system root to the compiler through hidden environmental variables and other tricks (such as the cross-compiler's default system root). If the package supports also the --with-build-sysroot to support having one system-root at runtime and another at build-time, then you should enable both options: pkg.configure.with-sysroot=true pkg.configure.with-build-sysroot=true If you are in doubt whether this works, simply disable these options (they are disabled by default) and the default system-root tricks will do just fine. ### Wrapping configure and make ### If you are unlucky, the configure script is custom and doesn't support common options or have other flaws. In that case you best yell a lot at the developers and question whether we want to port such a package in the first place, if the developers can't get the build system right. However, you can often work-around such problems by wrapping the configure and make commands in custom shell scripts: pkg.configure.cmd=./srctix-configure.sh pkg.make.cmd=./srctix-make.sh You'll need to set the executable bit on these files and these scripts should emulate the standard configure and make tools and do whatever magic to work around the broken build system. This should be considered the last resort - you should check if there are other key-values you can set that works around the issue or whether patching the software solves the issue. ### CC, CXX ### If the package somehow fails to detect the correct compiler from the `--build`, `--host`, and `--target` configure options, but it honours the CC and CXX environmental variables, then you can set these keys and they will be set: pkg.make.needed-vars.CC=true pkg.make.needed-vars.CXX=true ### DESTDIR ### During the install phase, the DESTDIR environmental variable points to a temporary staging directory, which the port *must* use as an additional prefix. If the package does not honour this, please yell a lot at the developers and question whether we want to port such a package in the first place. You can possibly work around this by wrapping make and configure. ### make distclean ### The source tix will automatically be cleaned whenever it is built. If you are using the configure build system, then the `make distclean` target will be used. There currently is no way to override this, but you can wrap the make command to work-around this or add support to `tix-build` for configuration. Simple Makefile --------------- Unfortunately, there are not enough common conventions on how to write a "standard" makefile to the point where Tix can just use most makefiles without special `tixbuildinfo` magic. However, Tix does have a plain makefile as a buikld system. pkg.build-system=makefile However, such makefiles *must* follow a common interface and it must respect the environmental variables through which Tix expresses its wishes. Here is an example makefile that detects installation directories and the compiler correctly: # Determine where files will be installed at run-tine, PREFIX?=/usr/local EXEC_PREFIX?=$(PREFIX) BINDIR?=$(EXEC_PREFIX)/bin SBINDIR?=$(EXEC_PREFIX)/sbin LIBEXECDIR?=$(EXEC_PREFIX)/libexec BOOTDIR?=$(PREFIX)/boot DATAROOTDIR?=$(PREFIX)/share DATADIR?=$(DATAROOTDIR) SYSCONFDIR?=$(PREFIX)/etc SHAREDSTATEDIR?=$(PREFIX)/com LOCALSTATEDIR?=$(PREFIX)/var RUNSTATEDIR?=$(LOCALSTATEDIR)/run INCLUDEDIR?=$(PREFIX)/include DOCDIR?=$(DATAROOTDIR)/doc INFODIR?=$(DATAROOTDIR)/info HTMLDIR?=$(DOCDIR) DVIDIR?=$(DOCDIR) PSDIR?=$(DOCDIR) PDFDIR?=$(DOCDIR) PSDIR?=$(DOCDIR) LIBDIR?=$(EXEC_PREFIX)/lib LOCALEDIR?=$(DATAROOTDIR/locale MANDIR?=$(DATAROOTDIR)/man MAN1DIR?=$(MANDIR)/man1 MAN2DIR?=$(MANDIR)/man2 MAN3DIR?=$(MANDIR)/man3 MAN4DIR?=$(MANDIR)/man4 MAN5DIR?=$(MANDIR)/man5 MAN6DIR?=$(MANDIR)/man6 MAN7DIR?=$(MANDIR)/man7 MAN8DIR?=$(MANDIR)/man8 MAN9DIR?=$(MANDIR)/man9 MANEXT?=.1 MAN1EXT?=.1 MAN2EXT?=.2 MAN3EXT?=.3 MAN4EXT?=.4 MAN5EXT?=.5 MAN6EXT?=.6 MAN7EXT?=.7 MAN8EXT?=.8 MAN9EXT?=.9 # An additional prefix during the installation. DESTDIR?= # Determine which compilation tools to use. HOST_TOOL_PREFIX?=$(if $(HOST),$(HOST)-,) DEFAULT_CC:=$(HOST_TOOL_PREFIX)gcc DEFAULT_CXX:=$(HOST_TOOL_PREFIX)g++ CC?=$(DEFAULT_CC) CXX?=$(DEFAULT_CXX) # Defaults for common variables if not set in the environment. CFLAGS?=-O2 -g CXXFLAGS?=-O2 -g CPPFLAGS?=-DNDEBUG LDFLAGS?= LIBS?= # Required options in common variables. CFLAGS:=$(CFLAGS) -Wall -Wextra CXXFLAGS:=$(CXXFLAGS) -Wall -Wextra CPPFLAGS:=-Iinclude LDFLAGS:=$(LDFLAGS) LIBS:=$(LIBS) -lqux PROGRAMS=foo bar all: $(PROGRAMS) .PHONY: all install clean distclean %: %.c $(CC) (CPPFLAGS) $(CFLAGS) $< -o $@ $(LDFLAGS) $(LIBS) %: %.c++ $(CXX) (CPPFLAGS) $(CXXFLAGS) $< -o $@ $(LDFLAGS) $(LIBS) install: all install $(PROGRAMS) $(DESTDIR)$(BINDIR) clean: rm -f $(PROGRAMS) distclean: clean This is a bit verbose. Simply remove the variables you don't need, but keep in mind that the logic that determines the remaining variables is important. The correct design here is that the makefile respects environmental variables with standard names whenever they are present. Default values are assigned to the environmental variables using ?= to make sure the defaults takes precedence. The logic that determines the compiler is important, as the `HOST` environmental variable contains the platform triplet of the platform that the program will run on. You should not assume that you can execute the programs you compile, as this simple scheme doesn't allow compiling for multiple platforms. In that case, we would need additional variables such as `BUILDCC` and `HOSTCC`. Additionally, it is important to implement the standard targets here (all, install, clean, and distclean). If everyone sticks to simple programs with makefiles that implement this simple interface correctly, then the process of compiling a given project is much simpler. Tix will currently compile such a makefile correctly and will continue to, but it doesn't set all the environmental variables at this point -- it may set then in the future, however. Testing the Port ---------------- You can now proceed to attempt to compile your port. The simplest solution is simply to rebuild the system along with all its ports, as usual. Keep in mind that you will likely run into compile warnings or errors. It might be useful to pass the `-O` or '--output-sync` option to Make to ensure the output is consistent, even in the face of parallel execution - if you are using the -j make option. If everything goes well, congratulations, you just ported a piece of software to Sortix. You should test it out and see if it works as expected and whether the port can be improved -- be sure to follow the rest of the guide and publish it to the Sortix community. If things didn't work, it's time to troubleshoot and examine whether the port is actually possible and what modifications will have to be made, as described in the next section. Porting the Software -------------------- Alright, so things didn't work in the first try. Tough luck. A lot of packages do the same silly things that makes it harder to port them. You'll have to be creative to determine the correct strategy for completing the port, sometimes you will have to implement features in Sortix or add some compatibility. Other times, you can work around the build system or patch the software. Try ask around and examine other archived port tixes and see their patches reveal any useful advise on how to resolve the situation: Here is a list of common problems: ### Configure fails ### Try and examine the config.log file inside the build directory. At the bottom, there is a dump of environmental variables, but just above it is information about what particular test program failed. If you examine the ./configure script closely, you will find that most checks have an associated cache environmental variable. If a particular check fails erroneously or isn't cross-compilation compatible, you can work around this issue by modifying the `tixbuildinfo` file such that it declares these environmental variables. ### Running cross-compiled programs ### Some packages are silly and think they are always able to execute compiled test programs, but this is inherently untrue for cross-compilation. Fortunately most packages know better as cross-compilation is somewhat common. Still -- some packages still do this, sometimes even as part of the check whether the compiler works. The result is often that the process hangs, as Sortix executables are Linux speak for "just hang and do nothing" in many cases. You'll have to fix the package or somehow work around the problem. Don't forget to yell at the developers of the package and question whether we really want it. ### Broken Locale Check ### Some packages (like GNU tar) suffers from a defect where they scan for many locales by running cross-compiled programs (see above). This can be worked around by declaring the appropriate cache variables. ### Gnulib ### Yuck! There's a lot of dirty hacks here and everybody has partial copies of it integrated into the build system, but people rarely update such copies. As such, it'll continue to be a pain to repeatedly work around. You should really check existing ports and see if their patches solve this particular gnulib problem. The problems here are often nasty, as gnulib occasionally wants to access libc internals it has no business dealing with to work-around some obscure bug on long-forgotten systems. You can probably safely `#if defined(__sortix__)` at the relevant places where there seem to be no better solution. You may even run into gnulib adding a warning to the gets(3) prototype telling people to never use it. This ironically fails to compile on Sortix, which has no gets(3) function in the first place. ### PATH_MAX ### Sortix has no such limit and programs shouldn't rely on such a limit existing. Nonetheless, sometimes it's easier to work-around the problem rather than really fixing the problem: #if defined(__sortix__) && !defined(PATH_MAX) #define PATH_MAX 32768 #endif ### Other problems ### There's lots of problems that can arise when porting packages. While Sortix itself is at fault for many of them, as it is still young and much remains to be implemented, a lot of problems is the direct result of poor packages with cross-compilation problems or not adhering strictly enough to standards. It can be useful to maintain a branch of the main system, which has had a number of hacks applied for compatibility. Ideally, this allows the Sortix developers to know what compatibility problems needs to be addressed. Indeed, many features originally started out as compatibility hacks. Be also sure to consult the `obsolete-stuff` document, as it enumerates a list of problematic APIs that we'd like to remove or refuse to implement - and what the modern replacement APIs are. Publishing the Port ------------------- Now that your port works, it's time to publish it and let the community enjoy your work. ### Cleaning the Source Directory ### Your source tix likely have a lot of left-over temporary files as the result of the testing phase. We'll need to clean the source directory before proceeding: cd $SORTIX_PORTS_DIR/libfoo && ./configure && make distclean Or perhaps you'll need to do something else. The important thing is that no object files and other problematic binary files are left behind. If the source tree is already configured, you can skip the ./configure step. ### Creating the Normalized Tree ### Unfortunately, some upstream releases are not actually distcleaned. This is a problem because the diff between the upstream release and your port will contain a lot of garbage. We'll then proceed to create a normalized tree that you can diff cleanly against. cd $SORTIX_PORTS_DIR && tar --extract --file $THAT_SAFE_LOCATION/libfoo-0.42.tar.xz && mv libfoo-0.42 libfoo.normalized & (cd libfoo.original && ./configure && colormake distclean) This should hopefully ensure that the two trees should be identical, except the few changes you had to make to port the package. Be mindful if the upstream developers are silly and don't put a single directory in their tarballs. You can then proceed to analyse the difference between the two trees: diff -Naur libfoo.normalized libfoo If everything went well, you should only see your changes. If there are other changes, you should resolve the situation by deleting files from the normalized tree, or by copying files from the normalized tree into the patched tree. Be mindful that diff(1) and patch(1) doesn't preserve the executable bit. Don't put a `tixbuildinfo` into the normalized tree, or it might unexpectedly become a source tix. ### Creating the Archived Port Tix ### The next step is to create the archived port tix, which contains the upstream tarball and the patches done to it (including the normalization step). This file allows others to easily review your port and ensure you have not made any malicious changes. It will also allow others to recreate the normalized tree and continue further development of the port. To create the archived port tix, you simply have to invoke this command: cd $SORTIX_PORTS_DIR && porttix-create --tarball $THAT_SAFE_LOCATION/libfoo-0.42.tar.xz \ --normalized libfoo.normalized \ libfoo This will create a `libfoo.porttix.tar.xz` file, which is publishable. It is not, however, ideal for simple extraction as a source tix. ### Creating the Archived Source Tix ### In concept, it's easy to create an archived source tix: You simply tar up the source tix you previously created. However, while it is easy to convert an archived port tix into an archived source tix, it is not possible to go in the other direction. It is recommended to create a port tix as it forces you to carefully consider all the applied patches and is respectful to the community. As mentioned, it is easy to convert the `.porttix.tar.xz` file into the desired archived source tix: srctix-create libfoo.porttix.tar.xz This will create an suitabe `libfoo.srctix.tar.xz` in the current directory, which is an archive containing the normalized tree with all patches applied. When reviewing, be sure to verify the included tarball in the archived port tix is entirely identical to the upstream release byte-for-byte to avoid security problems. ### Publishing ### Now that you have completed your port and built the archived port tix, it is time to share your work with the Sortix community. The simplest solution is sending the archived port tix (not the archived source tix) to the Sortix developers. They will then review your work and graciously publish your port through the appropriate channels. Alternatively, you can upload the port tix and the source tix to your own site and maintain it as a third party port. Conclusion ---------- This should be a basic walk-through the process of porting a piece of software to the Sortix operating system. Note how the porting facilities are experimental and much is subject to change in the near future. Porting software is an advanced topic and this documentation merely touches the more common situations and surely important advise is missing from this documentation. It's always a good idea to consult the Sortix developers for advise.