northport-v0.3.0

v0.3.0

• Added 'interrupt router' to replace previous interrupt manager, which offloads memory allocation to the caller.

  • It also stores in-use interrupt vectors sparsely, no more big allocation up front which may go unused.

• System clock handles events being inserted at the front of the queue properly.

  • Clock events now queue a DPC upon expiry, rather than running the callback inside the interrupt handler.
  • Clock events use the generic "ScaledTime" format rather than only nanoseconds.

• Extracted runlevel management from scheduler, and management of thread/process lifetime. The scheduler now just handles scheduling threads, as it should.

  • Scheduling and thread/process lifetime management are exposed through the driver API now.
  • Runlevel control and usage of DPCs is also available through the driver API.
  • Scheduler partially rewritten to make use of 'shared workqueues', where a group of cores can co-operatively share work between them. This replaces work stealing in the previous scheduler.
  • Waitable object updated to use new scheduler.
  • Removed previous event and filtering system.

• Added system uptime to stats display.

• Added IO subsystem, with support for stacking IO device APIs.

  • Device APIs can be layered, allowing transport drivers and device drivers to be combined (or not) as necessary.
  • IO subsystem uses 'packets' (referred to as IO packets/IOPs) to represent each operation and its status.
    • Currently these must be executed synchronously, but the API is built around them being async.

• Rewrite of driver subsystem: drivers are now external programs to the kernel, loaded from modules.

  • Subsystem now has 3 main moving parts and a manager within the kernel (DriverManager).
  • Driver manifests represent a driver program, the program can be loaded/unloaded as necessary.
  • Device descriptors represent an abstract piece of hardware or software within the system that the kernel doesn't understand how to use.
  • Device APIs are a table of function pointers provided by a driver to the kernel, so that the kernel can interact with a piece of hardware in a known way.
  • The workflow is that the driver manager matches unclaimed descriptors with driver manifests, and then marks the descriptor as under the control of that driver. Once the driver has performed some setup, it should export a number of device APIs back to the kernel, so that the hardware can be used.
  • Drivers can also register unclaimed descriptors, so support for new busses and transports can be easily added.
  • Some basic device APIs were added: IO, framebuffer and filesystem.
  • Added concept of a 'driver shadow', which is a thread-local pointer that tracks whether the current thread is executing on behalf of a driver or not. It allows for tracking per-driver state as it were global, from the driver's point of view.
  • Added device node tree, which tracks the relationships between descriptors, device apis and loaded driver instances.
  • Added system power API, allowing for simple poweroff/reboot functions to be provided to kernel in liu of full power management support.

• Added np-driverlib library, which contains utility code for C++ drivers targetting the northport kernel. A large chunk of the supporting code for drivers has been moved here (and out of the kernel).

  • As a result the kernel source tree is slightly smaller.

• Added external drivers:

  • PCI driver moved from kernel to be an external driver. It currently supports ECAM (pci-over-mmio) and x86 port IO.
    • It will make use of a pci.ids file if available to resolve vendor/device ids to human readable strings.
  • Qemu/bochs extended VGA.
  • Qemu fw_cfg interface, both as a filesystem and a framebuffer for the ramfb file.
  • Removed previously built-in drivers: qemu/bochs vga, nvme, and virtio devices.

• Added ability to bake-in compile-time constants to the kernel, like the git commit the build is based on, c++ flags used to build it, as well as the kernel version.

  • Some of this information is shown on the stats display.

• LICENSE file is now copied to initdisk when building.

• Any enabled first-party drivers are added to the initdisk.

• Kernel symbol store now stores some type info about each symbol, like it's visibility (for linking purposes to other modules).

  • ELF modules now have their symbols added to the kernel symbol store.

• Rewrite of VFS, in order to support new driver API.

  • The previous VFS was mostly the same as the original northport, which assumed all drivers were inside the kernel and had access to language features (like inheritence).
  • New VFS is written with the device API as it's primary interface, and operations on the VFS happen through global functions that wrap the driver functions.
    • As a result of the global functions, some portions of operations are now done generically.
    • The rewritten tempfs uses the filesystem device API now.
  • New VFS uses VfsIds to refer to vfs nodes, whether they're loaded in memory or not. To compliment this, vfs nodes (and their metadata) are now cached in memory similar to file contents, and can be evicted from memory if the cache grows too large.
    • File metadata is now fully transient, file contents are not (yet).
  • FileCaches (used to store a file's contents in memory) can be located by a file's VfsId.

• VFS is exposed via kernel module API.

• PMM rewrite: the previous PMM had grown organically overtime, this iteration is a fresh implemenation with current goals in-mind.

  • Its external functionality is mostly the same as the previous one.

• Fixed message string not making it through panic sequence properly.

• Fixed misalignment bug when loading driver metadata causing entrypoint to sometimes be wrong.

• Fixed incorrect calculation of resident set size in AnonVmDriver.

• Fixed accidentally not using PIT for platform-wide timer if HPET is present.

• Fixed bug in pool heap which assumed that new regions of virtual memory were allocated with ascending base addresses.

• Fixed all kernel symbols being available for linking with drivers, only symbols explicitly made visible can be used now.

• Moved nanoprintf from kernel to syslib, so the same implementation is available to drivers. Previously we have separate versions per-program.

  • Nanoprintf implementation also updated to use the latest upstream version, at time of writing.
  • This includes upstreamed fix for dereferencing null strings when passed a length of 0.

• Fixed buffer overflow in logging subsystem.

• Updated to use latest limine binary branch (v7).

v0.2.0

• Updated to use limine v5:

  • Since the bootloader now supports riscv, the boot shim for that platform was removed. It'll be uploaded as a separate project for anyone who wants to use it.
  • With the shim being removed, the device tree parser no longer needs to operate with the 'no allocation' constraint, so it's been rewriten to be efficient and build a tree in memory. Previously it parsed chunks of the device tree for every operation. It operates like smoldtb without the statically allocated buffer.
  • Bootable ISO for riscv64 is now included in the releases tab.

• Documentation improvements:

  • Earlier chapters rewritten to be more consistent.
  • Added VMM chapter.

• Memory management stack improvements:

  • Added per-core caches for slab allocation in the kernel.
    • Functionally this is similar to magazines as described in 'magazines and vmem' from the BSD team.
    • This greatly reduces contention between cores when allocating memory.
  • VMM: fixed demang paging + SMP bug, general cleanup of vm driver interface.
    • Added 'zero paging': when demang paging a readonly page of all zeroes is mapped instead of nothing. This should help cause a few less page faults, and later on.
    • Added VFS driver: we can now memory map files, including expanding them. Private mappings aren't supported just yet, but there have been provisions made. The VFS VM driver also makes use of demand paging where appropriate.
    • Switched to a red-black tree for managing vm ranges (instead of a singly linked list). Managing free address space is also done with another rbtree instead of inferring which regions were in-use. This is a little easier to reason about at the cost of more runtime memory.
    • Now tracks basic statistics about the working and resident set sizes.

• Added development cache (develcache): limine and any OVMF firmware files for testing are downloaded and cached inside the project directly, reducing setup for getting started with the project.

  • This is opt-out, and you can point the build system to your own limine and OVMF firmware files if you prefer that.

• Added stats display to graphical terminals, inspired by it's use in Keyronex.

• Kernel/drivers: module loader and external drivers now fully suppoported.

  • Module loader supports ELF shared objects, rather than the conventional relocatable objects.
  • Proof of concept driver written. Previously implemented drivers will be ported at some point to use the new system.

• Kernel: rewrite of symbol lookup. Before symbols used to be generated after the kernel was linked, compiling the symbols into an ELF object and sticking that object at the end of the kernel file (so as not to shuffle existing symbol addresses around). This certainly worked, but it's not as clean as the newer version: which just parses the kernel file request from the bootloader.

• Kernel/riscv: now uses tp for core-local info instead of reading sscratch everytime. Thanks to xvanc and sigterm for the feedback.

• Kernel/riscv: support for acpi based initialization, in addition to the device tree.

• Kernel: added concept of system topology, currently logical and physical cores are tracked, are as NUMA domains and their corresponding physical cores and ...

northport-v0.2.0

v0.2.0

• Updated to use limine v5:

  • Since the bootloader now supports riscv, the boot shim for that platform was removed. It'll be uploaded as a separate project for anyone who wants to use it.
  • With the shim being removed, the device tree parser no longer needs to operate with the 'no allocation' constraint, so it's been rewriten to be efficient and build a tree in memory. Previously it parsed chunks of the device tree for every operation. It operates like smoldtb without the statically allocated buffer.
  • Bootable ISO for riscv64 is now included in the releases tab.

• Documentation improvements:

  • Earlier chapters rewritten to be more consistent.
  • Added VMM chapter.

• Memory management stack improvements:

  • Added per-core caches for slab allocation in the kernel.
    • Functionally this is similar to magazines as described in 'magazines and vmem' from the BSD team.
    • This greatly reduces contention between cores when allocating memory.
  • VMM: fixed demand paging + SMP bug, general cleanup of vm driver interface.
    • Added 'zero paging': when demand paging a readonly page of all zeroes is mapped instead of nothing. This should help cause a few less page faults, and later on.
    • Added VFS driver: we can now memory map files, including expanding them. Private mappings aren't supported just yet, but there have been provisions made. The VFS VM driver also makes use of demand paging where appropriate.
    • Switched to a red-black tree for managing vm ranges (instead of a singly linked list). Managing free address space is also done with another rbtree instead of inferring which regions were in-use. This is a little easier to reason about at the cost of more runtime memory.
    • Now tracks basic statistics about the working and resident set sizes.

• Added development cache (develcache): limine and any OVMF firmware files for testing are downloaded and cached inside the project directly, reducing setup for getting started with the project.

  • This is opt-out, and you can point the build system to your own limine and OVMF firmware files if you prefer that.

• Added stats display to graphical terminals, inspired by it's use in Keyronex.

• Kernel/drivers: module loader and external drivers now fully suppoported.

  • Module loader supports ELF shared objects, rather than the conventional relocatable objects.
  • Proof of concept driver written. Previously implemented drivers will be ported at some point to use the new system.

• Kernel: rewrite of symbol lookup. Before symbols used to be generated after the kernel was linked, compiling the symbols into an ELF object and sticking that object at the end of the kernel file (so as not to shuffle existing symbol addresses around). This certainly worked, but it's not as clean as the newer version: which just parses the kernel file request from the bootloader.

• Kernel/riscv: now uses tp for core-local info instead of reading sscratch everytime. Thanks to xvanc and sigterm for the feedback.

• Kernel/riscv: support for acpi based initialization, in addition to the device tree.

• Kernel: added concept of system topology, currently logical and physical cores are tracked, are as NUMA domains and their corresponding physical cores and memory.

• Moved to internal stdint.h/stddef.h headers, meaning no external headers are required for building. This removes the need for management external sysroots and platform-specific headers.

v0.1.0

• Initial release using git tags.
• Added new kernel features:
  • VFS, follows a traditional unix design.
  • File cache, caches chunks of file contents in memory.
  • TempFS driver, operates like any other filesystem just with no backing storage.
• Releases include a bootable x86_64 iso for use with your emulator of choice now.
  • A risc-v version will follow soon.

northport-v0.1.0

• Initial release using git tags.
• Added new kernel features:
  • VFS, follows a traditional unix design.
  • File cache, caches chunk of file contents in memory.
  • TempFS driver, operates like any other filesystem just with no backing storage.
• Releases include a bootable x86_64 iso for use with your emulator of choice now.
  • A risc-v version will follow soon.