FOnline Engine

Engine currently in completely unusable state due to heavy refactoring.
Estimated finishing date is middle of 2021.

Table of Content

• Features
• Media
• Usage
  • Workflow
  • Public API
  • Setup
    • Windows Subsystem for Linux
    • Visual Studio Code
    • Package dependencies
    • Statically linked packages
  • Footprint
  • Tutorial
• Work in progress
  • Roadmap
  • Roadmap for Visual Studio Code extension
  • Code refactoring plans
  • Todo list (generated from source code)
• Repository structure
• Frequently Asked Questions
• Help and support

Features

• Isometric graphics (Fallout 1/2/Tactics or Arcanum -like games)
• Multiplayer mode with authoritative server
• Singleplayer mode (work without distinguish between server and client sides)
• Supporting of hexagonal and square map tiling
• Perendered sprites for environment but with possibility of using 3D models for characters
• Engine core written in C++ (favored C++17 standard)
• Flexible scripting system with varies supporting languages:
  • Mono C# (modern and safe scripting)
  • Native C++ (for performance critical places)
  • AngelScript (engine legacy scripting support)
• Cross-platform with target platforms:
  • Windows
  • Linux
  • macOS
  • iOS
  • Android
  • Web
  • PlayStation
  • UWP (PC, Mobile, Xbox)
• Supporting of following asset file formats:
  • Fallout 1/2
  • Fallout Tactics
  • Arcanum
  • 3D characters in modern graphic formats (FBX)
  • And common graphics formats like PNG or TGA

Important note: Not all from described above features are already implemented, for additional information look at 'Work in progress' section below.

Media

Usage

Repository contains source code of engine, third-party sources and build tools for composing all this stuff into final platform-specific bundles.
You may work on your game using shell scripts manually but project hosts own extension for Visual Studio Code for simplify these things.
Shell scripts targeted for work under Windows 10 within WSL2 and Ubuntu-20.04 as distro.
Almost all will be work under native Linuxes but some of scripts (like build.sh win32) must be run only from WSL2 shell because runs Windows binaries.
So main point of all of this that you build your game fully from source, there is no prebuilt binaries, full control over the process.

Workflow

As described above all what you need to build and package your game in one place for different platforms is WSL2.
You may do it in separate environments (like build Windows binaries in your IDE, build macOS/iOS binaries on macOS and rest on native Linux distro) but better do it in one place.

Process of creating your game in two words looks like this:

• Once prepare workspace where all intermediate build files will be stored
• Build executables from source to platforms that you needed
• Compile scripts (AngelScript and/or Mono; but Native were compiled with built executables)
• Bake all resources (shaders, images and etc) to special formats that will be loaded super fast by server/client
• Package built executables and baked resources from steps above to final platform specific bundle (zip, msi, app, apk and etc)
• Enjoy your shipped game and iterate development

There are couple of shell scripts that help us to do it:

• BuildTools/prepare-workspace.sh - prepare our workspace to futher work (install linux packages, setup emscripten, download android ndk and etc)
• BuildTools/build.sh - build executable for specific platform
• BuildTools/compile-scripts.sh - compile scripts from scripting layers
• BuildTools/bake-resources.sh - bake game assets (images, shaders, scripts, models and etc) to special intermediate formats
• BuildTools/make-packages.sh - finally package server(s) and client(s) for end user
• BuildTools/validate.sh and BuildTools/validate.bat - that scripts designed for validate that our sources compiling in general; you don't need that scripts and they need for automatic checking of repo consistency and run from ci/cd system like github actions

Scripts can accept additional arguments (build.sh for example accept platform for build for) and this information additionaly described in BuildTools/README.md.

Public API

Todo: write about versioning SemVer https://semver.org and what public API included to itself
Documents related to public API:

• Public API
• Multiplayer Script API
• Singleplayer Script API
• Mapper Script API

Setup

Clone with git this repository.
Run ./fonline-setup in the repository root in your PowerShell.
This script interactively check your system for all requirements and helps to generate new project.
Todo: Provide additional info for non Windows 10 users

Windows Subsystem for Linux

Main point of WSL2 for us that we can run Windows programs from Linux.
That feature allows unify almost all our build scripts into one environment.
Minimum version of Windows 10 is 2004 build 19041 from May 2020.
Recommended Linux distro is Ununtu-20.04 on which all build scripts tested.
You may use other distro but there is no guarantee that it will work out of the box.

Visual Studio Code

Engine hosts own Visual Studio Code extension for simplify work with engine stuff.
In editor go to the Extensions tab and then find and install 'FOnline' extension (it's already available in marketplace).
Extension activates automatically when editor finds any file that contains fonline in name of any file at workspace root.
More about extension usage you can find in Tutorial document.

Package dependencies

Following Linux packages will help us build our game for target platforms.
These packages will automatically installed during workspace preparing (i.e. prepare-workspace.sh).

• Common:
  clang clang-format build-essential git cmake python wget unzip
• Building for Linux:
  libc++-dev libc++abi-dev binutils-dev libx11-dev freeglut3-dev libssl-dev libevent-dev libxi-dev curl
• Building for Web:
  nodejs default-jre
• Building for Android:
  android-sdk openjdk-8-jdk ant
• macOS/iOS cross-compilation within OSXCross:
  patch lzma-dev libxml2-dev llvm-dev uuid-dev

Also our build scripts download and install following packages:

• Emscripten - for building Web apps
• OSXCross - cross-compilation for macOS/iOS
• Android NDK - compilation for Android devices

And fonline-setup.ps1 might install following Windows packages for you (some optional):

• Chocolatey - package manager for Windows system (helps to install other packages automatically)
• CMake - utility that helps build program from source on any platform for any platform without much pain
• Visual Studio 2019 - IDE for Windows
• Build Tools for Visual Studio 2019 - just build tools without full IDE
• Visual Studio Code - IDE for Windows with supporting of our engine management
• WiX Toolset - building installation packages (like .msi)

Other stuff used in build pipeline:

• Android CMake Toolchain
• iOS CMake Toolchain
• msicreator

Statically linked packages

These packages included to this repository, will compile and link statically to our binaries.
They are located in ThirdParty directory.

• ACM by Abel - sound file format reader
• AngelScript - scripting language
• Asio - networking library
• backward-cpp - stacktrace obtaining
• Catch2 - test framework
• GLEW - library for binding opengl stuff
• glslang - glsl shaders front-end
• Json - json parser
• diStorm3 - library for low level function call hooks
• PNG - png image loader
• SDL2 - low level access to audio, input and graphics
• SHA1 by Steve Reid - hash generator
• SHA2 by Olivier Gay - hash generator
• SPIRV-Cross - spir-v shaders to other shader languages converter
• Theora - video library
• Vorbis - audio library
• FBX SDK - fbx file format loader
• {fmt} - strings formatting library
• Dear ImGui - gui library
• libbson - bson stuff
• MongoC Driver - mongo db driver
• Mono - c# scripting library
• libogg - audio library
• openssl - library for network transport security
• unqlite - nosql database engine
• websocketpp - websocket asio extension
• zlib - compression library

Footprint

Despite on many third-party libraries that consumed by the whole engine one of the main goal is small final footprint of client/singleplayer output.
Aim to shift most of things of loading specific image/model/sound/ect file formats at pre publishing steps and later use intermediate binary representation for loading resources in runtime as fast as possible and without additional library dependencies.
This process in terms of fonline engine called Bakering.
Also as you can see all third-party dependencies linked statically to final executable (except one - proprietary libfbxsdk.dll/so for Baker but the last one is not target for small footprint) and this frees up end user from installing additional runtime to play in your game.
Todo: write about memory footprint
Todo: write about network footprint

Tutorial

Please follow these instructions to understand how to use this engine by design:

• Tutorial document

Work in progress

Roadmap

First release version:

• Rework scripting system (improve Native C++, AngelScript and C#/Mono scripting layers)
• Improve DirectX rendering with Universal Windows Platform
• Major part of code refactoring (look at separate section below)
• Improve Singleplayer mode (at least stable from architectural point of view)
• Multiplayer mode stabilization
• Documentation (all public API, tutorial, but core engine not included)
• API freezing and continuing development with it's backward compatibility

Futher releases:

• Finish with code refactoring (see below)
• Particle system
• Singleplayer mode stabilization
• Tasks related to Visual Studio Code extension (see separate section below)
• Improve supporting of PlayStation
• Improve Metal rendering for macOS/iOS
• Integration with Unity engine (research)

Visual Studio Code extension

• Integrate mapper (as javascript module) for editing .fomap
• Integrate dialog editor for editing .fodlg
• Integrate some property grid for protos editing
• Integrate server (as separate process but render ui in editor)
• Integrate client (as javascript module)
• Improve viewers for supported graphic formats (frm, spr, png, fofrm and etc)
• Add supporting of AngelScript language (highlight, auto-completion, compilation)
• Take and tune some of extensions for C# and C++ scripting
• Improve debugging of code (core and scripting)
• Improve debugging of game logic (like run game on this map with these scripts)
• Integrate gui editor for editing .fogui
• Add snippets for common tasks (like create map, create script, create proto)

Code refactoring plans

• Move errors handling model from error code based to exception based
• Eliminate singletons, statics, global functions
• Preprocessor defines to constants and enums
• Eliminate raw pointers, use raii and smart pointers for control objects lifetime
• Hide implementation details from headers using abstraction and pimpl idiom
• Fix all warnings from PVS Studio and other static analyzer tools
• Improve more unit tests and gain code coverage to at least 80%
• Improve more new C++ features like std::array, std::filesystem, std::string_view and etc
• Decrease platform specific code to minimum (we can leave this work to portable C++ or SDL)
• C-style casts to C++-style casts
• Add constness as much as necessary

Todo list (generated from source code)

• Common: rework all commented code during refactoring
• Common: make entities positioning free in space, without hard-linking to hex
• Common: add third 'up' coordinate to positioning that allow create multidimensional maps
• Common: use Common.h as precompiled header
• Common: use smart pointers instead raw
• Common: fix all PVS Studio warnings
• Common: SHA replace to openssl SHA
• Common: add #undef for every local #define
• Common: improve valgrind
• Common: add behaviour for SDL_WINDOW_ALWAYS_ON_TOP
• Common: move defines to const and enums
• Common: don't use rtti/typeid and remove from compilation options?
• Common: wrap fonline code to namespace?
• Common: fix build warnings for all platforms
• Common: enable threating warnings as errors
• Common: id and hash to 8 byte integer
• Common: research about std::string_view
• Common: research about std::filesystem
• Common: compile with -fpedantic
• Common: c-style arrays to std::array
• Common: use more STL (for ... -> auto p = find(begin(v), end(v), val); find_if, begin, end...)
• Common: use par (for_each(par, v, [](int x))
• Common: improve some single standard to initialize objects ({} or ())
• Common: add constness as much as nessesary
• Common: iterator -> const_iterator, auto -> const auto
• Common: use using instead of typedef
• Common: rework unscoped enums to scoped enums
• Common: use more noexcept
• Common: use more constexpr
• Common: improve BitReader/BitWriter to better network/disk space utilization
• Common: organize class members as public, protected, private; methods, fields
• Common: prefer this construction if(auto i = do(); i < 0) i... else i...
• Common: improve std::to_string or fmt::format to string conversions
• Common: cast between numeric types via numeric_cast(from)
• Common: minimize platform specific API (ifdef FO_os, WinApi-Include.h...)
• Common: clang debug builds with sanitiziers
• Common: time ticks to uint64
• Common: improve custom exceptions for every subsustem
• Common: improve particle system based on SPARK engine
• Common: research about Steam integration
• Common: speed up content loading from server
• Common: temporary entities, disable writing to data base
• Common: RUNTIME_ASSERT to assert
• Common: move all return values from out refs to return values as tuple and nodiscard (and then use structuured binding)
• Common: remove dynamic_cast?
• Common: split meanings if int8 and char in code
• Common: move from 32 bit hashes to 64 bit
• Common: rename uchar to uint8 and use uint8_t as alias
• Common: rename ushort to uint16 and use uint16_t as alias
• Common: rename uint to uint32 and use uint32_t as alias
• Common: rename char to int8 and use int8_t as alias
• Common: rename short to int16 and use int16_t as alias
• Common: rename int to int32 and use int32_t as alias
• Common: replace depedency from Assimp types (matrix/vector/quaternion/color)
• Common: auto expand exception parameters to readable state
• Common: recursion guard for EventDispatcher
• Common: improve ptr<> system for leng term pointer observing
• Common: add _hash c-string literal helper
• Common: move WriteData/ReadData to DataWriter/DataReader
• Common: fix TRect Width/Height
• Common: move NetProperty to more proper place
• Common: eliminate as much defines as possible
• Common: convert all defines to constants and enums
• Common: remove all id masks after moving to 64-bit hashes
• Common: remove critter flags
• BakerApp: sound and video preprocessing move to baker
• BakerApp: add dialogs verification during baking
• ServerApp: fix data racing
• ServerServiceApp: convert argv from wchar_t** to char**
• 3dAnimation: add interpolation for tracks more than two
• 3dStuff: add reverse playing of 3d animation
• 3dStuff: process default animations
• 3dStuff: GetAnim1/GetAnim2 int to uint return type
• 3dStuff: fix AtlasType referencing in 3dStuff
• Client: handle mouse wheel
• Client: synchronize effects showing (for example shot and kill)
• Client: need attention!
• Client: fix soft scroll if critter teleports
• Client: add working in IPv6 networks
• Client: rename FOClient to just Client (after reworking server Client to ClientConnection)
• CritterView: migrate critter on head text moving in scripts
• CritterView: do same for 2d animations
• HexManager: optimize lighting rebuilding to skip unvisible lights
• HexManager: need attention! (3)
• HexManager: move HexManager to MapView?
• Keyboard: merge Keyboard into App::Input and Client/Mapper
• ResourceManager: why I disable offset adding?
• SpriteManager: restore texture saving
• SpriteManager: improve DirectX rendering
• SpriteManager: maybe restrict fps at 60?
• SpriteManager: optimize sprite atlas filling
• SpriteManager: improve client rendering brightness
• SpriteManager: move fonts stuff to separate module
• Sprites: MapSprite releasing
• Sprites: : incapsulate all sprite data
• Application: move different renderers to separate modules
• Application: remove GLEW and bind OpenGL functions manually
• Application: map all framebuffer ext functions
• Application: recognize tablet mode for Windows 10
• Application: split ModelBuffer by number of supported bones (1, 5, 10, 20, 35, 54)
• CacheStorage: store Cache.bin in player local dir for Windows users?
• CacheStorage: add in-memory cache storage and fallback to it if can't create default
• Entity: fix proto name recognition
• Entity: use passkey for SetId
• FileSystem: handle apply file writing
• GenericUtils: script handling in ConvertParamValue
• KeyCodes-Include: rename DIK_ to Key or something else
• Log: server logs append not rewrite (with checking of size)
• Log: add timestamps and process id and thread id to file logs
• Log: delete \n appendix from WriteLog
• MapLoader: restore supporting of the map old text format
• MapLoader: pass errors vector to MapLoaderException
• MapLoader: remove mapper specific IsSelected from MapTile
• MessageBox: fix workaround for strange behaviour of button focus
• MsgFiles: move loading to constructors
• Properties: don't preserve memory for not allocated components in entity
• Properties: pack bool properties to one bit
• ScriptSystem: rework FONLINE_
• ScriptSystem: fill settings to scripts
• ScriptSystem: rework FONLINE_
• Settings-Include: rework global Quit setting
• Settings: exclude server specific settings from client
• Settings: remove VAR_SETTING; must stay only constant values
• StringUtils: restore hash parsing
• StringUtils: make isNumber const
• Testing: restore stack trace dumping in file
• Testing: improve global exceptions handlers for mobile os
• Testing: fix script system
• Testing: exclude using of dynamic memory allocation in this module and decrease chance of exception throwing
• Testing: send client dumps to server
• MonoScripting: set Mono domain user data
• MonoScripting: get Mono domain user data
• ScriptApi-Client: solve recursion in GetMapPos
• ScriptApi-Client: need attention!
• ScriptApi-Common: fix script system
• ScriptApi-Mapper: need attention! (4)
• ScriptApi-Server: fix ITEM_ACCESSORY_CONTAINER recursion
• ScriptApi-Server: need attention!
• ScriptApi: add FO_API_VIRTUAL_PROPERTY and FO_API_VIRTUAL_READONLY_PROPERTY
• ScriptApi: add supporting of components for properties
• ScriptApi: split FO_API_SETTING by subgroups (CLIENT, SERVER, RENDER, etc)
• ScriptApi: improve RPC calls
• ScriptApi: add custom entities handling
• ScriptApi: add Proto* to scripts that have readable properties
• ScriptApi: add local entity events
• ScriptApi: add ImGui bindings
• ScriptApi: hex coords to single structure (hx, hy -> coord)
• ScriptApi: remove for better portability (2)
• AdminPanel: admin panel network to Asio
• Critter: rework Client class to ClientConnection
• CritterManager: don't remeber but need check (IsPlaneNoTalk)
• Dialogs: check item name on DR_ITEM
• Location: encapsulate Location data
• MapManager: if path finding not be reworked than migrate magic number to scripts
• MapManager: check group
• Networking: catch exceptions in network servers
• Server: restore hashes loading
• Server: restore settings (2)
• Server: disable look distance caching
• Server: remove from game (2)
• Server: control max size explicitly, add option to property registration
• Server: disable send changing field by client to this client
• Server: don't remeber but need check (IsPlaneNoTalk)
• Server: add container properties changing notifications
• Server: make BlockLines changable in runtime
• Server: rename FOServer to just Server
• Server: run network listeners dynamically, without restriction, based on server settings
• ImageBaker: finish with GLSL to SPIRV to GLSL/HLSL/MSL
• ImageBaker: add supporting of APNG file format
• ImageBaker: swap colors of fo palette once in header
• Mapper: need attention! (24)
• Mapper: mapper render iface layer
• Mapper: rename FOMapper to just Mapper

Repository structure

• BuildTools - scripts for automatical build in command line or any ci/cd system
• Resources - resources for build applications but not related to code
• Source - fonline engine specific code
• ThirdParty - external dependencies of engine, included to repository

Frequently Asked Questions

Following document contains some issues thats give additional information about this engine:

• FAQ document

Help and support

• Site: fonline.ru
• GitHub: github.com/cvet/fonline
• E-Mail: cvet@tut.by