Ghaf security hardening #661

gangaram-tii · 2024-06-11T12:57:26Z

Hardening of Ghaf security

Security modules
Integration of third party security modules (e.g. FireJail, ClamAV, Fail2Ban etc)
Options to enhance kernel security
Enabled basic security in VMs

Checklist for things done

Testing

Tested basic system and security functionality on Lenovo X1.
What to test?
- Functional testing on all platforms
- Impact on overall system performance

- Added Apparmor configuration for Chromium and Firefox Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

- A intrusion prevention software framework - Bans IP addresses conducting too many failed login attempts. Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

- Used to sandbox untrusted applications Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

modules/common/security/firejail/default.nix

humaidq-tii · 2024-06-12T05:42:40Z

modules/common/security/firejail/default.nix

+          executable = "${lib.getBin pkgs.firefox}/bin/firefox";
+          profile = "${pkgs.firejail}/etc/firejail/firefox.profile";
+          extraArgs = [
+            "--whitelist=/run/current-system/sw/bin/firefox"


Wouldn't both ${lib.getBin pkgs.firefox}/bin/firefox and /run/current-system/sw/bin/firefox link to the same binary?

Also on the main branch, firefox is no longer added in the system path for the Orin, so it isn't found in /run/current-system/sw/bin

/run/current-system/sw/bin/firefox is jailed firefox, wraped by firejail.

But /run/current-system/sw/bin/firefox doesn't exist?

If you enable jailed firefox then it exists:
ghaf.security.firejail.enable = true;
ghaf.security.firejail.apps.firefox.enable = true;

It is not recommended to have have Apparmor and Firejail both enabled for same application. For Lenovo X1 we have Apparmor security for Chromium. Jailed firefox and chromium I had tested on vm and generic x86.

humaidq-tii · 2024-06-12T05:47:03Z

modules/common/security/networking.nix

+  config.boot.kernel.sysctl = lib.mkMerge [
+    (lib.mkIf cfg.ipsecurity.enable {
+      # Disable IPv6
+      "net.ipv6.conf.all.disable_ipv6" = lib.mkForce 1;


Disabling IPv6 benefit seems to be for making implementing filtering policies easier.

If IPv6 needs to be disabled for filtering, shouldn't that be done on corporate VPN side rather than on the OS?

I disabled IPv6 to reduce attack surface. I didn't think from VPN perspective.

Why would IPv6 be an attack surface? Is the kernel code for IPv6 implementation buggy?

We don't know what is buggy, what is not. More code means larger attack surface. Many security audits recommend to disable IPv6.

Looking at NIST, they have guidelines on secure deployment of IPv6 with NIST 800-119^1. Also NIST 800-53 did use an example of IPv4 as older technology and IPv6 as newer technology, but I can't seem to find any control or control enhancement that suggests disabling IPv6.

Is there some kind of guideline that I have missed that suggests disabling IPv6?

1: https://csrc.nist.gov/pubs/sp/800/119/final
2: https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final

https://www.tenable.com/audits/items/search?q=ipv6&sort=&page=1
https://www.safedns.com/userfiles/uploads/checklist%20linux.pdf
https://www.cyberciti.biz/tips/linux-security.html [section: 23]

For sure if we are going to use IPv6 then we need to enable this.

modules/common/security/system.nix

modules/microvm/virtualization/microvm/adminvm.nix

modules/common/security/system.nix

humaidq-tii · 2024-06-12T06:18:59Z

modules/common/security/system.nix

+    ## Linux security
+    (lib.mkIf cfg.system-security.enable {
+      services.openssh = {
+        extraConfig = lib.optionalString config.ghaf.profiles.release.enable ''


If release build doesn't enable openssh service, why are we configuring it?

May be openssh will be enabled to login in net-vm from different machine.

Is this a use case documented anywhere?

I am not sure but this is not going to harm. If openssh is enabled then this will enforce security otherwise it will not have any effect.

milva-unikie · 2024-06-12T06:22:36Z

Hi! I have a few questions about testing:

Do some of the added security modules have concrete impact on the system that we should verify by testing? Could you please add instructions to the Testing-section.
There are "Options to enhance kernel security" added. Should those be tested? If yes, could you also add how to enable them and their expected results to the Testing-section.
Testing was also asked for Orin and Polarfire. Are there any changes to be expected on these platforms, or just to make sure nothing was broken?

humaidq-tii · 2024-06-12T06:36:42Z

modules/common/security/system.nix

+        SCUDO_OPTIONS = lib.mkDefault "ZeroContents=1";
+      };
+
+      boot.tmp.useTmpfs = lib.mkForce true;


Does it make sense to use tmpfs in all VMs? Since it is using RAM and that can be scarce.
Maybe boot.tmp.cleanOnBoot is enough, and we can mount /tmp with noexec flag?

It is in our TODO list to mount some directories with appropriate permission. We'll revisit it when we'll do it.

Because I'm worried enabling tmpfs would cause the VMs to run out of RAM or /tmp is filled, causing bad side effects.

Why would enabling tmpfs cause VMs to run out of RAM? By default, the size of tmpfs in Nix is set to 50% of the available memory. I believe the opposite is true: if tmpfs is not enabled, all temporary storage will still occupy RAM without any limits imposed on it, potentially causing the VMs to run out of RAM. In the first scenario, an application may fail due to limited space in tmpfs, but the VM itself will remain responsive.

Here is an example. AppFlowy has only 512MB of RAM: https://github.com/tiiuae/ghaf/blob/main/modules/reference/appvms/appflowy.nix#L12

Is 256MB of /tmp enough? Can AppFlowy run with only 256MB of free RAM? Have we tested this?

gangaram-tii · 2024-06-13T11:07:45Z

@milva-unikie, Please see my comments inline:

Hi! I have a few questions about testing:

1. Do some of the added security modules have concrete impact on the system that we should verify by testing? Could you please add instructions to the Testing-section.

==> I have done few functionality tests for newly added security features. Please test overall functionality of the system to make sure nothing is broken in this PR. Also do performance testing, as security features may have some performance impact.

2. There are "Options to enhance kernel security" added. Should those be tested? If yes, could you also add how to enable them and their expected results to the Testing-section.

==> You need not to do security testing, we are working on security test plan.

3. Testing was also asked for Orin and Polarfire. Are there any changes to be expected on these platforms, or just to make sure nothing was broken?

==> On other platforms like Orin and Polarfire, please make sure nothing is broken.

humaidq-tii · 2024-06-13T11:32:29Z

modules/common/security/system.nix

+    (lib.mkIf cfg.system-security.enable {
+      services.openssh = {
+        extraConfig = lib.optionalString config.ghaf.profiles.release.enable ''
+          AllowTcpForwarding yes


Why is this needed for release build?

humaidq-tii · 2024-06-13T11:36:07Z

modules/common/security/system.nix

+      services.openssh = {
+        extraConfig = lib.optionalString config.ghaf.profiles.release.enable ''
+          AllowTcpForwarding yes
+          X11Forwarding no


X11 would not be included in Ghaf's reference implementation, then why are we configuring it?

humaidq-tii · 2024-06-13T11:43:22Z

modules/common/security/system.nix

+        "kernel.kexec_load_disabled" = lib.mkForce config.ghaf.profiles.release.enable;
+
+        # Disable ptrace
+        "kernel.yama.ptrace_scope" = lib.mkForce 3;


Default is 1, which is already restrictive. Restricting it further may break sandboxing applications?
https://www.kernel.org/doc/html/v4.15/admin-guide/LSM/Yama.html

Yes, if we are not breaking any thing, then shouldn't we keep restriction at highest level? We can reduce it to 2 or even 1 if it is needed.

Then we need to document this for the option system-security.enable, so whoever uses Ghaf library would know this setting is changed? Also instead of mkForce, does it make sense to use mkDefault or mkOverride (setting a priority)?

humaidq-tii · 2024-06-13T11:58:12Z

modules/common/security/system.nix

+      boot.kernelParams =
+        [
+          # Fill freed pages and heap objects with zeroes
+          "init_memory=0"


I'm trying to find out what parameter this is, my search engine only returns one result on medium.com. I checked Linux kernel source, also couldn't find anything. Could you please help me find where I can find a reference to this?

Oh! I don't know how I messed this up. I was supposed to use 'init_on_alloc' and 'init_on_free'.

humaidq-tii · 2024-06-13T12:00:56Z

modules/common/security/system.nix

+          "init_memory=0"
+
+          # Panic on any uncorrectable errors through the machine check exception system
+          "mce=0"


This seems to only be applicable for systems that use ECC memory, none of our reference devices have ECC memory.

Machine Check Exceptions (MCEs) are not exclusive to ECC memory; they can occur in various hardware scenarios beyond ECC memory alone. For reference platform it should be fine to keep it.

I see, is it only applicable for x86 systems?
https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

DDR5 will have some light form of ECC by default.

humaidq-tii · 2024-06-13T12:04:47Z

modules/common/security/system.nix

+        ]
+        ++ lib.optionals config.ghaf.security.network.ipsecurity.enable [
+          # Disable IPv6 to reduce attack surface.
+          "ipv6.disable=1"


Similar to the previous comment on disabling IPv6, I don't see why IPv6 is a vulnerability. This is going to be an issue as we slowly move away from IPv4.

I will check if we really need it. I have seen in security audits this is recommended to disable IPv6 to reduce attack vector until unless we are not using this.

humaidq-tii · 2024-06-13T12:10:24Z

modules/common/security/system.nix

+
+        # This will avoid unintentional writes to an attacker-controlled FIFO/Regular file.
+        # Extend the restriction to group sticky directories
+        "fs.protected_fifos" = lib.mkForce 0;


Wouldn't setting this to 0 (unrestricted) be counter-intuitive?
https://github.com/torvalds/linux/blob/2ccbdf43d5e758f8493a95252073cf9078a5fea5/Documentation/admin-guide/sysctl/fs.rst#protected_fifos

humaidq-tii · 2024-06-13T12:18:05Z

modules/common/security/firejail/default.nix

+        })
+      ];
+
+      launchers =


Shouldn't the launchers be defined inside of the reference apps module?

humaidq-tii · 2024-06-13T12:19:53Z

modules/common/security/firejail/default.nix

+      launchers =
+        lib.optional cfg.apps.firefox.enable {
+          name = "Firefox-safe";
+          path = "/run/current-system/sw/bin/firefox";


This path doesn't exist on the Orin. Shouldn't this be something like:
${pkgs.firefox}/bin/firefox
Or if we have our own version, we can directly refer to it ${firejailFirefox}/bin/firefox-safe

humaidq-tii · 2024-06-13T12:20:29Z

modules/common/security/firejail/default.nix

+        }
+        ++ lib.optional cfg.apps.chromium.enable {
+          name = "Chromium-safe";
+          path = "/run/current-system/sw/bin/chromium";


Same with above issue with firefox, we should refer directly to the path of the chromium binary. As this file might not exist.

humaidq-tii · 2024-06-13T12:24:17Z

modules/common/security/system.nix

+      sudo = {
+        enable = lib.mkOption {
+          type = lib.types.bool;
+          default = true;


This is already enabled by default, why is it redefined here?

And do we want sudo for release builds?

humaidq-tii · 2024-06-13T12:27:21Z

modules/common/security/system.nix

+        type = lib.types.bool;
+        default = true;
+      };
+      memory-allocator = lib.mkOption {


This option already exists upstream with environment.memoryAllocator.provider.

If we provide option for alternative allocators, it means we are supporting it. Were these tested? Is there reason behind the omission of graphene-hardened-light?

humaidq-tii · 2024-06-13T12:31:58Z

modules/common/security/system.nix

+      security.unprivilegedUsernsClone = lib.mkDefault false;
+
+      # Disable Hyperthreading (To reduce risk of side channel attack)
+      security.allowSimultaneousMultithreading = lib.mkDefault (!(cfg.system-security.misc.disableHyperthreading || cfg.system-security.misc.enable-all));


What do we benefit the users of Ghaf framework by providing this alias? And are we supporting this use case?

The end user can simply use security.allowSimultaneousMultithreading if they really want to do this. Even NixOS option documentation states:

[...] potentially at significant performance cost. [...] This attack vector is unproven.

humaidq-tii · 2024-06-13T12:40:40Z

modules/common/security/apparmor/profiles/chromium.nix

+  };
+  ## Apparmor profile for Chromium
+  config.security.apparmor.policies."bin.chromium" = lib.mkIf cfg.apps.chromium.enable {
+    profile =


How does this compare to: https://gitlab.com/apparmor/apparmor/-/blob/master/profiles/apparmor/profiles/extras/chromium_browser?ref_type=heads

Which is already packaged in NixOS (apparmor-profiles).

And if we have improvements to the Chromium apparmor policy, should it be upstreamed or included as a patch to apparmor-profiles package?

humaidq-tii · 2024-06-13T12:43:10Z

modules/common/security/apparmor/profiles/firefox.nix

+  config.security.apparmor.policies."bin.firefox" = lib.mkIf cfg.apps.firefox.enable {
+    profile =
+      ''
+        abi <abi/3.0>,


Just like Chromium, apparmor-profiles package also contains policy for firefox: https://gitlab.com/apparmor/apparmor/-/blob/master/profiles/apparmor/profiles/extras/firefox?ref_type=heads

I think if we package our own version, it should be clear how it is different than the profile that is packaged in NixOS?

humaidq-tii · 2024-06-13T12:49:27Z

modules/common/security/clamav/default.nix

+
+        # Always block cloaked URLs, even if URL isn't in database.
+        # This can lead to false positives.
+        PhishingAlwaysBlockCloak = "no";


This is already default set to "no".

humaidq-tii · 2024-06-13T12:54:46Z

modules/common/security/clamav/default.nix

+        HeuristicScanPrecedence = "yes";
+
+        # Enable the Data Loss Prevention module
+        StructuredDataDetection = "yes";


https://docs.clamav.net/manual/Development/libclamav.html#data-loss-prevention

detect the following credit card issuers: AMEX, VISA, MasterCard, Discover, Diner’s Club, and JCB and U.S. social security numbers inside text files.

How would this be beneficial to the end user? Would false positives mean that files that contain that pattern be quarantined? How would the user know?

humaidq-tii · 2024-06-13T12:55:47Z

modules/common/security/clamav/default.nix

+        StructuredDataDetection = "yes";
+
+        # Stop daemon when libclamav reports out of memory condition.
+        ExitOnOOM = "yes";


Should this be the job of systemd-oomd to determine whether it reaches out of memory?

Yes. It does stop the whole service in this case.

humaidq-tii · 2024-06-13T12:57:06Z

modules/common/security/clamav/default.nix

+
+        # With this option clamav will try to detect broken executables (both PE and
+        # ELF) and mark them as Broken.Executable.
+        DetectBrokenExecutables = "yes";


If the user-writable directories are all mounted with noexec, this would be redundant?

humaidq-tii · 2024-06-13T13:01:11Z

modules/common/security/clamav/default.nix

+    };
+    updater = lib.mkIf cfg.live-update {
+      # Enable live update of virus database
+      enable = true;


How would this behave on a system with impermanence? What if the "storage VM" have no internet access?

humaidq-tii · 2024-06-13T13:04:07Z

modules/common/security/fail2ban/default.nix

+
+  ## Enable fail2ban sandboxing
+  config = {
+    services.fail2ban = lib.mkIf cfg.enable {


Do we really need fail2ban for debug builds, especially with default password as ghaf?
If we disable password authentication, wouldn't that be enough?
Also why are we enabling fail2ban for release build, which has ssh disabled?

- use icons for jailed firefox and chromium from icon pack - users options set to default in all VMs - KASLR option removed as it is by default enabled in kernel - sysrq disabled only in release build - fail2ban enabled in net-vm only Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

mbssrc · 2024-06-14T12:45:12Z

I propose we make this a series of PRs, with individual purposes. That makes it easier
to discuss and refine. Perhaps we could start with:

Apparmor: Profiles require testing for performance (and security)
Fail2ban: At least some configurations will have sshd enabled, for administrator access.
Can be enabled in release only, if sshd is enabled.
Firejail: No a big fan but can be a reference service option. Opinions welcome.
User configuration (basic)
As the user setup and configuration is still work in progress, we can use the
basic settings that will likely be required across all systems, e.g., root user configuration.
We can also use the work that has been done already (users: creating login and system users #459).

For system, kernel parameters/sysctl settings, let's re-visit when we have the performance
data and security analysis ready, as well as the kernel parameter management to avoid double work.

Mic92 · 2024-06-24T16:01:29Z

modules/common/security/firejail/default.nix

+  ## Enable Firejail sandboxing
+  config = lib.mkIf cfg.enable {
+    ghaf.graphics = lib.mkIf config.ghaf.profiles.graphics.enable {
+      demo-apps = lib.mkMerge [


Most of the NixOS ecosystem uses camel case for options i.e.:

Suggested change

demo-apps = lib.mkMerge [

demoApps = lib.mkMerge [

It looks like ghaf is doing the same in most places.

Mic92 · 2024-06-24T16:06:43Z

modules/common/security/apparmor/default.nix

+
+  config = lib.mkIf cfg.enable {
+    security.apparmor.enable = true;
+    security.apparmor.killUnconfinedConfinables = lib.mkDefault true;


I was always a bit skeptical about this option when it was added to NixOS. Does firefox/chromium properly restarts itself afterwards?

gangaram-tii added 7 commits June 11, 2024 16:31

Enable Apparmor security

23265d8

- Added Apparmor configuration for Chromium and Firefox Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Integration of ClamAV anti-virus toolkit

74a555a

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Integration of Fail2Ban

86df98c

- A intrusion prevention software framework - Bans IP addresses conducting too many failed login attempts. Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Integrate FireJail security module

26854c1

- Used to sandbox untrusted applications Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Security module for networking

b31c1a8

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Module for system security

f753f2d

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

Enable basic security features in VMs

a5ffa7b

Signed-off-by: Ganga Ram <Ganga.Ram@tii.ae>

gangaram-tii temporarily deployed to internal-build-workflow June 11, 2024 12:57 — with GitHub Actions Inactive

gangaram-tii requested review from brianmcgillion, mbssrc, humaidq-tii and milva-unikie June 11, 2024 12:57

humaidq-tii requested changes Jun 12, 2024

View reviewed changes

humaidq-tii reviewed Jun 12, 2024

View reviewed changes

gangaram-tii force-pushed the __wip-nixos-hardening branch from a5ffa7b to f2162e3 Compare June 13, 2024 10:27

gangaram-tii temporarily deployed to internal-build-workflow June 13, 2024 10:27 — with GitHub Actions Inactive

gangaram-tii force-pushed the __wip-nixos-hardening branch from f2162e3 to a4481e6 Compare June 13, 2024 11:14

gangaram-tii temporarily deployed to internal-build-workflow June 13, 2024 11:14 — with GitHub Actions Inactive

humaidq-tii reviewed Jun 13, 2024

View reviewed changes

gangaram-tii force-pushed the __wip-nixos-hardening branch from a4481e6 to 74afc89 Compare June 14, 2024 10:26

gangaram-tii temporarily deployed to internal-build-workflow June 14, 2024 10:27 — with GitHub Actions Inactive

Mic92 reviewed Jun 24, 2024

View reviewed changes

Ghaf security hardening #661

Are you sure you want to change the base?

Ghaf security hardening #661

Conversation

gangaram-tii commented Jun 11, 2024

Checklist for things done

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gangaram-tii Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gangaram-tii Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milva-unikie commented Jun 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gangaram-tii commented Jun 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humaidq-tii Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humaidq-tii Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humaidq-tii Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

humaidq-tii Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

mbssrc commented Jun 14, 2024

Mic92 Jun 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gangaram-tii Jun 14, 2024 •

edited

Loading

gangaram-tii Jun 14, 2024 •

edited

Loading

humaidq-tii Jun 13, 2024 •

edited

Loading

humaidq-tii Jun 13, 2024 •

edited

Loading

humaidq-tii Jun 13, 2024 •

edited

Loading

humaidq-tii Jun 13, 2024 •

edited

Loading

Mic92 Jun 24, 2024 •

edited

Loading