url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init

[mweinelt]

Motivation for this change

Add some sorely needed metadata to IRC conversations.

<@hexa-> https://github.com/NixOS/nixpkgs/pull/105315
<urlmetadata> ⤷ url-bot-rs: init at 0.3.1; nixos/url-bot-rs: init by mweinelt · Pull Request #105315 · NixOS/nixpkgs · GitHub

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

  NAME                                                        DESCRIPTION                                                                    EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                                            0.5
✓ User=/DynamicUser=                                          Service runs under a transient non-root user identity                                  
✓ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service cannot change UID/GID identities/capabilities                                  
✓ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has no administrator privileges                                                
✓ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has no ptrace() debugging abilities                                            
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                               0.3
✓ RestrictNamespaces=~CLONE_NEWUSER                           Service cannot create user namespaces                                                  
✓ RestrictAddressFamilies=~…                                  Service cannot allocate exotic sockets                                                 
✓ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service cannot change file ownership/access mode/capabilities                          
✓ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service cannot override UNIX file/IPC permission checks                                
✓ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has no network configuration privileges                                        
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules                                                     
✓ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has no raw I/O access                                                          
✓ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes cannot change the system clock                                       
✗ DeviceAllow=                                                Service has a device ACL with some special devices                                  0.1
✗ IPAddressDeny=                                              Service does not define an IP address allow list                                    0.2
✓ KeyringMode=                                                Service doesn't share key material with other services                                 
✓ NoNewPrivileges=                                            Service processes cannot acquire new privileges                                        
✓ NotifyAccess=                                               Service child processes cannot alter service state                                     
✓ PrivateDevices=                                             Service has no access to hardware devices                                              
✓ PrivateMounts=                                              Service cannot install system mounts                                                   
✓ PrivateTmp=                                                 Service has no access to other software's temporary files                              
✓ PrivateUsers=                                               Service does not have access to other users                                            
✓ ProtectClock=                                               Service cannot write to the hardware clock or system clock                             
✓ ProtectControlGroups=                                       Service cannot modify the control group file system                                    
✓ ProtectHome=                                                Service has no access to home directories                                              
✓ ProtectKernelLogs=                                          Service cannot read from or write to the kernel log ring buffer                        
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules                                             
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)                                    
✓ ProtectSystem=                                              Service has strict read-only access to the OS file hierarchy                           
✓ RestrictAddressFamilies=~AF_PACKET                          Service cannot allocate packet sockets                                                 
✓ RestrictSUIDSGID=                                           SUID/SGID file creation by service is restricted                                       
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI                                  
✓ SystemCallFilter=~@clock                                    System call allow list defined for service, and @clock is not included                 
✓ SystemCallFilter=~@debug                                    System call allow list defined for service, and @debug is not included                 
✓ SystemCallFilter=~@module                                   System call allow list defined for service, and @module is not included                
✓ SystemCallFilter=~@mount                                    System call allow list defined for service, and @mount is not included                 
✓ SystemCallFilter=~@raw-io                                   System call allow list defined for service, and @raw-io is not included                
✓ SystemCallFilter=~@reboot                                   System call allow list defined for service, and @reboot is not included                
✓ SystemCallFilter=~@swap                                     System call allow list defined for service, and @swap is not included                  
✗ SystemCallFilter=~@privileged                               System call allow list defined for service, and @privileged is included             0.2
✗ SystemCallFilter=~@resources                                System call allow list defined for service, and @resources is included              0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities                                  
✓ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has no audit subsystem access                                                  
✓ CapabilityBoundingSet=~CAP_KILL                             Service cannot send UNIX signals to arbitrary processes                                
✓ CapabilityBoundingSet=~CAP_MKNOD                            Service cannot create device nodes                                                     
✓ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has no elevated networking privileges                                          
✓ CapabilityBoundingSet=~CAP_SYSLOG                           Service has no access to kernel logging                                                
✓ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has no privileges to change resource use parameters                            
✓ RestrictNamespaces=~CLONE_NEWCGROUP                         Service cannot create cgroup namespaces                                                
✓ RestrictNamespaces=~CLONE_NEWIPC                            Service cannot create IPC namespaces                                                   
✓ RestrictNamespaces=~CLONE_NEWNET                            Service cannot create network namespaces                                               
✓ RestrictNamespaces=~CLONE_NEWNS                             Service cannot create file system namespaces                                           
✓ RestrictNamespaces=~CLONE_NEWPID                            Service cannot create process namespaces                                               
✓ RestrictRealtime=                                           Service realtime scheduling access is restricted                                       
✓ SystemCallFilter=~@cpu-emulation                            System call allow list defined for service, and @cpu-emulation is not included         
✓ SystemCallFilter=~@obsolete                                 System call allow list defined for service, and @obsolete is not included              
✓ RestrictAddressFamilies=~AF_NETLINK                         Service cannot allocate netlink sockets                                                
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                                       0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups                                                    
✓ CapabilityBoundingSet=~CAP_MAC_*                            Service cannot adjust SMACK MAC                                                        
✓ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service cannot issue reboot()                                                          
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree                      
✓ LockPersonality=                                            Service cannot change ABI personality                                                  
✓ MemoryDenyWriteExecute=                                     Service cannot create writable executable memory mappings                              
✓ RemoveIPC=                                                  Service user cannot leave SysV IPC objects around                                      
✓ RestrictNamespaces=~CLONE_NEWUTS                            Service cannot create hostname namespaces                                              
✓ UMask=                                                      Files created by service are accessible only by service's own user by default          
✓ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service cannot mark files immutable                                                    
✓ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service cannot lock memory into RAM                                                    
✓ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service cannot issue chroot()                                                          
✓ ProtectHostname=                                            Service cannot change system host/domainname                                           
✓ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service cannot establish wake locks                                                    
✓ CapabilityBoundingSet=~CAP_LEASE                            Service cannot create file leases                                                      
✓ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service cannot use acct()                                                              
✓ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service cannot issue vhangup()                                                         
✓ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service cannot program timers that wake up the system                                  
✓ RestrictAddressFamilies=~AF_UNIX                            Service cannot allocate local sockets                                                  

→ Overall exposure level for url-bot-rs.service: 1.3 OK 🙂

@nuxeh

[mweinelt]

I'd totally be down for writing a test, but we'd need to spin up some IRCd. If someone can recommend an IRCd that I can spin up with just a few lines of config, that'd be great.

[SuperSandro2000]

Patch to make it build on darwin:

diff --git a/pkgs/applications/networking/irc/url-bot-rs/default.nix b/pkgs/applications/networking/irc/url-bot-rs/default.nix
index 97445029dce5..5fe7f631bc2b 100644
--- a/pkgs/applications/networking/irc/url-bot-rs/default.nix
+++ b/pkgs/applications/networking/irc/url-bot-rs/default.nix
@@ -1,9 +1,11 @@
-{ lib
+{ stdenv
 , rustPlatform
 , fetchFromGitHub
 , pkg-config
 , openssl
 , sqlite
+, libiconv
+, Security
 }:
 
 rustPlatform.buildRustPackage rec {
@@ -21,13 +23,13 @@ rustPlatform.buildRustPackage rec {
 
   nativeBuildInputs = [ pkg-config ];
 
-  buildInputs = [ openssl sqlite ];
+  buildInputs = [ openssl sqlite ] ++ stdenv.lib.optionals stdenv.isDarwin [ libiconv Security ];
 
   preBuild = ''
     export HOME=$TMPDIR
   '';
 
-  meta = with lib; {
+  meta = with stdenv.lib; {
     description = "Minimal IRC URL bot in Rust";
     homepage = "https://github.com/nuxeh/url-bot-rs";
     license = licenses.isc;
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 4df81a9efa63..0a8666e369a7 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -7901,7 +7901,9 @@ in
 
   uriparser = callPackage ../development/libraries/uriparser {};
 
-  url-bot-rs = callPackage ../applications/networking/irc/url-bot-rs {};
+  url-bot-rs = callPackage ../applications/networking/irc/url-bot-rs {
+    inherit (darwin.apple_sdk.frameworks) Security;
+  };
 
   urlscan = callPackage ../applications/misc/urlscan { };

[nuxeh]

This is awesome, thanks for taking the time to package url-bot-rs!

@mweinelt, totally agree about testing, this is something I've been thinking about for a long time, but have no personal experience with nix tests, so wasn't sure about the correct way to do it.

For my own testing I tend to use inspircd, since it can simply be run on the command line to give you a working ircd without having to play around with a configuration too much, if at all, and out of all the ircds i tried it seemed by far the easiest in this regard. I had guessed it could be easy enough to run it in a nix-shell for testing, but hadn't thought much further than that. However, it's not in nixpkgs, although I do have a derivation written for it that hasn't been submitted anywhere yet. Maybe this could be a good reason to get inspircd submitted too.

[aanderse]

@mweinelt exceptional work as always! I left a few optional suggestions that are worth bonus points, if you're interested.

[mweinelt]

Added StateDirectory, so the optional sqlite database backend has a place to store its state.

[nuxeh]

@mweinelt I made a PR mweinelt#4 against this branch to update the module and the config generation to make it feature complete, allowing a number of things to be configured, but primarily for connecting to multiple networks and enabling other additional features on a per-network basis.

Also, tested thoroughly using nixos-shell.

[SuperSandro2000]

Result of nixpkgs-review pr 105315 run on x86_64-linux 1

1 package blacklisted:

• tests.nixos-functions.nixos-test

1 package built:

• url-bot-rs

[stale]

I marked this as stale due to inactivity. → More info

[nuxeh]

Would like to get this merged. I see the benefit of automatic toml generation from expressions - seems to be the modern way, while also feel an ergonomic version as I proposed in the PR i made, is also nice to have, even if the implementation is more complex, it's a conundrum for me.

However, one essential feature as I see it is being able to specify multiple networks/connections in the nix. If I can get it to work I might suggest updates to this PR to do this, using the toml generation.

[nuxeh]

Btw... it's currently not possible to connect to multiple networks, since the expression will only generate a single configuration, but with the way url-bot-rs is designed, you currently need multiple configuration files to do this.

[mweinelt]

Sorry, I was a bit overwhelmed at the time when you put up that huge pull request to my fork and was thinking about how to reconcile that with RFC42 (https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md) and that is still an open question to me.

[nuxeh]

Makes total sense, i felt conflicted about the PR myself. Have never seen this RFC, but glad there are some guidelines in this area, otherwise it does seem to be a bit open ended how you implement modules.

I think it certainly seems that the expression to toml thing seems like the way to go, I'll see if I can knock up something to allow multiple configs using it.

Something like services.url-bot-rs.configs."name" = {};. Since I guess you can have an attribute set of the to-toml type i guess that should be straight forward.

[mweinelt]

From mweinelt#4 (comment)

The key thing it addresses, however, which I do think is essential (at least for my uses), is the facility to specify connections to multiple servers. As a service this is essential for anything more than a simple single irc network deployment.

Is there a good example of a module that supports multiple instances? From what I remember people always just went with NixOS containers to side-step this issue.

[nuxeh]

@mweinelt i'm sure i have come across it many times, although the only one i can think of off the top of my head is nginx, for example. However I'm sure all of the cases I have seen are "old school" nix modules.

Since url-bot-rs natively supports multiple connections with the same binary, it seems wasteful to spawn multiple copies, also in containers, to me.

However, another way to sidestep this would be for url-bot-rs to accept a single config containing multiple networks, in a future release. I'd be quite happy to do that, it seems like a simpler option overall, and it would be better to get this merged sooner rather than later!

[mweinelt]

Sounds good to me.

[mweinelt]

The 0.3.2 update chokes reproducibly on a test:

---- plugins::vimeo::tests::evaluate stdout ----
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 98, kind: AddrInUse, message: "Address already in use" }', src/plugins/vimeo.rs:124:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'plugins::vimeo::tests::evaluate' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "http", host: Some(Ipv4(127.0.0.1)), port: Some(28286), path: "/videos/53603603", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }', src/plugins/vimeo.rs:135:64


failures:
    plugins::vimeo::tests::evaluate

test result: FAILED. 54 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 22.39s

[nuxeh]

@mweinelt 0.3.2 hasn't changed, it's just outstanding stuff that hasn't had a release yet. But indeed, have noticed this too, I think it is intermittent. Perhaps something about the port it uses in the test.

I have multi configs in nuxeh/url-bot-rs#420 (!) which appears to work, just having a final look over it, and still some tests to sort out, but I think this could be the basis of a new version.

[mweinelt]

https://github.com/nuxeh/url-bot-rs/blob/1ccfadcf79b75ec186b7052fb06f028fc38edc3e/src/plugins/vimeo.rs#L119-L124
https://github.com/nuxeh/url-bot-rs/blob/bade0f41fc976be4e370d53b5247824b88b64a63/src/http.rs#L619-L627

These two want to bind to the same port, maybe that is the problem?

[nuxeh]

So now there's https://github.com/nuxeh/url-bot-rs/releases/tag/v0.4.0-rc.1. Still think a bit more testing is needed which I can't do till tomorrow, but it's a start at least.

Wondering if a HashMap TOML serialisation might map better to a Nix expression, rather than a vector, maybe I'll play around with that some more, also.

[nuxeh]

https://github.com/nuxeh/url-bot-rs/releases/tag/v0.4.0-rc.2 is a hash map, think this is much better. Also, incase it tickles anyone's fancy I'll work on getting shell completion generation quite soon.

[nuxeh]

@mweinelt I made mweinelt#9, which is the simplest method i could come up with to allow both the existing module behaviour, and multiple network configurations which are available in url-bot-rs since v0.4.0, and by now have been reasonably well tested.

Would be interested to hear what you think. I hope it doesn't obfuscate things too much - i'm thinking mainly that the example I've added for the settings option is a multiple network configuration, which could be confusing, if it were possible to have multiple examples, i'd put a single network there too

[mweinelt]

I hope we can update the the module, so we can make the config look like services.url-bot-rs.settings.<name>.connection.server, where name would be the network instance.

That would be an types.attrsOf types.submodule construction IIRC.