Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux network plugin: NetworkManager & systemd-networkd #932

Merged
merged 9 commits into from
Nov 20, 2024

Conversation

twiggler
Copy link
Contributor

@twiggler twiggler commented Nov 5, 2024

Implement the NetworkPlugin for Linux.

Initial support is for:

  • NetworkManager
  • systemd-networkd

The ips method of the Linux OS plugin is implemented using the new parser; other methods such as dns and dhcp still use the legacy solution until we implemented parsers for the remaining network configuration systems. Let me know if this is wrong.

Update: I retained the old ips. The new implementation can be invoked by prefixing with the network namespace, i.e. -f network.interfaces

While doing research I discovered that systemd supports "drop-in" configuration directories (see https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html) . Since this feature applicable to all systemd domains and not only networking, I propose we extend the systemd config parser with support for this (#933). I checked this with Stefan de Reuver.

Keep in mind that unlike the OSX and Windows implementations, which report actual values, the Linux implementation reports configuration of interfaces, or "potential" interfaces, as there is often no way to determine which configuration was active. I think this results in some friction with the interface records; for example, the mac address needed to be extended to a list.

Closes #776

@twiggler twiggler marked this pull request as draft November 5, 2024 08:47
@twiggler twiggler force-pushed the linux-network-manager branch 4 times, most recently from f2e7ce1 to b91d450 Compare November 5, 2024 09:37
Copy link

codecov bot commented Nov 5, 2024

Codecov Report

Attention: Patch coverage is 88.94231% with 23 lines in your changes missing coverage. Please review.

Project coverage is 77.90%. Comparing base (ed9ec23) to head (608f8c3).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dissect/target/plugins/os/unix/linux/network.py 88.77% 22 Missing ⚠️
dissect/target/plugins/os/unix/bsd/osx/network.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #932      +/-   ##
==========================================
+ Coverage   77.82%   77.90%   +0.07%     
==========================================
  Files         323      324       +1     
  Lines       27647    27850     +203     
==========================================
+ Hits        21517    21696     +179     
- Misses       6130     6154      +24     
Flag Coverage Δ
unittests 77.90% <88.94%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@twiggler twiggler marked this pull request as ready for review November 5, 2024 09:49
Comment on lines 212 to 263
dhcp_ipv4, dhcp_ipv6 = self._parse_dhcp(network_section.get("DHCP"))
if link_mac := link_section.get("MACAddress"):
mac_addresses.add(link_mac)
if match_macs := match_section.get("MACAddress"):
mac_addresses.update(match_macs.split(" "))
if permanent_macs := match_section.get("PermanentMACAddress"):
mac_addresses.update(permanent_macs.split(" "))

if dns_value := network_section.get("DNS"):
if isinstance(dns_value, str):
dns_value = [dns_value]
dns.update({self._parse_dns_ip(dns_ip) for dns_ip in dns_value})

if address_value := network_section.get("Address"):
if isinstance(address_value, str):
address_value = [address_value]
ip_interfaces.update({ip_interface(addr) for addr in address_value})

if gateway_value := network_section.get("Gateway"):
if isinstance(gateway_value, str):
gateway_value = [gateway_value]
gateways.update({ip_address(gateway) for gateway in gateway_value})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be an idea to create some helper functions and change it to something like this:

def _splitter(value: str | None, seperator=" ") -> list[str]:
    if not value:
       return []
    return value.split(seperator)

def _create_list(func, value: str | list | None):
     if not value:
         return []
    if isinstance(value, str):
        value = [value]
    return value

then for the code itself:

mac_addresses.update(_splitter(match_section.get("MACAddress"))

ip_interfaces.update(map(ip_address, create_list(network_section.get("Address")))

Or create a function with a map inside of it, cause this is still a bit lengthy ofc

Copy link
Contributor Author

@twiggler twiggler Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With regard to _splitter

I re-read the documentation of str.split() and if you don´t specify a separator, it will separate on white space, and return an empty list if the string is empty.

With regard to _create_list

Wrote general purpose to_list in dissect/target/helpers/utils.py.
Did not opt for the map in this case, one line for value retrieval and one line for the update is fine imo

vlan_values = network_section.get("VLAN", [])
vlan_ids = {
virtual_networks[vlan_name]
for vlan_name in ([vlan_values] if isinstance(vlan_values, str) else vlan_values)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put this above the comprehension statement. It becomes a bit more readable then. Currently a lot is going on imho

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified with to_list

configurator="systemd-networkd",
)
except Exception as e:
self._target.log.warning("Error parsing network config file %s: %s", config_file, e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its better to put e inside a self._target.log.debug("", exc_info=e) there are some examples for that everywhere. such as dissect/target/container.py

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a warning is better; that way, the analyst can respond by analyzing the potentially broken configuration file manually.

There are apparently also some wrong examples; I took this from

self.target.log.warning("Error reading configuration for network device %s: %s", name, e)

Will fix it there too.


# Can be enclosed in brackets for IPv6. Can also have port, iface name, and SNI, which we ignore.
# Example: [1111:2222::3333]:9953%ifname#example.com
dns_ip_patttern = re.compile(r"((?:\d{1,3}\.){3}\d{1,3})|\[(\[?[0-9a-fA-F:]+\]?)\]")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to use capture groups for this, instead of using indexes of 1 and 2 on L276?
Would make it a bit more readable.

Copy link
Contributor Author

@twiggler twiggler Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, although Imo it is still unreadable.

Alternative could be parser combinators, but we don't use them yet. I would have to search for a suitable python implementation.

Comment on lines 274 to 299
match = self.dns_ip_patttern.search(address)
if match:
return ip_address(match.group(1) or match.group(2))
else:
raise ValueError(f"Invalid DNS address format: {address}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
match = self.dns_ip_patttern.search(address)
if match:
return ip_address(match.group(1) or match.group(2))
else:
raise ValueError(f"Invalid DNS address format: {address}")
if match:= self.dns_ip_patttern.search(address):
return ip_address(match.group(1) or match.group(2))
raise ValueError(f"Invalid DNS address format: {address}")

Comment on lines 145 to 151
def _parse_lastconnected(self, value: str) -> datetime | None:
"""Parse last connected timestamp."""
if not value:
return None

timestamp_int = int(value)
return datetime.fromtimestamp(timestamp_int, timezone.utc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def _parse_lastconnected(self, value: str) -> datetime | None:
"""Parse last connected timestamp."""
if not value:
return None
timestamp_int = int(value)
return datetime.fromtimestamp(timestamp_int, timezone.utc)
def _parse_lastconnected(self, last_connected: str) -> datetime | None:
"""Parse last connected timestamp."""
if not last_connected:
return None
return datetime.fromtimestamp(int(last_connected), timezone.utc)

vlan_id_by_interface[parent_interface] = int(vlan_id)
continue

dns = set[ip_address]()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dns = set[ip_address]()
dns: set[ip_address] = set()



class LinuxConfigParser:
VlanIdByName = dict[str, int]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could put this type definition outside the class. Then it can just be used by any class without needing to first call LinuxConfigParser

Copy link
Contributor Author

@twiggler twiggler Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I want to prevent polluting the module scope. But moved it, since the requirement to fully qualify decreases readability

yield from manager.interfaces()


class LinuxConfigParser:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class LinuxConfigParser:
class LinuxNetworkParser:

wouldn't this be more accurate?

Copy link
Contributor Author

@twiggler twiggler Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed into LinuxNetworkConfigParser, which is perhaps even better

dissect/target/plugins/os/unix/linux/network.py Outdated Show resolved Hide resolved
@twiggler twiggler force-pushed the linux-network-manager branch 6 times, most recently from 83258be to 3b56d58 Compare November 8, 2024 16:45
Comment on lines +29 to +45
T = TypeVar("T")


def to_list(value: T | list[T]) -> list[T]:
"""Convert a single value or a list of values to a list.

Args:
value: The value to convert.

Returns:
A list of values.
"""
if not isinstance(value, list):
return [value]
return value


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a bit overkill to put here as no other class uses it yet, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the method is too general to put anywhere else, although the module names helpers and utils are a bit nondescript.

Besides, if we keep it in for example the SystemdNetworkConfigParser, then it is likely that no one is going to use it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

until we find everyone is making it again and move it then :P but i get your point

source=self.source,
last_connected=self.last_connected,
name=self.name,
mac=[self.mac_address] if self.mac_address else [],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not going to use to_list here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has different semantics than to_list: it maps None to an empty list, which to_list does not do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I was remembering an implementation where None was transformed into an empty list. So that is fair

Comment on lines 127 to 128
uuid = vlan_id_by_interface.get(context.uuid) if context.uuid else None
name = vlan_id_by_interface.get(connection.name) if connection.name else None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to do the if conditions here, the get on a dictionary returns a None by default. if context.uuid is not, it will try to index it using a None which returns a None cause it does not exist.

Copy link
Contributor Author

@twiggler twiggler Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am defending against lookup of None in vlan_id_by_interface, which would be a logic error.
But perhaps overly cautious, removed.

However, there is an edge case where there are multiple vlans bound to the same interface, where the first vlan is bound by iface name, and the second by uuid.

Rewrote and adjusted unit test

Comment on lines +166 to +181
elif key.startswith("route"):
if gateway := self._parse_route(value):
context.gateways.add(gateway)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
elif key.startswith("route"):
if gateway := self._parse_route(value):
context.gateways.add(gateway)
elif key.startswith("route") and (gateway := self._parse_route(value)):
context.gateways.add(gateway)

?

Copy link
Contributor Author

@twiggler twiggler Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, I actually prefer the nested if because they inner and outer if have different purposes.

The outer if switches on the key type, and the inner if is part of the action of handling a certain configuration value. This way, we can scan more easily over the different key types.

I made all other outer clauses consistent. This leads to more indentation but I think it is for the better

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, so you want to treat it like some kind of switch/match statement. That's fair

Comment on lines 162 to 165
elif key == "method" and ip_version == "ipv4":
context.dhcp_ipv4 = value == "auto"
elif key == "method" and ip_version == "ipv6":
context.dhcp_ipv6 = value == "auto"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
elif key == "method" and ip_version == "ipv4":
context.dhcp_ipv4 = value == "auto"
elif key == "method" and ip_version == "ipv6":
context.dhcp_ipv6 = value == "auto"
elif key == "method":
setattr(context, f"dhcp_{ip_version}", value == "auto")

tho maybe this is a bit too much magic :P

Copy link
Contributor Author

@twiggler twiggler Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clever, but I think the original is more readable.

Comment on lines 151 to 152
if key == "dns" and (stripped := value.rstrip(";")):
context.dns.update({ip_address(addr) for addr in stripped.split(";")})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if key == "dns" and (stripped := value.rstrip(";")):
context.dns.update({ip_address(addr) for addr in stripped.split(";")})
if key == "dns":
context.dns.update({ip_address(addr) for addr in value.split(";") if addr})

wouldn't this also work?

Copy link
Contributor Author

@twiggler twiggler Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also work, but I think the original more clearly reflects that we are guarding against a trailing ;.

Then again, maybe empty values are somehow allowed so your suggestion is slightly more robust.

Applied suggestion

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the empty values get ignored due to the if addr if addr ='' for example. that is some truethy magic in python. same with an empty list and such

Copy link
Contributor Author

@twiggler twiggler Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With maybe empty values are somehow allowed

I meant according to the Network Manager spec. (I applied your suggestion)

Comment on lines 171 to 172
parent_interface = sub_type.get("parent", None)
vlan_id = sub_type.get("id", None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get() doesn't need an explicit None as it is the default for dicts :)

Comment on lines 260 to 262
vlan_id
for vlan_name in to_list(vlan_names)
if (vlan_id := virtual_networks.get(vlan_name)) is not None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
vlan_id
for vlan_name in to_list(vlan_names)
if (vlan_id := virtual_networks.get(vlan_name)) is not None
virtual_networks.get(_name) for _name in to_list(vlan_names) if _name in virtual_networks

isn't this a bit more clear?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I don't like is that the lookup will be performed twice.

In any case, rewrote as an explicit loop. Also fixed a type-inconsistency.

Also adjusted unit test to check for multiple vlans.

@twiggler twiggler force-pushed the linux-network-manager branch 2 times, most recently from 99bb458 to e3982a7 Compare November 12, 2024 22:08
return

if key == "dns":
context.dns.update({ip_address(addr) for addr in trimmed.split(";") if addr})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed you put these explicitly inside a set, however this isn't needed as the update already accepts any kind of iterator.

Suggested change
context.dns.update({ip_address(addr) for addr in trimmed.split(";") if addr})
context.dns.update(ip_address(addr) for addr in trimmed.split(";") if addr)

this is similar to the previous soltion, but doesn't allocate a new set first.
I think you can do this for all the ones inside this file.

Comment on lines +166 to +181
elif key.startswith("route"):
if gateway := self._parse_route(value):
context.gateways.add(gateway)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, so you want to treat it like some kind of switch/match statement. That's fair

if gateway := self._parse_route(value):
context.gateways.add(gateway)

def _parse_vlan(self, sub_type: dict["str", any], vlan_id_by_interface: VlanIdByInterface) -> None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the "str" wasn't needed like that, and any is a function, use from typing import Any instead

Suggested change
def _parse_vlan(self, sub_type: dict["str", any], vlan_id_by_interface: VlanIdByInterface) -> None:
def _parse_vlan(self, sub_type: dict[str, Any], vlan_id_by_interface: VlanIdByInterface) -> None:


yield UnixInterfaceRecord(
source=str(config_file),
type=match_section.get("Type", None),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type=match_section.get("Type", None),
type=match_section.get("Type"),

enabled=None, # Unknown, dependent on run-time state
dhcp_ipv4=dhcp_ipv4,
dhcp_ipv6=dhcp_ipv6,
name=match_section.get("Name", None),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name=match_section.get("Name", None),
name=match_section.get("Name"),

Comment on lines 3 to 14
import re
from dataclasses import dataclass, field
from datetime import datetime, timezone
from ipaddress import ip_address, ip_interface
from typing import Iterator, Literal, NamedTuple

from dissect.target import Target
from dissect.target.helpers import configutil
from dissect.target.helpers.record import UnixInterfaceRecord
from dissect.target.helpers.utils import to_list
from dissect.target.plugins.general.network import NetworkPlugin
from dissect.target.target import TargetPath
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import re
from dataclasses import dataclass, field
from datetime import datetime, timezone
from ipaddress import ip_address, ip_interface
from typing import Iterator, Literal, NamedTuple
from dissect.target import Target
from dissect.target.helpers import configutil
from dissect.target.helpers.record import UnixInterfaceRecord
from dissect.target.helpers.utils import to_list
from dissect.target.plugins.general.network import NetworkPlugin
from dissect.target.target import TargetPath
from ipaddress import ip_address, ip_interface
from typing import TYPE_CHECKING, Iterator, Literal, NamedTuple
from dissect.target.helpers import configutil
from dissect.target.helpers.record import UnixInterfaceRecord
from dissect.target.helpers.utils import to_list
from dissect.target.plugins.general.network import NetworkPlugin
if TYPE_CHECKING:
from ipaddress import IPv4Address, IPv6Address
from dissect.target import Target
from dissect.target.target import TargetPath
NetAddress = IPv4Address | IPv6Address

maybe something like this for _parse_dns_ip.
with NetAddress as the return type of _parse_nds_ip

Copy link
Contributor Author

@twiggler twiggler Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, also added NetInterface

self._target.log.warning("Error parsing network config file %s", config_file, exc_info=e)

def _parse_dns_ip(self, address: str) -> ip_address:
"""Parse DNS address from systemd network configuration file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ip_address isn't a type but a function here.
Maybe its better to put the type definition at the top. I'll put an example there


raise ValueError(f"Invalid DHCP value: {value}")

def _parse_gateway(self, value: str | None) -> ip_address | None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same thing here ip_address is a function, not a type

Comment on lines 302 to 306
match = self.dns_ip_patttern.search(address)
if not match:
raise ValueError(f"Invalid DNS address format: {address}")

return ip_address(match.group("withoutBrackets") or match.group("withBrackets"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
match = self.dns_ip_patttern.search(address)
if not match:
raise ValueError(f"Invalid DNS address format: {address}")
return ip_address(match.group("withoutBrackets") or match.group("withBrackets"))
if match := self.dns_ip_patttern.search(address):
return ip_address(match.group("withoutBrackets") or match.group("withBrackets"))
raise ValueError(f"Invalid DNS address format: {address}")

a bit more readable IMHO

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanted to keep the happy path on the left, but fair. Applied

vlan_ids = virtual_networks.setdefault(name, set())
vlan_ids.add(int(vlan_id))
except Exception as e:
self._target.log.warning("Error parsing virtual network config file %s", config_file, exc_info=e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also the issue with too much information in the warning log of a target.

Comment on lines +284 to +285
if mac_address := _try_value(subkey, "NetworkAddress"):
device_info["mac"] = [mac_address]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this, wouldn't network.mac fail for windows if it doesn't have a NetworkAddress now? Cause it checks it with an in for a None value.

to avoid that:

Suggested change
if mac_address := _try_value(subkey, "NetworkAddress"):
device_info["mac"] = [mac_address]
mac_address = _try_value(subkey, "NetworkAddress")
device_info["mac"] = [mac_address] if mac_address else []

Copy link
Contributor Author

@twiggler twiggler Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flow record constructor creates an empty list in that case.

We also have a unit test for this case.

(but good challenge)

connections.append(context)

except Exception as e:
self._target.log.warning("Error parsing network config file %s: %s", connection_file_path, e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe also an idea to put a debug exc_info here?

@twiggler twiggler merged commit 6e6a546 into main Nov 20, 2024
18 of 20 checks passed
@twiggler twiggler deleted the linux-network-manager branch November 20, 2024 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Linux systemd network interface plugin
2 participants