From d89cf7a0622509327934d691f550b97d8c2b849d Mon Sep 17 00:00:00 2001 From: Andreas Misje Date: Mon, 28 Aug 2023 08:05:35 +0900 Subject: [PATCH] Fix issues in AptSources artifact and support deb822 format (#2851) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Linux.Debian.AptSources The Linux.Debian.AptSources artifact has a number of limitations, all of which the changes in this pull request aims to resolve. The new artifact also provides parsers that can be imported in other artifacts for inspecting every single field in these configuration files. ## Resolved issues The artifact does not work on current Debian-based systems. One reason is an increased usage of the option "signed-by", used to manually specify the gpg key file location. The original regex only supports the option "arch" or no options. The main reason is the that Release files have been replaced with InRelease files. The format is similar, just with a gpg key embedded at the end of the file. Note that these keys often contain a gpg header "Version: […] that confuses the original VQL. **Fixes and enhancements**: - Support for sources files (multi-line [deb822 format](https://manpages.debian.org/bookworm/apt/sources.list.5.en.html#DEB822-STYLE_FORMAT)) - Support options other than "arch" in list files - Support for any valid usage of whitespace in list files - Support for comments in (at the end of) list files - Support for protocols other than http/https in list files - Support for multiple values in fields/options (arch, lang, targets, components) - Parse InRelease files as well as Release files, and prevent the "Version: GnuPG" header line from interfering with the "Version" field - Group the query results by cache file, preventing duplicate entries without records After improving the list file parser, I also added a parser for the new file format, likely to replace the one-line list file format in the future: deb822. This file format allows each entry to have multiple values (deb + deb-src, multiple URIs and options), and each file may contain multiple entries separated by blank lines. GPG keys may be embedded, and the format is very lenient on whitespace usage and allowing multi-line values. The parser should handle any kind of strange, but valid, formatting, like ``` Key: Value Key2 : Value Key3: # Comment # Comment Value ``` and so on. Due to limitations in the go regex library (no support for look-arounds) and the complexity of multi-value fields and options, the parser VQL is complex and split in a number of functions. Columns are dynamic in order to support every known (and future/unknown) options. This has pros and cons, but it is future-proof. Every known option name is normalised to the names referred to in the man page, which are camel-cased (pascal-cased) and plural. ## Considerations The main motivation behind improving this artifact was to have a working apt sources parser. Since this artifact is part of velociraptor and not the exchange (velociraptor-docs), I've tried to make the parsers handle everything I can throw at it. I am happy to include tests, but before I do that, I would like to get some feedback. I kept the original query, but I also included two others. I feel that a table including every parsed field, like all options, is handy. Likewise, a flattened version is also very useful to work on in a notebook. This results in two rounds of parsing sources files (due to how flatting happens at an early stage), and the third query parses cache files as well, like before. This shouldn't have any noticeable effect on any normal system, but if there are ways I can rewrite the VQL do only parse the files once, for both the combined and flattened results, that would of course by much nicer. And perhaps something belongs in notebook suggestions? I see lots of potential in this artifact, but for starters, I'd like it to work and be useful for importing in other artifacts. Working with inrelease-path and signed-by options in future enhancements may be useful. I find it a bit tricky to get an overview of all the supported features of artifact YAML, so feel free to point out how I may improve the documentation, ways to export functions (without including all "private" helper functions), and how to improve the current three–query solution. --------- Co-authored-by: Mike Cohen --- .github/workflows/go.yml | 2 +- .../definitions/Linux/Debian/AptSources.yaml | 495 ++++++++++++++--- .../testdata/files/debian/.gitattributes | 1 + ...ve.ubuntu.com_ubuntu_dists_jammy_InRelease | 32 ++ artifacts/testdata/files/debian/sources.list | 52 ++ artifacts/testdata/files/debian/status | 138 +++++ .../testdata/server/testcases/debian.in.yaml | 32 ++ .../testdata/server/testcases/debian.out.yaml | 525 ++++++++++++++++++ 8 files changed, 1213 insertions(+), 64 deletions(-) create mode 100644 artifacts/testdata/files/debian/.gitattributes create mode 100644 artifacts/testdata/files/debian/au.archive.ubuntu.com_ubuntu_dists_jammy_InRelease create mode 100644 artifacts/testdata/files/debian/sources.list create mode 100644 artifacts/testdata/files/debian/status create mode 100644 artifacts/testdata/server/testcases/debian.in.yaml create mode 100644 artifacts/testdata/server/testcases/debian.out.yaml diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 8ceea10b3e1..0e526a91c1a 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -12,7 +12,7 @@ jobs: name: Build # We used to build on ubuntu-18.04 but that is now deprecated by # GitHub. Earlier distributions will have to use the musl build. - runs-on: ubuntu-20.04 + runs-on: ubuntu-latest steps: - name: Check out code into the Go module directory diff --git a/artifacts/definitions/Linux/Debian/AptSources.yaml b/artifacts/definitions/Linux/Debian/AptSources.yaml index 056c9e7ca48..debe7368bde 100644 --- a/artifacts/definitions/Linux/Debian/AptSources.yaml +++ b/artifacts/definitions/Linux/Debian/AptSources.yaml @@ -2,85 +2,444 @@ name: Linux.Debian.AptSources description: | Parse Debian apt sources. - We first search for \*.list files which contain lines of the form + This Artifact searches for all apt sources files and parses all + fields in both one–line `*.list` files and `*.sources` files + (deb822-style format). The results are presented both in a readable + table and a flattened version for parsing. - .. code:: console + `*.list` files contains lines of the form - deb http://us.archive.ubuntu.com/ubuntu/ bionic main restricted + ``` + deb http://us.archive.ubuntu.com/ubuntu/ bionic main restricted + deb-src [arch=amd64,i386 signed-by=/usr/share/keyrings/foo.gpg] https://foo.bar.baz/ubuntu/main jammy main restricted universe multiverse # Comment + ``` - For each line we construct the cache file by spliting off the - section (last component) and replacing / and " " with _. + deb indicates a source for binary packages, and deb-src instructs APT where + to find source code for packages. - We then try to open the file. If the file exists we parse some - metadata from it. If not we leave those columns empty. + `*.sources` files (deb822-style format) are in the form of key–value + lines, and as opposed to the one–line format, they can contain + multiple URIs, components and types (deb/deb-src), along with + embedded GPG keys. Example: + + ``` + Types: deb deb-src + URIs: file:/home/apt/debian http://foo.bar.baz/main + Suites: unstable + Components: main contrib non-free + ``` + + The exported function parse_aptsources(OSPath, flatten) parses + both formats and returns a (optionally flattened) table with + - OSPath + - Types (deb/deb-src) + - Components (e.g. main/contrib/non-free/restricted,universe) + - Suites (e.g. unstable/bookworm/jammy) + - _URIBase (.e.g us.archive.ubuntu.com/ubuntu/) + - _Transport (e.g. http/https/file/cdrom/ftp) + - URIs (e.g. http://us.archive.ubuntu.com/ubuntu/) + + Any option is added to an individual column. Typical options are + - Architectures (e.g. amd64/i386/armel) + - Signed-By (e.g. /usr/share/keyrings/osquery.gpg) + + All known option names are transformed to the plural PascalCase + variants as listed in the sources.list man page. Any undocumented + options will still be included in the results, with names unchanged. + Options in the one-line format of the form "lang+=de"/"arch-=i386" + will be in columns like "Languages-Add"/"Architectures-Remove", matching + the option names having the same effect in deb822. + + Entries in deb822 sources files may be disabled by including + "Enabled: no" instead of commenting out all lines. If this field + is not present with a falsly value, the entry is enabled. Use the + exported functions DebTrue()/DebFalse() to correctly parse all + accepted true/false strings, or use the VQL suggestion "Enabled" + to filter on this column (true), if present. + + If the GPG key is embedded in a .sources file, the whole GPG key + will be included in the cell. Otherwise the value will be a file + path. + + If flatten is False, multi–value fields (like Components) will + be combined in a single-space-separated string in each row. + + In addition to the two apt sources tables, a third table correlates + information from InRelease and Release files to provide additional + metadata. The modification timestamps may tell when the package + lists where last updated. reference: - - https://osquery.io/schema/3.2.6#apt_sources + - https://manpages.debian.org/bookworm/apt/sources.list.5.en.html + - https://manpages.debian.org/bookworm/dpkg-dev/deb822.5.en.html + - https://salsa.debian.org/apt-team/apt/-/blob/main/apt-pkg/sourcelist.cc + - https://wiki.debian.org/DebianRepository/Format#A.22Release.22_files + +export: | + /* Remove whitespace from the beginning and end of a string: */ + LET Trim(string) = regex_transform(source=string, map=dict( + `(?m)^\\s+`='', + `(?m)\\s+$`='' + )) + + /* Replace any repeating whitespace with a single space: */ + LET Simplify(string) = regex_replace(source=string, re='''\s+''', replace=' ') + + /* The syntax in lists (deb822) and sources (one-line) files varies a bit, + and deb822 is case-insensitive. Normalise all known fields (as per + the man page): */ + LET NormaliseOpts(string) = regex_transform(source=string, map=dict( + `(?i)types|type`='Types', + `(?i)uris|uri`='URIs', + `(?i)suites|suite`='Suites', + `(?i)components|component`='Components', + `(?i)architectures$|arch$`='Architectures', + `(?i)architectures-add`='Architectures-Add', + `(?i)architectures-remove`='Architectures-Remove', + `(?i)languages$|lang$`='Languages', + `(?i)languages-add`='Languages-Add', + `(?i)languages-remove`='Languages-Remove', + `(?i)targets$|target$`='Targets', + `(?i)targets-add`='Targets-Add', + `(?i)targets-remove`='Targets-Remove', + `(?i)pdiffs`='PDiffs', + `(?i)by-hash`='By-Hash', + `(?i)allow-insecure`='Allow-Insecure', + `(?i)allow-weak`='Allow-Weak', + `(?i)allow-downgrade-to-insecure`='Allow-Downgrade-To-Insecure', + `(?i)trusted`='Trusted', + `(?i)signed-by`='Signed-By', + `(?i)check-valid-until`='Check-Valid-Until', + `(?i)valid-until-min`='Valid-Until-Min', + `(?i)valid-until-max`='Valid-Until-Max', + `(?i)check-date`='Check-Date', + `(?i)date-max-future`='Date-Max-Future', + `(?i)inrelease-path`='InRelease-Path', + `(?i)enabled`='Enabled' + )) + + LET DebTrue(string) = if( + condition=string=~'(?i)^(?:yes|true|with|on|enable)$', + then=true, else=false) + LET DebFalse(string) = if( + condition=string=~'(?i)^(?:no|false|without|off|disable)$', + then=true, else=false) + + /* Extract Key–Value pairs from option string. If assignment is -=/+=, + the -/+ operator is captured in Op: */ + LET OptStringToKeyValues__(string) = SELECT * + FROM parse_records_with_regex( + regex='''(?P[^ ]+?)(?P-|\+)?=(?P[^ ]+)''', + accessor='data', file=string + ) + + /* Since option values may have multiple words, split them and flatten + the results for further processing: */ + LET OptStringToKeyValues_(string) = SELECT * + FROM flatten(query={ + SELECT Key, + Op, + split(sep_string=',', string=Value) AS Value + FROM OptStringToKeyValues__(string=string) + }) + + /* Since options may be repeated, enumerate and group all values + per key and operation: */ + LET OptStringToKeyValues(string) = SELECT Key, + Op, + enumerate(items=Value) AS Value + FROM OptStringToKeyValues_(string=string) + GROUP BY Key, Op + + /* When an option is specified with +/-, represent this by appending + -Add/-Remove to the option name. These names match the syntax in + the deb822 format (i.e. "arch-=i386" == "Arhitectures-Remove: i386"). + The purpose of these assignments is to keep the default values + (rather than overriding them), but add or remove one or several + values: */ + LET OpName(op) = if(condition=op='+',then='-Add',else= + if(condition=op='-',then='-Remove',else='')) + + /* Convert a string of key–value pairs to a dict, and use consistent + option names: */ + LET OptStringToDict(string, flatten) = to_dict(item={ + SELECT NormaliseOpts(string=Key)+OpName(op=Op) AS _key, + if(condition=flatten, then=Value, + else=join(array=Value, sep=' ')) AS _value + FROM OptStringToKeyValues(string=string) + }) + + /* Parse a one-line deb sources.list file with options as a single string: */ + LET DebOneLine_Opts(OSPath) = SELECT OSPath, Type AS Types, + Simplify(string=Options) AS Options, URI AS URIs, + Transport AS _Transport, URIBase AS _URIBase, Suite AS Suites, + Simplify(string=Trim(string=Components)) AS Components + FROM parse_records_with_regex( + file=OSPath, + /* This regex attemps to cover most of the ways a sources + line can be written without being overly complex. Quotes + ("" and []) are actually allowed to certain degree by the + apt source code, but this is considered obscure syntax and + is not expected to be found in the wild. The exception is + "cdrom:[word word…]", which is capture correctly in order + to not end up with incorrectly captured words: */ + regex='''(?m)^\s*(?Pdeb(-src)?)(?:\s+\[(?P[^\]#]+)(?:#[^\]]+)?\])?\s+"?(?P(?P[^:]+):(?://)?(?P\[.+?\]|\S+?))"?\s+(?P\S+)\s+(?P[^\n#]+)''' + ) + + /* Parse a one-line deb sources.list file and output a dict: */ + LET DebOneLine_Dict(OSPath, flatten) = SELECT OSPath, * + FROM foreach(row=DebOneLine_Opts(OSPath=OSPath), + query={SELECT _value + + OptStringToDict(string=Options, flatten=flatten) AS Contents + FROM items(item={SELECT Types, URIs, _Transport, _URIBase, Suites, + if(condition=flatten, then=split(sep_string=' ', + string=Components), else=Components) AS Components + FROM scope() + }) + }) + + /* Parse a one-line deb sources.list file with options in individual columns: */ + LET DebOneLine(OSPath) = SELECT OSPath, * FROM foreach( + row=DebOneLine_Dict(OSPath=OSPath, flatten=false), + column='Contents' + ) + + /* Parse a one-line deb sources.list file with options in individual + columns and flatten: */ + LET DebOneLine_Flattened(OSPath) = SELECT OSPath, * FROM flatten( + query={SELECT * FROM foreach( + row=DebOneLine_Dict(OSPath=OSPath, flatten=true), + column='Contents' + ) + }) + + /* Extract the transport/protocol and base from a URI: */ + LET URIComponents(URI) = parse_string_with_regex( + regex='''(?P[^:]+):(?://)?(?P[^\s]+)''', + string=URI + ) + + /* Although the documentation says to use whitespace and not comma + for multi-values in deb822, comma still appears to be supported, + and this use is seen in the wild. Treat these values correctly. + Note that this does not affect all keys, like suites and + components: + */ + LET MaybeReplaceComma(key, value) = if( + condition=key=~'(?i)^(?:arch|lang|targets)', + then=regex_replace(re='\s*,\s*', source=value, replace=' '), + else=value) + + /* Parse a deb822 sources file section into a series of key–value pairs. + Notes about the format: + - Keys must be at the beginning of the line (no whitespace allowed) + - Keys are case-insensitive + - Keys may be repeated. Values are not overridden, but combined + - Special keys that end in -Add/-Remove uses the default values, + but add or remove individual values. These keys are treated as + individual option names. + - Comments may only appear at the beginning of the line + - Multiple values are separated by whitespace, not comma. However, + some multi-value fields separated by comma are still split, even + if this is not mentioned in the documentation. + - Values may be multi-line (like when containing an embedded GPG key), + but following lines must be prefixed by whitespace. Multilines + may contain comments (prefixed by whitespace or not). Empty lines + part of a multi-line value must be prefixed by whitespace and "." + - A file may contain multiple entries, separated by empty lines. + A file must be split into sections, fed individually to this function + */ + LET Deb822_KeyValues___(section) = SELECT Key, + /* Signed-By is special (it could be an embedded GPG key),and + shouldn't be split: */ + if(condition=NormaliseOpts(string=Key)!='Signed-By', + then=split(sep_string=' ', + string=MaybeReplaceComma(key=Key, + value=Simplify(string=Trim(string=Value)))), + else=Value) AS Value + FROM parse_records_with_regex( + accessor='data', + /* A key is anything but whitespace up to a colon + Values can continue on several lines, but only if the following + lines are indented with whitespace + */ + regex='''(?m)^(?P[^#:\s]+)\s*:[^\S\n]*(?P[^\n]*(?:\n[^\S\n]+[^\n]+)*)''', + /* Before parsing the key–values, remove all comments from the file + (otherwise forming a regex without lookarounds would be very + difficult, if not impossible), Luckily, comments follow strict + rules and must start with ^#. + */ + file=regex_replace( + re='''(?m)^#.+\n''', + source=section + ) + ) + + LET Deb822_KeyValues__(section) = SELECT * FROM flatten(query={ + SELECT * FROM Deb822_KeyValues___(section=section) + }) + + LET Deb822_KeyValues_(section) = SELECT Key, + enumerate(items=Value) AS Value + FROM Deb822_KeyValues__(section=section) + GROUP BY Key + + /* Parse a deb822 sources file section into a dict with consistent option + names: */ + LET Deb822_KeyValues(section, flatten) = SELECT to_dict( + item={ + SELECT NormaliseOpts(string=Key) as _key, + if(condition=flatten, then=Value, + else=join(array=Value, sep=' ')) AS _value + FROM Deb822_KeyValues_(section=section) + }) AS Contents + FROM scope() + + /* Split paragraphs in a file (separated by one or several empty + lines) into rows. ('regex' is just anything that is illegal in Deb822Sections + to prevent splitting data into records.): */ + LET Deb822Sections(OSPath) = SELECT OSPath,* FROM split_records( + filenames=OSPath, + columns='Section', + regex='^ #', record_regex='''\n{2,}''' + ) + /* Sections may be empty due to several newlines or comments on their own + separated by newlines. Ensure that at least one field is present + (URIs are mandatory): */ + WHERE URIs + + LET Deb822_Flattened_(OSPath) = SELECT * FROM foreach( + row=Deb822Sections(OSPath=OSPath), + query={SELECT OSPath, * FROM flatten(query={ + SELECT * FROM foreach( + row=Deb822_KeyValues(section=Section, flatten=true), + column='Contents' + ) + })} + ) + + /* Parse a deb822 sources file with options in individual columns. + Note that, as opposed to DebOneLine and Deb822_Flattened, this + function does not return the columns _URIBase and _Transport, since + this format supports mulitple URIs to be specified: */ + LET Deb822(OSPath) = SELECT * FROM foreach( + row=Deb822Sections(OSPath=OSPath), + query={SELECT OSPath, * FROM foreach( + row=Deb822_KeyValues(section=Section, flatten=false), + column='Contents' + )} + ) + + /* Parse a deb822 sources file with options in individual columns, flattened: */ + LET Deb822_Flattened(OSPath) = SELECT * FROM flatten(query={ + SELECT OSPath, *, URIComponents(URI=URIs).URIBase AS _URIBase, + URIComponents(URI=URIs).Transport AS _Transport + FROM Deb822_Flattened_(OSPath=OSPath) + }) + + /* Parse an apt sources/list file */ + LET parse_aptsources(OSPath, flatten) = if( + condition=OSPath=~'.list$', + then=if(condition=flatten, + then=DebOneLine_Flattened(OSPath=OSPath), + else=DebOneLine(OSPath=OSPath) + ), + else=if(condition=flatten, + then=Deb822_Flattened(OSPath=OSPath), + else=Deb822(OSPath=OSPath) + ) + ) + + LET files = SELECT OSPath FROM glob( + globs=linuxAptSourcesGlobs.ListGlobs) + + LET deb_sources = SELECT * FROM foreach(row=files, + query={SELECT * FROM parse_aptsources(OSPath=OSPath, flatten=true)} + ) + parameters: - name: linuxAptSourcesGlobs - description: Globs to find apt source *.list files. - default: /etc/apt/sources.list,/etc/apt/sources.list.d/*.list - - name: aptCacheDirectory + description: Globs to find apt source *.list and .sources files. + type: csv + default: | + ListGlobs + /etc/apt/sources.list + /etc/apt/sources.list.d/*.list + /etc/apt/sources.list.d/*.sources + - name: aptCacheDirectory description: Location of the apt cache directory. default: /var/lib/apt/lists/ + +precondition: + SELECT OS From info() where OS = 'linux' + sources: - - precondition: - SELECT OS From info() where OS = 'linux' + - name: Sources query: | - /* Search for files which may contain apt sources. The user can - pass new globs here. */ - LET files = SELECT OSPath from glob( - globs=split(string=linuxAptSourcesGlobs, sep=",")) - - /* Read each line in the sources which is not commented. - Deb lines look like: - deb [arch=amd64] http://dl.google.com/linux/chrome-remote-desktop/deb/ stable main - Contains URL, base_uri and components. - */ - LET deb_sources = SELECT * - FROM parse_records_with_regex( - file=files.OSPath, - regex="(?m)^ *(?Pdeb(-src)?) (?:\\[arch=(?P[^\\]]+)\\] )?" + - "(?Phttps?://(?P[^ ]+))" + - " +(?P.+)") - - /* We try to get at the Release file in /var/lib/apt/ by munging + /* Output sources in a readable format: */ + SELECT * FROM foreach(row=files, + query={SELECT * FROM parse_aptsources(OSPath=OSPath, flatten=false)} + ) + notebook: + - type: vql_suggestion + name: Only enabled sources + template: | + SELECT * FROM source(artifact='Custom.Linux.Debian.AptSources/Sources') + WHERE get(field='Enabled', default='yes') =~ '(?i)^(?:yes|true|with|on|enable)$' + + - type: vql_suggestion + name: Trusted sources (apt-secure bypassed) + template: | + SELECT * FROM source(artifact='Custom.Linux.Debian.AptSources/Sources') + WHERE get(field='Trusted', default='') =~ '(?i)^(?:yes|true|with|on|enable)$' + + - name: SourcesFlattened + query: | + /* Output sources flattened for ease of analysis: */ + SELECT * FROM deb_sources + + - name: SourcesCacheFiles + query: | + /* We try to get at the Release file in /var/lib/apt/ by munging the components and URL. Strip the last component off, convert / and space to _ and - add _Release to get the filename. - */ - LET parsed_apt_lines = SELECT Arch, URL, - base_uri + " " + components as Name, Type, + add _Release/_InRelease to get the filename. + */ + LET parsed_apt_lines = SELECT get(field='Architectures', default='') AS Architectures, URIs, + _URIBase + " " + Suites + " " + Components as Name, Types, OSPath as Source, aptCacheDirectory + regex_replace( replace="_", re="_+", source=regex_replace( replace="_", re="[ /]", - source=base_uri + "_dists_" + regex_replace( - source=components, - replace="", re=" +[^ ]+$")) + "_Release" - ) as cache_file - FROM deb_sources + source=_URIBase + "_dists_" + Suites + )) as cache_file + FROM deb_sources + GROUP BY URIs, Suites - /* This runs if the file was found. Read the entire file into + /* This runs if the file was found. Read the entire file into memory and parse the same record using multiple RegExps. - */ - LET parsed_cache_files = SELECT Name, Arch, URL, Type, - Source, parse_string_with_regex( - string=Record, - regex=["Codename: (?P[^\\s]+)", - "Version: (?P[^\\s]+)", - "Origin: (?P[^\\s]+)", - "Architectures: (?P[^\\s]+)", - "Components: (?P[^\\s]+)"]) as Record - FROM parse_records_with_regex(file=cache_file, regex="(?sm)(?P.+)") + */ + LET parsed_cache_files(file) = SELECT Name, Architectures, URIs, Types, + Source, parse_string_with_regex( + string=regex_replace(source=Record, + re='(?m)^Version: GnuPG v.+$', replace='' + ), + regex=["Codename: (?P[^\\n]+)", + "Version: (?P[^\\n]+)", + "Origin: (?P[^\\n]+)", + "Architectures: (?P[^\\n]+)", + "Components: (?P[^\\n]+)"]) as Record + FROM parse_records_with_regex(file=file, regex="(?sm)(?P.+)") // Foreach row in the parsed cache file, collect the FileInfo too. - LET add_stat_to_parsed_cache_file = SELECT * from foreach( + LET add_stat_to_parsed_cache_file(file) = SELECT * from foreach( query={ - SELECT OSPath, Mtime, Ctime, Atime, Record, Type, - Name, Arch, URL, Source from stat(filename=cache_file) - }, row=parsed_cache_files) + SELECT OSPath, Mtime, Ctime, Atime, Record, Types, + Name, Architectures, URIs, Source from stat(filename=file) + }, row=parsed_cache_files(file=file)) + WHERE Record + GROUP BY OSPath /* For each row in the parsed file, run the appropriate query depending on if the cache file exists. @@ -90,16 +449,26 @@ sources: */ LET parse_cache_or_pass = SELECT * from if( condition={ - SELECT * from stat(filename=cache_file) + SELECT * from stat(filename=cache_file + '_InRelease') }, - then=add_stat_to_parsed_cache_file, - else={ - SELECT Source, Null as Mtime, Null as Ctime, - Null as Atime, Type, - Null as Record, Arch, URL, Name from scope() + then=add_stat_to_parsed_cache_file(file=cache_file + '_InRelease'), + else={SELECT * FROM if( + condition={ + SELECT * from stat(filename=cache_file + '_Release') + }, + then=add_stat_to_parsed_cache_file(file=cache_file + '_Release'), + else={ + SELECT Source, NULL AS OSPath, Null as Mtime, Null as Ctime, + Null as Atime, Types, + Null as Record, Architectures, URIs, Name from scope() + }) }) -- For each parsed apt .list file line produce some output. SELECT * from foreach( - row=parsed_apt_lines, - query=parse_cache_or_pass) + row={ + SELECT * FROM parsed_apt_lines + }, + query={ + SELECT * FROM parse_cache_or_pass + }) diff --git a/artifacts/testdata/files/debian/.gitattributes b/artifacts/testdata/files/debian/.gitattributes new file mode 100644 index 00000000000..07764a78d98 --- /dev/null +++ b/artifacts/testdata/files/debian/.gitattributes @@ -0,0 +1 @@ +* text eol=lf \ No newline at end of file diff --git a/artifacts/testdata/files/debian/au.archive.ubuntu.com_ubuntu_dists_jammy_InRelease b/artifacts/testdata/files/debian/au.archive.ubuntu.com_ubuntu_dists_jammy_InRelease new file mode 100644 index 00000000000..13388142c2b --- /dev/null +++ b/artifacts/testdata/files/debian/au.archive.ubuntu.com_ubuntu_dists_jammy_InRelease @@ -0,0 +1,32 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA512 + +Origin: Ubuntu +Label: Ubuntu +Suite: jammy +Version: 22.04 +Codename: jammy +Date: Thu, 21 Apr 2022 17:16:08 UTC +Architectures: amd64 arm64 armhf i386 ppc64el riscv64 s390x +Components: main restricted universe multiverse +Description: Ubuntu Jammy 22.04 +MD5Sum: + 8f73d18065a4f05ee7362c50553ea4b26ca2b4b3786472676b6b1ee58b4cff72 17832892 universe/source/Sources.xz +Acquire-By-Hash: yes +-----BEGIN PGP SIGNATURE----- +Version: GnuPG v1 + +iQIcBAEBCgAGBQJiYZF3AAoJEIcZINGZG8k8oMMQAJGLXDzjk/IxqaxUwhgx8F/S +ultWd3vuBeWK1guBZAl4dbmunNZBbKNYcOGlOmpVF9jBZtVQ0jVWhcuVbTPfkUO/ +sz1fVWn12lzBnETsV5v6Nscz03NgNlojkD8jlEWV7xxPWh9r+CNZ8y5LzwTwlufZ +e4dm8L5d2nhHDsajX8f7ZRcch4TH7jPN2AwHWtn7yASTzUbrQ/S9shQjP1dQI38Z +VrKbBzvma9neoTzRalDf+CGGmtykfhdbT0nMGFdHXtUGNZU0QUoSw+0KxWuXwo8O +nJ5H8bN5w1+Dur8LUn2yPXZcjF7GkeGnIk1FSyleV+UOzGN1f/0T7OpId36spQ9Y +2w24Fij4xvRHQC+uD6VYSz+YXjB5qD6u00PrzqBHP8wDgJwJPBHbncuwBbpsiJBv +HQ4mYRLPWz9UCmWXXVOao/nYM+Y++M1FYwhUuptFx9HHJFuR/UILzP4/WFJTMfzs +UGVcOfyoXqi/cTZJ+Rr/S6jhecpNbUAoA2VG4ZjX1ZSaS3Hupd4t7PdCHoWB7ODT +AscDVBEidU5kimmZzo6nWNDU8LfhSUrS0inbX0YMcBweaLWpMMtB+Ffu6mvy6Ejn +P+w15a4DgOg53uxHaq/pzp7IIEfhJGfwyhznN4f86pOX6BHWikGcGHBL62WywHSK +pxgP/fNHYqIhYWCgUWeM +=DmDL +-----END PGP SIGNATURE----- diff --git a/artifacts/testdata/files/debian/sources.list b/artifacts/testdata/files/debian/sources.list new file mode 100644 index 00000000000..26098bd88f2 --- /dev/null +++ b/artifacts/testdata/files/debian/sources.list @@ -0,0 +1,52 @@ +## EOL upgrade sources.list +# Required +deb http://archive.ubuntu.com/ubuntu jammy main restricted universe multiverse +deb http://archive.ubuntu.com/ubuntu jammy-updates main restricted universe multiverse +deb http://archive.ubuntu.com/ubuntu jammy-security main restricted universe multiverse + +# Optional +deb http://archive.ubuntu.com/ubuntu jammy-backports main restricted universe multiverse + +# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to +# newer versions of the distribution. +deb http://au.archive.ubuntu.com/ubuntu jammy main restricted +# deb-src http://au.archive.ubuntu.com/ubuntu groovy main restricted + +## Major bug fix updates produced after the final release of the +## distribution. +deb http://au.archive.ubuntu.com/ubuntu jammy-updates main restricted +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-updates main restricted + +## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu +## team. Also, please note that software in universe WILL NOT receive any +## review or updates from the Ubuntu security team. +deb http://au.archive.ubuntu.com/ubuntu jammy universe +# deb-src http://au.archive.ubuntu.com/ubuntu groovy universe +deb http://au.archive.ubuntu.com/ubuntu jammy-updates universe +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-updates universe + +## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu +## team, and may not be under a free licence. Please satisfy yourself as to +## your rights to use the software. Also, please note that software in +## multiverse WILL NOT receive any review or updates from the Ubuntu +## security team. +deb http://au.archive.ubuntu.com/ubuntu jammy multiverse +# deb-src http://au.archive.ubuntu.com/ubuntu groovy multiverse +deb http://au.archive.ubuntu.com/ubuntu jammy-updates multiverse +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-updates multiverse + +## N.B. software from this repository may not have been tested as +## extensively as that contained in the main release, although it includes +## newer versions of some applications which may provide useful features. +## Also, please note that software in backports WILL NOT receive any review +## or updates from the Ubuntu security team. +deb http://au.archive.ubuntu.com/ubuntu jammy-backports main restricted universe multiverse +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-backports main restricted universe multiverse + + +deb http://au.archive.ubuntu.com/ubuntu jammy-security main restricted +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-security main restricted +deb http://au.archive.ubuntu.com/ubuntu jammy-security universe +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-security universe +deb http://au.archive.ubuntu.com/ubuntu jammy-security multiverse +# deb-src http://au.archive.ubuntu.com/ubuntu groovy-security multiverse diff --git a/artifacts/testdata/files/debian/status b/artifacts/testdata/files/debian/status new file mode 100644 index 00000000000..855491cf532 --- /dev/null +++ b/artifacts/testdata/files/debian/status @@ -0,0 +1,138 @@ +Package: adduser +Status: install ok installed +Priority: important +Section: admin +Installed-Size: 608 +Maintainer: Ubuntu Developers +Architecture: all +Multi-Arch: foreign +Version: 3.118ubuntu5 +Depends: passwd, debconf (>= 0.5) | debconf-2.0 +Suggests: liblocale-gettext-perl, perl, ecryptfs-utils (>= 67-1) +Conffiles: + /etc/deluser.conf 773fb95e98a27947de4a95abb3d3f2a2 +Description: add and remove users and groups + This package includes the 'adduser' and 'deluser' commands for creating + and removing users. + . + - 'adduser' creates new users and groups and adds existing users to + existing groups; + - 'deluser' removes users and groups and removes users from a given + group. + . + Adding users with 'adduser' is much easier than adding them manually. + Adduser will choose appropriate UID and GID values, create a home + directory, copy skeletal user configuration, and automate setting + initial values for the user's password, real name and so on. + . + Deluser can back up and remove users' home directories + and mail spool or all the files they own on the system. + . + A custom script can be executed after each of the commands. +Original-Maintainer: Debian Adduser Developers + +Package: adwaita-icon-theme +Status: install ok installed +Priority: optional +Section: gnome +Installed-Size: 5234 +Maintainer: Ubuntu Developers +Architecture: all +Multi-Arch: foreign +Version: 41.0-1ubuntu1 +Replaces: adwaita-icon-theme-full (<< 41.0-1ubuntu1), gnome-themes-standard-data (<< 3.18.0-2~) +Depends: hicolor-icon-theme, gtk-update-icon-cache, ubuntu-mono | adwaita-icon-theme-full +Recommends: librsvg2-common +Breaks: adwaita-icon-theme-full (<< 41.0-1ubuntu1), gnome-themes-standard-data (<< 3.18.0-2~) +Description: default icon theme of GNOME (small subset) + This package contains the default icon theme used by the GNOME desktop. + The icons are used in many of the official gnome applications like eog, + evince, system monitor, and many more. + . + This package only contains a small subset of the original GNOME icons which + are not provided by the Humanity icon theme, to avoid installing many + duplicated icons. Please install adwaita-icon-theme-full if you want the full + set. +Original-Maintainer: Debian GNOME Maintainers + +Package: adwaita-icon-theme-full +Status: install ok installed +Priority: optional +Section: gnome +Installed-Size: 21330 +Maintainer: Ubuntu Developers +Architecture: all +Multi-Arch: foreign +Source: adwaita-icon-theme +Version: 41.0-1ubuntu1 +Replaces: adwaita-icon-theme (<< 41.0-1ubuntu1), gnome-themes-standard-data (<< 3.18.0-2~) +Provides: gnome-icon-theme-symbolic +Depends: adwaita-icon-theme (= 41.0-1ubuntu1), gtk-update-icon-cache +Recommends: librsvg2-common +Breaks: adwaita-icon-theme (<< 41.0-1ubuntu1), gnome-themes-standard-data (<< 3.18.0-2~) +Description: default icon theme of GNOME + This package contains the default icon theme used by the GNOME desktop. + The icons are used in many of the official GNOME applications like eog, + Evince, system monitor, and many more. +Original-Maintainer: Debian GNOME Maintainers + +Package: alsa-topology-conf +Status: install ok installed +Priority: optional +Section: libs +Installed-Size: 420 +Maintainer: Ubuntu Developers +Architecture: all +Multi-Arch: foreign +Version: 1.2.5.1-2 +Replaces: libasound2-data (<< 1.2.1) +Breaks: libasound2-data (<< 1.2.1) +Enhances: libasound2-data +Description: ALSA topology configuration files + This package contains ALSA topology configuration files that can be used + by libasound2 for specific audio hardware. + . + ALSA is the Advanced Linux Sound Architecture. +Original-Maintainer: Debian ALSA Maintainers +Homepage: https://www.alsa-project.org/ + +Package: alsa-ucm-conf +Status: install ok installed +Priority: optional +Section: libs +Installed-Size: 560 +Maintainer: Ubuntu Developers +Architecture: all +Multi-Arch: foreign +Version: 1.2.6.3-1ubuntu1.7 +Depends: libasound2 (>= 1.2.4) +Description: ALSA Use Case Manager configuration files + This package contains ALSA Use Case Manager configuration of audio + input/output names and routing for specific audio hardware. They can be + used with the alsaucm tool. + . + ALSA is the Advanced Linux Sound Architecture. +Homepage: https://www.alsa-project.org/ +Original-Maintainer: Debian ALSA Maintainers + +Package: amd64-microcode +Status: install ok installed +Priority: standard +Section: non-free/admin +Installed-Size: 82 +Maintainer: Ubuntu Developers +Architecture: amd64 +Version: 3.20191218.1ubuntu2.1 +Recommends: initramfs-tools (>= 0.113~) | dracut (>= 044) | tiny-initramfs +Breaks: intel-microcode (<< 2) +Conffiles: + /etc/default/amd64-microcode eaa8a5fa3fb59af4f0f55e851a6e9b20 + /etc/modprobe.d/amd64-microcode-blacklist.conf 71327241f6583b34944e638a955aba91 +Description: Processor microcode firmware for AMD CPUs + This package contains microcode patches for all AMD AMD64 + processors. AMD releases microcode patches to correct + processor behavior as documented in the respective processor + revision guides. + . + For Intel processors, please refer to the intel-microcode package. +Original-Maintainer: Henrique de Moraes Holschuh diff --git a/artifacts/testdata/server/testcases/debian.in.yaml b/artifacts/testdata/server/testcases/debian.in.yaml new file mode 100644 index 00000000000..3f82f07d6fc --- /dev/null +++ b/artifacts/testdata/server/testcases/debian.in.yaml @@ -0,0 +1,32 @@ +Queries: + - SELECT OSPath.Basename AS OSPath, * + FROM Artifact.Linux.Debian.AptSources( + source="Sources", + linuxAptSourcesGlobs=[dict(ListGlobs=srcDir + "/artifacts/testdata/files/debian/sources.list")], + aptCacheDirectory=srcDir + "/artifacts/testdata/files/debian/" + ) + + - SELECT OSPath.Basename AS OSPath, * + FROM Artifact.Linux.Debian.AptSources( + source="SourcesFlattened", + linuxAptSourcesGlobs=[dict(ListGlobs=srcDir + "/artifacts/testdata/files/debian/sources.list")], + aptCacheDirectory=srcDir + "/artifacts/testdata/files/debian/" + ) + + - SELECT OSPath.Basename as OSPath, + NULL AS Mtime, + NULL AS Ctime, + NULL AS Atime, + NULL AS Source, * + + FROM Artifact.Linux.Debian.AptSources( + source="SourcesCacheFiles", + linuxAptSourcesGlobs=[dict(ListGlobs=srcDir + "/artifacts/testdata/files/debian/sources.list")], + aptCacheDirectory=srcDir + "/artifacts/testdata/files/debian/" + ) + WHERE Record + + + - SELECT * + FROM Artifact.Linux.Debian.Packages( + linuxDpkgStatus=srcDir + "/artifacts/testdata/files/debian/status") diff --git a/artifacts/testdata/server/testcases/debian.out.yaml b/artifacts/testdata/server/testcases/debian.out.yaml new file mode 100644 index 00000000000..a30a9d27a37 --- /dev/null +++ b/artifacts/testdata/server/testcases/debian.out.yaml @@ -0,0 +1,525 @@ +SELECT OSPath.Basename AS OSPath, * FROM Artifact.Linux.Debian.AptSources( source="Sources", linuxAptSourcesGlobs=[dict(ListGlobs=srcDir + "/artifacts/testdata/files/debian/sources.list")], aptCacheDirectory=srcDir + "/artifacts/testdata/files/debian/" )[ + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "archive.ubuntu.com/ubuntu", + "Suites": "jammy", + "Components": "main restricted universe multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "archive.ubuntu.com/ubuntu", + "Suites": "jammy-updates", + "Components": "main restricted universe multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "archive.ubuntu.com/ubuntu", + "Suites": "jammy-security", + "Components": "main restricted universe multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "archive.ubuntu.com/ubuntu", + "Suites": "jammy-backports", + "Components": "main restricted universe multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy", + "Components": "main restricted", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-updates", + "Components": "main restricted", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy", + "Components": "universe", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-updates", + "Components": "universe", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy", + "Components": "multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-updates", + "Components": "multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-backports", + "Components": "main restricted universe multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-security", + "Components": "main restricted", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-security", + "Components": "universe", + "_Source": "Linux.Debian.AptSources/Sources" + }, + { + "OSPath": "sources.list", + "Types": "deb", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "Suites": "jammy-security", + "Components": "multiverse", + "_Source": "Linux.Debian.AptSources/Sources" + } +]SELECT OSPath.Basename AS OSPath, * FROM Artifact.Linux.Debian.AptSources( source="SourcesFlattened", linuxAptSourcesGlobs=[dict(ListGlobs=srcDir + "/artifacts/testdata/files/debian/sources.list")], aptCacheDirectory=srcDir + "/artifacts/testdata/files/debian/" )[ + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy-updates", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy-updates", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy-updates", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy-updates", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy-security", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy-security", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy-security", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy-security", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy-backports", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy-backports", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy-backports", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy-backports", + "_URIBase": "archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy-updates", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy-updates", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy-updates", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy-updates", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy-backports", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy-backports", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy-backports", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy-backports", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "main", + "Suites": "jammy-security", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "restricted", + "Suites": "jammy-security", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "universe", + "Suites": "jammy-security", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + }, + { + "OSPath": "sources.list", + "Components": "multiverse", + "Suites": "jammy-security", + "_URIBase": "au.archive.ubuntu.com/ubuntu", + "_Transport": "http", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "Types": "deb", + "_Source": "Linux.Debian.AptSources/SourcesFlattened" + } +]SELECT OSPath.Basename as OSPath, NULL AS Mtime, NULL AS Ctime, NULL AS Atime, NULL AS Source, * +FROM Artifact.Linux.Debian.AptSources( source="SourcesCacheFiles", linuxAptSourcesGlobs=[dict(ListGlobs=srcDir + "/artifacts/testdata/files/debian/sources.list")], aptCacheDirectory=srcDir + "/artifacts/testdata/files/debian/" ) WHERE Record[ + { + "OSPath": "au.archive.ubuntu.com_ubuntu_dists_jammy_InRelease", + "Mtime": null, + "Ctime": null, + "Atime": null, + "Source": null, + "Record": { + "Release": "jammy", + "Version": "22.04", + "Origin": "Ubuntu", + "Architectures": "amd64 arm64 armhf i386 ppc64el riscv64 s390x", + "Components": "main restricted universe multiverse" + }, + "Types": "deb", + "Name": "au.archive.ubuntu.com_ubuntu_dists_jammy_InRelease", + "Architectures": "", + "URIs": "http://au.archive.ubuntu.com/ubuntu", + "_Source": "Linux.Debian.AptSources/SourcesCacheFiles" + } +]SELECT * FROM Artifact.Linux.Debian.Packages( linuxDpkgStatus=srcDir + "/artifacts/testdata/files/debian/status")[ + { + "Package": "adduser", + "InstalledSize": 608, + "Version": "3.118ubuntu5", + "Source": null, + "Architecture": "all", + "_Source": "Linux.Debian.Packages" + }, + { + "Package": "adwaita-icon-theme", + "InstalledSize": 5234, + "Version": "41.0-1ubuntu1", + "Source": null, + "Architecture": "all", + "_Source": "Linux.Debian.Packages" + }, + { + "Package": "adwaita-icon-theme-full", + "InstalledSize": 21330, + "Version": "41.0-1ubuntu1", + "Source": "adwaita-icon-theme", + "Architecture": "all", + "_Source": "Linux.Debian.Packages" + }, + { + "Package": "alsa-topology-conf", + "InstalledSize": 420, + "Version": "1.2.5.1-2", + "Source": null, + "Architecture": "all", + "_Source": "Linux.Debian.Packages" + }, + { + "Package": "alsa-ucm-conf", + "InstalledSize": 560, + "Version": "1.2.6.3-1ubuntu1.7", + "Source": null, + "Architecture": "all", + "_Source": "Linux.Debian.Packages" + } +] \ No newline at end of file