Sync changes to query.publicsuffix.zone

[peterthomassen]

A domain's public suffix can be looked up via DNS by appending query.publicsuffix.zone and querying resulting name for type PTR. For details, see https://publicsuffix.zone/.

People have been complaining that this zone is not always up to date.

This commit adds a GitHub action that syncs new/removed entries whenever a change is made to the list on the master branch.

It has been running as a daily job for a few months in a separate repo, and has been working fine / no maintenance needed. However, it can't run there indefinitely, because GitHub will disable scheduled actions on repos that have no commits for 60 days.

To patch the query zone, an API token is required. The workflow expects it to be configured as a repo secret named DESEC_TOKEN.

@dnsguru As discussed. I'll send the token to you privately.

[simon-friedberger]

This seems like a strange request.

1. You're basically giving us write access to your zone definition instead of just running a cronjob which gets the new version from https://publicsuffix.org/list/public_suffix_list.dat once every few days which is a weird trade-off. Updating more often probably has no benefits because for most applications the list is distributed statically at a much slower pace.
2. We generally don't push list updates to other projects. I am not against it but it is work on our side.
3. I'm not sure why it's useful for this information to be available via DNS instead of something more common like a REST API and if it is, why this should be provided by a third party instead of just providing it on publicsuffix.org.
4. Our position so far has been "the PSL is a list" Trailing dots and the PSL. #792 , even if we specify an algorithm for interpreting it, that doesn't necessarily mean that everybody interprets it the same way. For example, does a public suffix of a.b.com imply that b.com is a public suffix? So would the public suffix of foo.b.com now be b.com or just com?

This looks like a cool hack and if Github action permissions are worth anything there shouldn't be any security issues but I'm not sure if we want to do this.

[wdhdev]

It would be worth noting that if we do have this sync job, people could be under the false presumption that publicsuffix.zone is associated with publicsuffix.org.

[peterthomassen]

1. You're basically giving us write access to your zone definition instead of just running a cronjob which gets the new version from https://publicsuffix.org/list/public_suffix_list.dat once every few days which is a weird trade-off.

Why?

Updating more often probably has no benefits because for most applications the list is distributed statically at a much slower pace.

Looking at the commit history, it looks like running on demand will, on average, cause it to run roughly daily as well, so I don't see a big difference. (I agree that timing details are not relevant.)

This could be a scheduled job as well, if you prefer. (Note that if the repo has no commits for 60 days, GitHub will disable it, so a dedicated repo for this purpose does not work.)

3. I'm not sure why it's useful for this information to be available via DNS instead of something more common like a REST API

Because it works everywhere, and because it does not require running a separate service, is cachable, and also has much less overhead otherwise. A number of folk are using it, indicating that it's useful.

and if it is, why this should be provided by a third party instead of just providing it on publicsuffix.org.

Very happy to help provide this under your domain. Note however that that's probably more work for you, as certain stuff would have to be set up.

4. Our position so far has been "the PSL is a list" Trailing dots and the PSL. #792 , even if we specify an algorithm for interpreting it, that doesn't necessarily mean that everybody interprets it the same way.

The lookup implements the documented algorithm.

[simon-friedberger]

Any additional information about why people want this would be greatly appreciated. I found situations where DNS is allowed but HTTP is blocked to quite rare.

[weppos]

I won't enter into the discussion of whether the service it is useful or not, I'm sure there can be use cases for this integration like for many others.

Regardless, I don't think we can accept to become responsible for interacting with an external consumer that is beyond our control. For operational and liability reasons, as also @wdhdev correctly pointed out.

I disagree on adding this task to our repo. It can't become another responsibility on us.

There are multiple ways to stay in sync. For instance, you could clone the repo locally and use the git protocol once a minute to fetch updates. Or you can just poll the activity for the same frequency and re-download the list once updated.

[simon-friedberger]

@peterthomassen If you have any further information on what this is used for we would still be interested!

[peterthomassen]

I found situations where DNS is allowed but HTTP is blocked to quite rare.

That may be true, but in some networks dealing e.g. with private signing keys, HTTP requests are not allowed. More importantly, it is operationally more lightweight to add a zone to an existing DNS service than it is to develop, run, and maintain a HTTP service. (Backend database upgrades, web server patching, deployments, ...)

If you have any further information on what this is used for we would still be interested!

Some example known uses:

• We use this internally on our DNS hosting platform in order to determine whether to allow creation of a zone. For example, we prohibit creation of zones that are public suffixes, so that a malicious user doesn't create the co.uk zone, which would prevent other customers to create third-level zones like foobar.co.uk. (Generally, subzone creation must only be admissible in the same account, as otherwise hijacking risks lurk.)
• It seems like other registries like https://www.is-a.dev/ also use it, probably for the same purpose (they have the Python dependency).
• The discord bot of is-a.dev uses it.
• We know of at least one CA that uses this in their checks to prevent issuance of wildcard certs directly under a public suffix.

I don't know about the purpose of other uses, but there definitely is other use, as indicated by users creating issues.

[wdhdev]

I'm an admin for is-a.dev, we use the package in our Discord bot to just check if a certain suffix is listed on the PSL.

[simon-friedberger]

We use this internally on our DNS hosting platform in order to determine whether to allow creation of a zone. For example, we prohibit creation of zones that are public suffixes, so that a malicious user doesn't create the co.uk zone, which would prevent other customers to create third-level zones like foobar.co.uk. (Generally, subzone creation must only be admissible in the same account, as otherwise hijacking risks lurk.)

Interesting point, thank you! It seems like you could resolve this for your server internally by making zone delegation an explicit action which delegates to a user instead of to another (the same, probably) server. My understanding (please correct me, I am not a DNS expert) is that DNS was simply not designed to support multiple users on a single server and so to support this use-case any necessary compartmentalization should be added on top.

Would deploying DNSSEC only internally be an option? Does the server even validate the entries if they are in the same file?

[peterthomassen]

I think we're fully off-topic now, but ... ;-)

It seems like you could resolve this for your server internally by making zone delegation an explicit action which delegates to a user instead of to another (the same, probably) server.

You're suggesting to change what it means to have a delegation, but that's fixed in the specs, so let's avoid that term. -- I guess you meant that, if a user attempts to create a zone whose parent is owned by another user, we could send an approval request to the parent user, or something like that, so they can decide whether they agree with the "hijack".

However, that requires keeping state about such pending requests, and it opens up a way for users to spam other users.

More importantly, it does not solve the problem: Assuming there are no zones configured under co.uk on our servers, and then user A creates co.uk, would it then be justified to compel other users B, C, ... to ask user A for approval when they're trying to create foobar.co.uk?

I think that would be pretty weird. So, the above approval mechanism probably should have an exception for names registerable under a public suffix -- but then we're back to where we started, which is that one has to determine the public suffix of any zone that is about to be created.

DNS was simply not designed to support multiple users on a single server and so to support this use-case any necessary compartmentalization should be added on top.

The DNS specs require that, if multiple zones are configured where one's root (apex) is a subdomain of the other, answers are to be given from the more specific one (= the subdomain zone).

This is required because a server might legitimately host both the parent and the child, as is for example the case for uk and co.uk, which are separate zones but hosted on the same nameservers. In this case, the NS records for co.uk exist in both the child and the parent zone, but only the child zone is "authoritative" (the parent's delegation is merely a hint, according to the DNS specs), so querying for NS must return the child zone records. This is ensured by the above requirement. (This ambiguity in record storage on both sides of a "zone cut" has been a repeating cause of headache in the DNS community, but as far as the protocol is concerned, it is what it is, until a major revision is done.)

As a result, the problem of subzone hijacking necessarily occurs in shared environments. Once could work around it by using different (perhaps virtual) nameservers. However, that in turn would require customer-specific nameserver hostnames, which adds quite some complexity, decreases UX (you no longer can state generic hostnames in documentation), and falls apart as soon as a user transfers ownership for a zone to another user. The cleaner solution is to acknowledge that the global DNS tree is shared, and implement write access controls accordingly.

This is typically done based on the PSL (although some operators have neglected that, see the paper linked earlier). We decided that, mostly for maintenance reasons, we did not want to embed the PSL within our application (as I'm sure many applications would rather not), so we were looking for a query service. As we couldn't find any, and as we happen to run a DNS service, we came up with a transformation for representing the PSL in a DNS zone, which allows PSL lookups via simple DNS queries. That solved the problem without any extra components.

Would deploying DNSSEC only internally be an option?

We use DNSSEC for all zones. However, DNSSEC in itself does not prevent creation of subzones by other users, although it prevents validation of responses from that subzone if the subzone's keys aren't included in the delegation from the parent (using DS records in the parent), so hijacked subzones return SERVFAIL responses to clients using a validating resolver. As a result, DNSSEC turns a hijack into a DoS, but it does not prevent the loss of availability.

Does the server even validate the entries if they are in the same file?

I'm not sure what file you mean.

@wdhdev, in sse-secure-systems/psl-dns#7 (comment) you hinted at Cloudflare using this service. I'd be interested in learning more!

[wdhdev]

Yes, I believe I sent you an email about that. Not sure if you received it or not.

[peterthomassen]

Yes, I believe I sent you an email about that. Not sure if you received it or not.

Unfortunately I didn't receive it, but I just shot you a message to the address listed in your GitHub profile.

[simon-friedberger]

I think we're fully off-topic now, but ... ;-)

I assume it doesn't bother anyone. :)

It seems like you could resolve this for your server internally by making zone delegation an explicit action which delegates to a user instead of to another (the same, probably) server.

You're suggesting to change what it means to have a delegation, but that's fixed in the specs, so let's avoid that term. -- I guess you meant that, if a user attempts to create a zone whose parent is owned by another user, we could send an approval request to the parent user, or something like that, so they can decide whether they agree with the "hijack".

I mean... I kind of agree, but on the other hand the spec is concerned with server interoperability and doesn't really care about what people do inside of their organizations. So since you are defining doing something like delegation but inside one server, it doesn't seem strange to me to call it delegation. Whatever we call it, I think we agree about the idea. :)

However, that requires keeping state about such pending requests, and it opens up a way for users to spam other users.

Correct, but that is a very solvable problem if all users are within your system.

More importantly, it does not solve the problem: Assuming there are no zones configured under co.uk on our servers, and then user A creates co.uk, would it then be justified to compel other users B, C, ... to ask user A for approval when they're trying to create foobar.co.uk?

I think that would be pretty weird. So, the above approval mechanism probably should have an exception for names registerable under a public suffix -- but then we're back to where we started, which is that one has to determine the public suffix of any zone that is about to be created.

I think you can fix this by defining that the apex is owned by you, the server owner, right? That way there is never any "unclaimed space" that people could just snatch up.

DNS was simply not designed to support multiple users on a single server and so to support this use-case any necessary compartmentalization should be added on top.

The DNS specs require that, if multiple zones are configured where one's root (apex) is a subdomain of the other, answers are to be given from the more specific one (= the subdomain zone).

This is required because a server might legitimately host both the parent and the child, as is for example the case for uk and co.uk, which are separate zones but hosted on the same nameservers. In this case, the NS records for co.uk exist in both the child and the parent zone, but only the child zone is "authoritative" (the parent's delegation is merely a hint, according to the DNS specs), so querying for NS must return the child zone records. This is ensured by the above requirement. (This ambiguity in record storage on both sides of a "zone cut" has been a repeating cause of headache in the DNS community, but as far as the protocol is concerned, it is what it is, until a major revision is done.)

I assume the problem you mean is that there might be a child-zone the parent is not aware of and is not expecting to delegate but it still takes precedence?

As a result, the problem of subzone hijacking necessarily occurs in shared environments. Once could work around it by using different (perhaps virtual) nameservers. However, that in turn would require customer-specific nameserver hostnames, which adds quite some complexity, decreases UX (you no longer can state generic hostnames in documentation), and falls apart as soon as a user transfers ownership for a zone to another user. The cleaner solution is to acknowledge that the global DNS tree is shared, and implement write access controls accordingly.

I think you are making one step too many here. :) First of all, I also thought about per-user maybe-virtual nameservers and I agree that it sounds like it would create more problems than it solves. However, the part I am wondering about here is that you are saying the problem necessarily occurs in shared environments and therefore it should be acknowledged that the global DNS tree is shared. What am I missing here? Why can't the shared environment take care of it internally without the rest of the world ever finding out?

This is typically done based on the PSL (although some operators have neglected that, see the paper linked earlier). We decided that, mostly for maintenance reasons, we did not want to embed the PSL within our application (as I'm sure many applications would rather not), so we were looking for a query service. As we couldn't find any, and as we happen to run a DNS service, we came up with a transformation for representing the PSL in a DNS zone, which allows PSL lookups via simple DNS queries. That solved the problem without any extra components.

Do you have any numbers or references for this "typically"?

Would deploying DNSSEC only internally be an option?

We use DNSSEC for all zones. However, DNSSEC in itself does not prevent creation of subzones by other users, although it prevents validation of responses from that subzone if the subzone's keys aren't included in the delegation from the parent (using DS records in the parent), so hijacked subzones return SERVFAIL responses to clients using a validating resolver. As a result, DNSSEC turns a hijack into a DoS, but it does not prevent the loss of availability.

Does the server even validate the entries if they are in the same file?

I'm not sure what file you mean.

Sorry, that was just my mental model of everything on the same server being in the same file. What I actually meant is, if the server is checking the validity of the response before sending it. It seems a bit silly to check its own response but in the kind of shared server scenario we're discussing it would make sense.

[peterthomassen]

However, that requires keeping state about such pending requests, and it opens up a way for users to spam other users.

Correct, but that is a very solvable problem if all users are within your system.

Of course that's solvable, it's just more complex and less data-minimalistic than necessary.

More importantly, it does not solve the problem: Assuming there are no zones configured under co.uk on our servers, and then user A creates co.uk, would it then be justified to compel other users B, C, ... to ask user A for approval when they're trying to create foobar.co.uk?
[...]

I think you can fix this by defining that the apex is owned by you, the server owner, right? That way there is never any "unclaimed space" that people could just snatch up.

That would mean that we get approval requests for all domains that get created on our systems and that don't have any parent zone on our servers. We have tens of thousands of zones; I don't think we should be doing this manual work.

Also, how should we be doing the assessment for approval purposes, if someone e.g. "applies" to create the zone asse.eu.org or asso.eu.org? The latter is a public suffix while the former isn't, and we'd have to look in the PSL to make sure. We're going in circles.

I assume the problem you mean is that there might be a child-zone the parent is not aware of and is not expecting to delegate but it still takes precedence?

Yes.

However, the part I am wondering about here is that you are saying the problem necessarily occurs in shared environments and therefore it should be acknowledged that the global DNS tree is shared. What am I missing here? Why can't the shared environment take care of it internally without the rest of the world ever finding out?

Indeed it can be taken care of internally, as I said above (at the cost of complexity, manual labor and/or other nastiness).

I did not say that "therefore" the sharedness of the global DNS tree should be acknowledged; I said that a cleaner solution is achieved when acknowledging that sharedness, with its boundaries of registerability modeled in the PSL.

Do you have any numbers or references for this "typically"?

In writing the paper quoted above, I tested around 25 operators and more than 20 were immune even for rare public suffixes, so must somehow be using the PSL (although perhaps not an up-to-date version). See Table 1 in that paper.

[simon-friedberger]

Also, how should we be doing the assessment for approval purposes, if someone e.g. "applies" to create the zone asse.eu.org or asso.eu.org? The latter is a public suffix while the former isn't, and we'd have to look in the PSL to make sure. We're going in circles.

Well, this is way outside my comfort zone so thank you for bearing with me! :)

Could you maybe explain what exactly the attack is? Is it the following?

Somebody comes to you and asks you to be the apex for apple.asso.eu.org and you add it but don't verify anything.
And now at a later point the actual owner of asso.eu.org also decides to use your service and actually delegates it to you and now suddenly the first entry becomes active?

[peterthomassen]

Also, how should we be doing the assessment for approval purposes, if someone e.g. "applies" to create the zone asse.eu.org or asso.eu.org? The latter is a public suffix while the former isn't, and we'd have to look in the PSL to make sure. We're going in circles.

Well, this is way outside my comfort zone so thank you for bearing with me! :)

Could you maybe explain what exactly the attack is?

Let's first consider two scenarios:

1. If someone wants to create the zone asse.eu.org (which is not a public suffix), that should be possible at a managed DNS hoster (who shares nameserver hostnames across customers). Subsequently, the same user should of course be able to create additional subzones of this zone (but not other users, because hijacking).
2. If someone wants to create the zone asso.eu.org (which is a public suffx), that should be denied, so that other users are not restricted and can legitimately can create subzones of this name (e.g., foobar.asso.eu.org). They can then inform the asso.eu.org registry (where they have registered the foobar subdomain) about which nameserver hostnames to include in their delegation.

Distinguishing these cases (so that subzone creation can be allowed/denied as appropriate) requires doing a PSL lookup. Without the PSL lookup, the two cases above cannot be distinguished, so you'll have to treat them the same, in one of the two following ways:

• either you don't restrict subzone creation to the owner of the parent zone (if present on the system), but allow any user to create any zone. This leads to the hijacking problem explained earlier.
• or you always restrict subzone creation to the owner of the parent zone (if present on the system), which enables a malicious user to register a public suffix as a zone, as a consequence of which nobody else can create zones for their legitimately registered names (like foobar.asso.eu.org, or foobar.co.uk).

To resolve this conundrum when creating a zone with some $name, you need to figure out whether any other user on the same system holds a parent zone of $name, and if so deny creation of the subzone.

[For full precision, only consider parent zones lying between $name and its (longest) public suffix included; any parent zones further away (in the above example: eu.org) are out of scope. -- This also blocks creation of public suffix zones like co.uk, which is OK unless you're hosting a registry which needs special treatment anyway for its public domain registration process.]

Note that for all of this, it doesn't matter whether decision is taken automatically or whether some kind of approval message is used (because in either case, the process involves a PSL lookup, manual or automatic). There's no way around it.

[simon-friedberger]

Thank you for all the info!