Find the registrable domain of a subdomain with the Public Suffix List

A public suffix is an ending under which names can be registered: com, but also co.uk or com.br. A registrable domain is a public suffix plus the one label to its left, such as example.co.uk. Any longer name, such as shop.example.co.uk, is a subdomain of it.

Why not the last two labels

Taking the last two labels works for www.example.com but gives co.uk for shop.example.co.uk. co.uk is a public suffix itself, like com: names are registered under it, so it is not anyone's registrable domain. The name alone does not say where the suffix ends: example.uk is a registrable domain under uk, and example.co.uk one under co.uk. Only a list of suffixes can tell.

In the domains file of the current release, 30,236,834 registrable domains (9.6%) sit under a public suffix of two or more labels; co.uk alone has 5,723,758. The last-two-labels rule gets each of them wrong.

The Public Suffix List

The Public Suffix List is a community-maintained text file, public_suffix_list.dat, with one rule per line in two sections:

• ICANN: the top-level domains and the endings below them that their registries open for registration, such as com, uk and co.uk.
• PRIVATE: endings that companies open to others, such as a service that gives each customer a name under its own domain.

Most rules are plain suffixes. A rule that starts with *. makes each name directly under an ending a public suffix, and a rule that starts with ! is an exception that turns one of those back into a registrable domain. The longest matching rule wins, and an exception wins over the others. Internationalized endings are written in Unicode in the list, while our files use their xn-- form: .ไทย is .xn--o3cw4h.

What our files already do

Each release splits its names with a fixed copy of the ICANN section. Registrable domains are in the domains file and the names below them are in the subdomains file, so the two files do not overlap: example.co.uk can only be in the domains file, and shop.example.co.uk only in the subdomains file. The PRIVATE section is not used, so a name under such a service is a subdomain of the service's own registrable domain.

You need the list for your own names: to find the registrable domain to look up in the domains file, or to group subdomains by the domain they belong to. A newer copy of the list may classify some names differently from the copy a release used.

A Python script with no packages

Download the list once and keep the copy:

curl -fsSL https://publicsuffix.org/list/public_suffix_list.dat -o public_suffix_list.dat

Save this as registrable.py. It reads only the ICANN section and turns internationalized rules into their xn-- form:

import sys

def ascii_label(label):
    # The list writes internationalized suffixes in Unicode; the files use xn-- labels.
    return label if label.isascii() else "xn--" + label.encode("punycode").decode("ascii")

def icann_rules(path):
    rules, icann = set(), False
    for line in open(path, encoding="utf-8"):
        line = line.strip()
        if line.startswith("// ===BEGIN ICANN DOMAINS==="):
            icann = True
        elif line.startswith("// ===END ICANN DOMAINS==="):
            break
        elif icann and line and not line.startswith("//"):
            rules.add(".".join(ascii_label(l) for l in line.split(".")))
    return rules

def registrable_domain(name, rules):
    labels = name.strip().lower().rstrip(".").split(".")
    n = 1                                    # no rule: the last label is the suffix
    for i in range(len(labels)):             # the longest candidate suffix first
        suffix = ".".join(labels[i:])
        if "!" + suffix in rules:            # an exception: one label less
            n = len(labels) - i - 1
            break
        if suffix in rules or "*." + ".".join(labels[i + 1:]) in rules:
            n = len(labels) - i
            break
    return ".".join(labels[-n - 1:]) if len(labels) > n else None

rules = icann_rules(sys.argv[1])
for line in sys.stdin:
    print(line.strip(), registrable_domain(line, rules) or "-")

It reads one name per line and prints each name with its registrable domain, or - for a name that is a public suffix itself:

printf '%s\n' www.example.com shop.example.co.uk a.b.example.com.br www.example.xn--o3cw4h co.uk | python3 registrable.py public_suffix_list.dat

www.example.com example.com
shop.example.co.uk example.co.uk
a.b.example.com.br example.com.br
www.example.xn--o3cw4h example.xn--o3cw4h
co.uk -

Check it on the free samples

curl -fsSL https://getdomainlists.com/samples/getdomainlists-domains-sample.txt -o domains-sample.txt
curl -fsSL https://getdomainlists.com/samples/getdomainlists-subdomains-sample.txt -o subdomains-sample.txt
python3 registrable.py public_suffix_list.dat < domains-sample.txt | awk '$1 == $2' | wc -l
python3 registrable.py public_suffix_list.dat < subdomains-sample.txt | awk '$1 != $2' | wc -l

With the copy of the list our files were split with, both print 1000: each name in the domains sample is its own registrable domain, and each name in the subdomains sample sits below one. For a full file, stream it: gzip -dc subdomains.txt.gz | python3 registrable.py public_suffix_list.dat. The script keeps only the rules in memory, but a pass over the subdomains file takes a while in plain Python.

Libraries

Libraries that read the Public Suffix List include tldextract, publicsuffixlist and publicsuffix2 for Python, tldts and psl for JavaScript, and golang.org/x/net/publicsuffix for Go. Before relying on one, check whether it also uses the PRIVATE section and how it handles xn-- names, because our files use only the ICANN section, in xn-- form.

Understand the result

A registrable domain comes from the rules of the list. It does not show who owns or runs a name, and it is not evidence that a name is registered, resolves or hosts content today.

Try the format first with the free 1,000-name samples. Both full files are $9 once, with 30 days to download.

To look up registrable domains in our files, see the matching guide or the DuckDB guide.