Add DNS response details

[pmeenan]

Currently, CNAMES and other DNS details are only stored as part of the main page, and largely done using dedicated lookups.

We should add the passive details that are already available in the netlog for all of the origins resolved (CNAMES, SVCB records, address list, etc).

The main question is WHERE to put the data. The options that come to mind are:

1. At a request level for the first request from a given origin
2. At a request level, copied to every request for a given origin (seems wasteful)
3. At a page level, with per-origin details (harder to query?)

The raw results look something like this:

"results": [
     {
       "alias_target": "ghs.googlehosted.com",
       "domain_name": "almanac.httparchive.org",
       "query_type": "AAAA",
       "source": "dns",
       "timed_expiration": "13411495234628602",
       "type": "alias"
     },
     {
       "alias_target": "ghs.googlehosted.com",
       "domain_name": "almanac.httparchive.org",
       "query_type": "HTTPS",
       "source": "dns",
       "timed_expiration": "13411495234636313",
       "type": "alias"
     },
     {
       "alias_target": "ghs.googlehosted.com",
       "domain_name": "almanac.httparchive.org",
       "query_type": "A",
       "source": "dns",
       "timed_expiration": "13411495234628470",
       "type": "alias"
     },
     {
       "domain_name": "ghs.googlehosted.com",
       "error": -105,
       "query_type": "HTTPS",
       "source": "dns",
       "timed_expiration": "13411494994636313",
       "type": "error"
     },
     {
       "domain_name": "ghs.googlehosted.com",
       "endpoints": [
         {
           "address": "2607:f8b0:4004:c08::79",
           "port": 0
         },
         {
           "address": "142.251.111.121",
           "port": 0
         }
       ],
       "hosts": [],
       "query_type": "UNSPECIFIED",
       "source": "dns",
       "strings": [],
       "timed_expiration": "13411494945628470",
       "type": "data"
     }
   ]

[pmeenan]

Main question will be how to make it useful to query because the raw result looks like it would be kind of a pain.

This is also explicitly only recording passively what Chrome actually looked up. We won't be actively probing MX records, etc.

@nrllh FYI

@tunetheweb thoughts?

[tunetheweb]

My vote is 3 as doesn’t have the issues of 1 (seems weird and somewhat arbitrary to be on the first one when querying it) and 2 (wasteful as you say).

[pmeenan]

I guess it depends if you want to query by page or by origin. If you want to lookup or count origin-level data, having it attached to the requests feels like it would be easier than pulling it out of the page data and exploding the individual origins from there (but I could easily be wrong).

Knowing what people would want to do with the information would help. The only thing that comes to mind for me is reporting on the number of origins that have adopted HTTPS records for various features.

[pmeenan]

Looks like I already added it at some point. It is dns_info_ on the request retails for the request that triggered a DNS lookup:

"dns_info": {
    "secure": false,
    "transactions_needed": [
        {
            "dns_query_type": "HTTPS"
        },
        {
            "dns_query_type": "A"
        }
    ],
    "results": [
        {
            "domain_name": "tls-ech.dev",
            "endpoints": [
                {
                    "address": "34.138.246.121",
                    "port": 0
                }
            ],
            "hosts": [],
            "query_type": "A",
            "source": "dns",
            "strings": [],
            "timed_expiration": "13411589730342294",
            "type": "data"
        },
        {
            "domain_name": "tls-ech.dev",
            "metadatas": [
                {
                    "metadata_value": {
                        "ech_config_list": "AEn+DQBFKwAgACABWIHUGj4u+PIggYXcR5JF0gYk3dCRioBW8uJq9H4mKAAIAAEAAQABAANAEnB1YmxpYy50bHMtZWNoLmRldgAA",
                        "supported_protocol_alpns": [
                            "http\/1.1"
                        ],
                        "target_name": "tls-ech.dev"
                    },
                    "metadata_weight": 1
                }
            ],
            "query_type": "HTTPS",
            "source": "dns",
            "timed_expiration": "13411589490340443",
            "type": "metadata"
        }
    ]
}

That's pretty verbose though and I'm not sure it is easy to query. I can boil down the useful data and either attach it to every request or put it in the page data:

"dns_details": {
    "HTTPS": {
        "ech": "AEn+DQBFKwAgACABWIHUGj4u+PIggYXcR5JF0gYk3dCRioBW8uJq9H4mKAAIAAEAAQABAANAEnB1YmxpYy50bHMtZWNoLmRldgAA",
        "alpn": [
            "http\/1.1"
        ]
    },
    "A": {
        "addr": [
            "34.138.246.121"
        ]
    }
}

[nrllh]

Thanks for moving this forward, Patrick!

Regarding the storage location: what if we store these details at the request level (Option 1) to cover all resources, but also expose these new DNS fields in the pages table specifically for the main origin? That way we get the depth for all requests without losing top-level accessibility, which is likely where most queries will focus..

Separately, I wonder if it’s manageable to also query full active DNS records (TXT, A, MX, etc.) for websites in CrUX? Since DNS lookups are generally lightweight, adding this would significantly enrich the HTTP Archive dataset. Do you think that is feasible to include in the pipeline?

[pmeenan]

The page-level JSON already has a bunch of DNS records that are explicitly looked up as part of the crawl under origin_dns_:

"origin_dns": {
    "a": [
        "34.138.246.121"
    ],
    "ns": [
        "ns-cloud-e2.googledomains.com.",
        "ns-cloud-e4.googledomains.com.",
        "ns-cloud-e3.googledomains.com.",
        "ns-cloud-e1.googledomains.com."
    ],
    "soa": [
        "ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300"
    ],
    "https": [
        "1 . ech=\"AEn+DQBFKwAgACABWIHUGj4u+PIggYXcR5JF0gYk3dCRioBW8uJq9H4mKAAIAAEAAQABAANAEnB1YmxpYy50bHMtZWNoLmRldgAA\""
    ],
    "aaaa": [],
    "cname": [],
    "mx": [],
    "txt": [],
    "svcb": []
},