DNS security: filtering, DNS tunneling, and DNSSEC

What DNS traffic reveals about an infection, how DNS filtering works, how to recognize DNS tunneling, and what DNSSEC actually protects.

Topic
Networks
Author
ARLITECH expert team
Published

Almost every network connection starts with name resolution. Before a browser, an email client or a service looking for updates connects to a server, it asks DNS (Domain Name System) which IP address belongs to the name. The query and the answer usually travel unencrypted, in small UDP packets, and in many networks the firewall lets them through without much inspection.

That gives DNS two roles in security. It is a good observation point, because an infection often shows up in the queries before any other sign appears. Attackers use it for the same reason: a channel that is allowed everywhere can be used to smuggle out both command traffic and data. Both sides are worth a look, together with encrypted DNS and DNSSEC, which people often assume do more than they actually do.

What DNS does, and why attackers care about it

DNS is defined in IETF RFC 1034 and RFC 1035 (1987). The namespace is hierarchical: below the root are the top-level domains (.hu, .com), and below those are the organizations' zones. Resolution involves two separate roles. The authoritative name server holds the authoritative data for a zone. The recursive resolver walks the hierarchy from the root down on the client's behalf and keeps the answer in its cache for the record's time to live (TTL). On a corporate network, clients typically query the internal DNS server, which either resolves recursively itself or forwards the query to an external resolver.

What DNS traffic reveals

Once the malware is installed, an infected machine tries to reach the attacker's command and control (C2) server. In most cases it has to resolve a name to do that, so the query shows up in the internal DNS server's log, even if the connection that follows is encrypted. Suspicious signs can include:

  • queries for known malware, phishing or C2 domains;
  • newly registered domains that no machine in the company has queried before;
  • many failed (NXDOMAIN) responses from a single machine, for names that look random;
  • unusually long subdomains that look like encoded data, and a high number of queries toward a single domain;
  • a machine that bypasses the internal DNS server and goes straight to an external resolver.

The MITRE ATT&CK knowledge base lists the use of DNS as a command channel as a technique of its own (T1071.004, Application Layer Protocol: DNS).

Tampering with name resolution also lets an attacker redirect traffic. A forged response can send the user to a fake login page while the familiar name stays in the address bar. On an HTTPS site, the browser will usually reject the fake server's certificate, but with an unencrypted service, or an application that does not check certificates, nothing raises a warning.

DNS filtering: categories, threat lists, custom rules

DNS filtering makes its decision at the resolver. If the requested name is blocked, the resolver returns an error response (NXDOMAIN or REFUSED) or the address of a warning page instead of the real address. The connection is never made. Because the decision happens at name resolution, filtering works regardless of protocol: it applies to web traffic, to the email client, and to malware that tries to connect over a protocol of its own. A joint guide from the US NSA and CISA (Selecting a Protective DNS Service, 2021) discusses this kind of filtering resolver under the name “protective DNS”.

Where the rules come from

  • Threat lists. Lists of malware, phishing and C2 domains that the filtering service or the firewall vendor keeps updating. Most of the protection comes from these, and their quality depends on how quickly a new campaign's domain gets added.
  • Categories. Domains classified by content, for example gambling, file sharing, newly registered or not yet categorized. Blocking or monitoring newly registered domains is a common setting, because phishing and malware campaigns often use names that are only a few days old. The downside is that a legitimate new service can get caught too until it has been categorized.
  • Custom rules. The organization's own block list and allow list. Domains found during an incident can go on the block list, and partner sites that the category filter blocks by mistake can go on the allow list.

Many DNS servers accept external block lists in Response Policy Zone (RPZ) format. The list is then a special zone that the resolver downloads from the publisher with an ordinary zone transfer, so updates need no separate mechanism.

Limitations

DNS filtering only works if the client queries the filtering resolver. A manually configured external DNS server, an application with its own resolver, or encrypted DNS gets around it. It also cannot see connections that go straight to an IP address without a name. And it cannot tell things apart within a single domain: if the malicious file sits on the domain of a legitimate hosting or file sharing service, blocking the name locks everyone out, and allowing it lets the malware through as well. In that case the firewall's URL and content filtering can make a finer-grained decision, as described in the article on next-generation firewalls.

A blocked query also makes a good alert. If a machine keeps querying a C2 domain, it is probably infected, even though the filtering stopped the connection. That is why the filter's log should be collected and monitored along with the other security logs, rather than just counting the blocks.

Detecting DNS tunneling and generated domain names

DNS tunneling uses queries to carry data. The client encodes the data it wants to send and writes it into a subdomain of a domain the attacker runs, as a long, random-looking string at the start of the name. The query passes through the internal DNS server and the external resolvers to the attacker's authoritative name server, which sends its reply back encoded as well, typically in a TXT, NULL or CNAME record. The infected machine does not need direct internet access for this. It is enough that the internal DNS server resolves external names recursively.

RFC 1035 limits a label to 63 octets and a full name to 255, so a single query carries little data. A tunnel therefore consists of many queries sent in quick succession, and that is what gives it away.

Telltale signs

  • long, high-entropy subdomains that look like random strings of letters and digits, often close to the maximum label length;
  • a very large number of subdomains under one domain, each one different;
  • unusual record types: many TXT or NULL queries from a machine that otherwise only asks for addresses;
  • queries at a steady rhythm, or spikes outside working hours;
  • a large total volume of data in queries toward one domain.

None of these signs is proof on its own. Legitimate services also use long, encoded subdomains, for example some antivirus cloud lookups, block list (DNSBL) lookups and telemetry queries. Detection therefore relies on the signs in combination, plus an exception list that takes known, benign sources out of the analysis.

Generated domain names (DGA)

A domain generation algorithm (DGA) is a routine built into the malware that regularly produces a large number of candidate names, for example from the date. The attacker registers only a few of them, and the malware works through the list until it reaches a live name. Blocking one name therefore does not cut the connection. On the network, it looks like a machine querying many meaningless names in a short time, with most of them getting an NXDOMAIN response because they are not registered. MITRE ATT&CK describes this technique under ID T1568.002.

A related method is fast flux (T1568.001): the same name points to IP addresses that change frequently, with a very short TTL, which makes the servers behind it harder to take down one by one or to block by address.

What you can do about it

A tunnel depends on internal machines' DNS traffic getting out freely, or on the internal server resolving any external name without question. If the firewall allows outbound port 53 (UDP and TCP) only for the internal DNS servers, every query passes through one place, where it can be logged and inspected. Rules based on the signs above can then be set up on the internal servers or in the firewall's DNS inspection, and many DNS filtering services also include DGA and tunnel detection. The article on logging covers in more detail which logs to collect and how long to keep them.

Encrypted DNS: DoH and DoT on the corporate network

A traditional DNS query is unencrypted. Anyone along the path can see it, and an attacker can also modify it. There are two standard answers to this. DNS over TLS (DoT, IETF RFC 7858, 2016) sends queries inside a TLS connection on the dedicated TCP port 853. DNS over HTTPS (DoH, IETF RFC 8484, 2018) wraps them in HTTPS requests and runs on port 443, the same port as web traffic.

On public Wi-Fi or a home network this is real protection, because neither the network operator nor an attacker can see or forge the queries. On a corporate network it cuts both ways. If a browser or application uses its own built-in DoH resolver, the queries bypass the internal DNS server: DNS filtering does not apply to them, internal names do not resolve, and they are missing from the query log. Malware can take advantage of the same thing, because on the network DoH traffic looks like ordinary HTTPS.

How to keep it under control

  • Encryption can stay, as far as the internal resolver. If the internal server supports it, clients can connect to it over DoT or DoH, and the query goes on from there.
  • In most browsers and operating systems, built-in DoH can be turned off through central policy or tied to a specified resolver. On managed machines this is the most reliable option.
  • Outbound port 853 can be blocked for everything except the internal resolvers, so DoT cannot leave the network.
  • DoH cannot be filtered by port. Firewalls block it using an address list of known public DoH resolvers or through application identification. This works for the large, well-known providers, but not for an attacker's own DoH server.

For unmanaged devices and malware, then, blocking is not a complete answer. Here the network treats any name resolution outside the internal DNS as suspicious, and a machine that talks to an unknown HTTPS endpoint conspicuously often instead of to the internal resolver deserves a closer look.

DNSSEC: what it authenticates and what it does not

DNSSEC (DNS Security Extensions, IETF RFC 4033, 4034 and 4035, 2005) uses digital signatures to prove that a DNS response comes from the zone owner and was not modified in transit. The zone operator signs its records. The signature goes into the zone as an RRSIG record, the public key as a DNSKEY record. The parent zone stores a hash of the child zone's key in a DS record, so the chain of trust can be followed back to the root zone, which has been signed since 2010. A validating resolver walks this chain, and if a signature does not match, it discards the response and returns a SERVFAIL error to the client.

What it protects against

DNSSEC protects against forged responses, whether they come from cache poisoning or from a packet rewritten along the path. In cache poisoning, the attacker tries to get a fake response to the resolver ahead of the real one. If that succeeds, the resolver hands the fake address to every client until the TTL expires. On top of the 16-bit transaction ID defined in RFC 1035, IETF RFC 5452 (2009) also recommends source port randomization, which makes guessing much harder but not impossible. With DNSSEC, a forged response can be detected cryptographically. Signed proof of nonexistence (NSEC, or NSEC3, IETF RFC 5155, 2008) also proves that a name really does not exist, so the attacker cannot forge a “no such name” response either.

What it does not protect against

  • It does not encrypt. Queries and responses are just as readable as before; the confidentiality of queries is what DoT and DoH protect.
  • It does not tell you whether a domain is benign. A phishing domain can be signed flawlessly too. DNSSEC only proves that the response comes from the zone owner.
  • It only works where the zone is signed and the resolver validates. The client typically does not validate on its own and relies on the resolver's “verified” flag (the AD bit). If the segment between the client and the resolver is unprotected, that flag can be forged too.
  • It does not protect against a compromised registrar account. Anyone who can change the domain's name servers or DS record can also produce valid signatures with their own key.

Operational risk

A signed zone needs maintenance. Signatures expire, keys have to be replaced from time to time, and the DS record at the registrar has to be updated in step with the key rollover. If this slips, validating resolvers see the whole domain as unreachable while the servers are working perfectly. Before you sign your own zone, it pays to work out the key rollover process and how expiry dates will be monitored. Turning on validation on the internal resolver carries less risk and immediately protects you for signed external names.

Protecting internal DNS servers

On many networks the internal DNS server is part of the directory service. If it goes down or returns false data, logons, file sharing and almost every internal application go down with it. Protecting it is therefore about more than filtering.

Roles and access

  • Recursive resolution should be available only to the internal network. Attackers use open resolvers, which answer anyone on the internet, for amplification denial-of-service attacks: they send a small query with the victim's address as the spoofed source address, and the resolver sends a much larger response to the target. The article on DDoS covers denial-of-service attacks.
  • The authoritative zone published to the internet should preferably be served by a separate server or provider, not by the internal server that also holds the internal names. Separating the internal and external views (split-horizon DNS) also ensures that internal names and addresses cannot be queried from outside.
  • Zone transfers (AXFR, IETF RFC 5936, 2010) should be allowed only to the named secondary servers, preferably authenticated with TSIG (IETF RFC 8945, 2020). An internal zone that anyone can download hands an attacker a ready-made map of the network.
  • Dynamic updates, which clients use to register their own records, should be allowed only with authentication. Otherwise an internal machine can overwrite other machines' names as well.

Outbound traffic

Clients should reach only the internal DNS servers, and outbound ports 53 and 853 should be open only from the resolvers. The internal resolver can forward queries to a filtering resolver, so filtering applies to every machine and the query log is produced in one place.

Operations

  • At least two servers, on different hardware or at different locations, because a DNS outage stops the whole network.
  • Updating the DNS software according to the vendor's security advisories. Resolvers process responses that come from outside, so their vulnerabilities can be exploited remotely.
  • Collecting and retaining the query log, so that during an incident you can look up which machine queried a suspicious name and when.
  • Administrative access through named accounts with multi-factor authentication, from a management network kept separate from the user network.
  • Monitoring record changes, because an unexpected change to an internal name can be part of an attack.
Quick self-test: from a user machine, look up a name directly against a well-known public DNS resolver on port 53 (for example with the nslookup command, giving the resolver's address). If you get an answer, clients can probably bypass the internal DNS, and the filtering with it, unless the firewall redirects the request to the internal resolver. The internal resolver's query log will tell you: if the query appears there, it was redirected. The test only checks unencrypted DNS on port 53, not bypass over DoT or DoH.

Related service: Firewall and VPN →

Contact

Ready to secure your critical systems?

Our experts will assess where your security stands and build a roadmap that fits your organization.

ARLITECH's office in Gödöllő

Visit us

You are welcome to visit our Gödöllő office in person.

Address
Centrum Office Building, 2100 Gödöllő, Dózsa György út 13, 3rd floor, office 309
Office hours
Mon–Fri, 9 a.m. to 5 p.m.
Get directions (opens in a new tab)