Ciprian Lucanu

Is It the Network or the Website?

Author image
Ciprian Lucanu(community contributor)

9 min read

0
Article lead image

Some visitors couldn't open a US school's website. Packet captures showed a mobile carrier forging TLS replies and a VPN refusing DNS lookups, both because the domain was on a reputation blocklist. With a credit grant from RIPE Atlas, we checked DNS and TLS from 45 probes inside 21 US ISPs.


We build and host websites for schools. Earlier this month, one of our client schools in the United States told us that some parents and staff could not open its website. On campus it worked fine. At home, on some connections, the browser showed a connection error. The site sits behind Cloudflare, alongside other school sites we host that had no complaints.

"It doesn't load for some people" is a hard report to act on. The cause could be the website, the CDN, DNS, the visitor's device, their home router or their internet provider. We needed to find out which, and then whether the problem affected one household or a whole provider's customer base.

Step 1: Packet captures from the affected connections

On Cloudflare's support suggestion, we started with Wireshark captures taken by school staff on four kinds of connection: the campus network, a home cable connection, a T‑Mobile connection, and a commercial VPN. We captured two hostnames in each case: the affected school's domain, and a second school's domain hosted on the same Cloudflare IP addresses, as a control.

Two failures stood out.

On T‑Mobile, TLS connections were being interrupted by something in the network. The TCP handshake with Cloudflare completed normally. But as soon as the browser sent its TLS ClientHello with the school's hostname in the SNI field, a 256‑byte "reply" arrived made up entirely of 0xFF bytes, followed by three TCP resets. The browser treated the junk as a broken server and gave up with a TLS protocol_version alert. It was easy to see that these packets did not come from Cloudflare:

Packet TTL IP ID DF flag
Cloudflare SYN‑ACK (genuine) 47 varies set
"Reply" to the ClientHello 255 0 not set
TCP resets 64 0 set

Genuine Cloudflare packets had crossed many routers, so their TTL had dropped into the 40s. The injected ones still had a TTL of 255. Plain HTTP to the same hostname got a forged 302 redirect to a T‑Mobile security warning page, which identified the filter. The control domain, on the same Cloudflare IP, loaded normally on the same connection. The filter was matching on the hostname in the SNI field, not on the destination IP.

On the VPN, DNS was being filtered. The VPN provider's resolver answered NXDOMAIN for the school's domain and added an additional record, explanation.invalid TXT "Proton DNSBL/1b", which marked the answer as coming from its blocklist. The browser never opened a connection.

Two unrelated security products blocking the same domain pointed to a shared reputation feed. A check of public domain blocklists found the domain on SURBL's ABUSE list (127.0.0.64 from multi.surbl.org). The control domain was not listed. SURBL's own lookup page confirmed the listing, and the school began working on the removal.

Step 2: The question we couldn't answer from here

The school's next question was reasonable: which internet providers block us? Families use many different providers, and a T‑Mobile capture tells you about one household on T‑Mobile.

This is hard to answer from outside. ISP filtering usually lives inside the provider's network: in its DNS resolvers, in the customer's gateway, or in in‑path equipment like the T‑Mobile filter above. Querying those resolvers from outside doesn't work, because most are closed to non‑customers. VPN exit nodes don't help either, since they sit in data centres rather than on residential ISPs, and the VPN applies its own filtering, as we had just seen. We checked OONI's public measurement data, but it had no measurements of this domain from the US.

What we needed was a set of vantage points inside each provider's network, on real home connections. That is exactly what RIPE Atlas provides.

Step 3: Measuring from inside 21 ISPs with RIPE Atlas

We did not have any RIPE Atlas credits, so we wrote to the RIPE Atlas team, explained the investigation and asked for a small grant. They credited our account with 5,000 credits within a day. Without that, this step would not have happened.

Probe selection

We mapped the 21 largest US residential providers to their ASNs. Several needed more than one: Charter's former Time Warner Cable and Bright House networks, Altice's Optimum and Suddenlink, Verizon's wireless ASNs, Frontier's legacy networks, and Astound's RCN, Grande and Wave networks. For each provider, we selected connected US probes using the public probe API:

  • status=1, country_code=US, asn_v4 in the provider's ASN list
  • required the system-ipv4-works tag
  • excluded probes tagged datacentre, and anchors
  • preferred probes their hosts had tagged home

Coverage varied a lot. Comcast had over 200 eligible probes, AT&T over 160 and Verizon Fios over 140. Mediacom and Cable One had exactly one each. To keep the cost low, we used two probes per provider: 40 probes in total.

Measurements

We ran three one‑off measurements on those probes, keeping the cost to a minimum:

  1. DNS: an A query for the school's www hostname, with use_probe_resolver: true. This sends the query through whatever resolver the probe's network gives it, usually the ISP's, so it shows DNS‑level filtering like the VPN case.
  2. TLS (sslcert): a TLS handshake with the school's www hostname, with hostname set so the probe sends the hostname in the SNI field, and resolve_on_probe: true. This shows SNI‑based interference like the T‑Mobile case. A probe that gets the site's real certificate has not been interfered with.
  3. Control TLS: the same sslcert measurement against the control school's hostname on the same Cloudflare IPs. This separates "this domain is blocked" from "this probe can't reach Cloudflare at all".

We classified the results with a small script: DNS was "OK" if the answer contained only Cloudflare addresses, and "blocked" on NXDOMAIN or any other address. TLS was "OK" if a certificate came back from a Cloudflare address, and "blocked" on an alert or reset.

What it cost

The DNS measurement cost 20 credits per probe (800 for 40 probes). In total, including a re‑run described below, the investigation used a little under half of the 5,000 credits granted.

Results

We didn’t reproduce the block on any of the 44 usable probes across 21 providers.

Provider(s) Probes reporting Result
Comcast Xfinity, Charter Spectrum, Verizon Fios, Verizon wireless, Cox, Altice Optimum, Lumen / CenturyLink, Frontier, Starlink, Windstream Kinetic, Brightspeed, Astound, Breezeline, WOW!, Ziply Fiber, TDS 2 each DNS and TLS normal
T‑Mobile 8 (re-run) DNS and TLS normal
AT&T 1 of 2 Normal (the second probe failed every test, including the control)
Google Fiber 1 of 2 Normal (the second probe did not report)
Mediacom, Cable One 1 each DNS and TLS normal

Every DNS answer returned the real Cloudflare addresses, and every TLS handshake returned the site's real certificate. Of the 48 probes scheduled, 45 returned results, and 44 of those were usable.

Neither of the two T‑Mobile probes in the first run returned results. Since T‑Mobile was the one provider where we had seen the block, we re‑ran the three measurements on eight different T‑Mobile probes. All eight reached the site normally.

What we learned

  • TTL and IP ID are a quick way to spot injected packets. A "server reply" with TTL 255, IP ID 0 and no DF flag, arriving moments after a ClientHello, almost certainly came from something on the path, not from the real server.
  • Always measure a control hostname on the same infrastructure. It turned "it's slow / broken" into "this hostname, specifically, is being filtered", and it let us discard a broken probe instead of reporting a false block.
  • RIPE Atlas answers questions you can't answer from outside a network. We had no other affordable way to get DNS and TLS results from inside 21 US residential providers in a single afternoon. use_probe_resolver and sslcert with an explicit hostname mapped directly onto the two filtering methods we had seen in the captures.
  • Coverage is uneven, so plan around it. The big cable and telco networks are very well covered. Several regional providers have one to five probes, which limits what you can conclude about them.
  • Credits are the barrier for a first‑time user. We started with zero, and the request process was quick and friendly. For an organisation like ours, which doesn't run networks and isn't a RIPE NCC member, that grant is what made the measurement possible.

Thanks

Thank you to the RIPE Atlas team for the credit grant and the fast response, and to the probe hosts on US home networks whose probes did the actual work. We hope this shows a less common but very practical use of the platform: helping a school and its web provider work out whether "the website is down" is really a website problem.

Thank you to the school's team for their patience and tremendous help in gathering captures across various locations and conditions whenever we needed them.

0

About the author

Author image
Ciprian Lucanu Based in Bucharest, Romania

Software Development Team Lead at Interactive Schools Ltd

Comments 0