Reply to comment:

Sander
Ah, found it. It seems that quite some sites have problems with smaller MTUs on IPv4. My test boxes are (intentionally) behind a link with an MTU of 1280 to check that pMTU is handled correctly. Apparently they don't :( I now added MSS clamping, and that seems to help. But it is disappointing that so many websites are sending packets with DF set and then don't handle fragmentation properly.