minus-squareAHemlocksLie@lemmy.ziptoSelfhosted@lemmy.world•Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlerslinkfedilinkEnglisharrow-up17arrow-down1·1 day agoPretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much? linkfedilink
Pretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?