hylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 1 day agoBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersjlai.luimagemessage-square57linkfedilinkarrow-up1480arrow-down16file-text
arrow-up1474arrow-down1imageBased on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlersjlai.luhylobates@jlai.lu to Selfhosted@lemmy.worldEnglish · 1 day agomessage-square57linkfedilinkfile-text
minus-squareAHemlocksLie@lemmy.ziplinkfedilinkEnglisharrow-up8·11 hours agoPretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?
minus-squareSv443@sh.itjust.workslinkfedilinkEnglisharrow-up4·8 hours agoLike a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys
Pretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?
Like a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys