101010.pl

Frontend DogmaWhat Is llms.txt, and Should You Care About It?, by <a href="https://mastodon.social/@ahrefs" class="u-url mention" rel="nofollow noopener" target="_blank">@ahrefs</a>:<a href="https://ahrefs.com/blog/what-is-llms-txt/" rel="nofollow noopener" translate="no" target="_blank">https://ahrefs.com/blog/what-is-llms-txt/</a><a href="https://mas.to/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mas.to/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> <a href="https://mas.to/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>

Frontend DogmaMeet LLMs.txt, a Proposed Standard for AI Website Content Crawling, by @searchengineland.bsky.social:<a href="https://searchengineland.com/llms-txt-proposed-standard-453676" rel="nofollow noopener" translate="no" target="_blank">https://searchengineland.com/llms-txt-proposed-standard-453676</a><a href="https://mas.to/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mas.to/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> <a href="https://mas.to/tags/scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraping</a> <a href="https://mas.to/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>

Frontend DogmaPoisoning Well, by <a href="https://front-end.social/@heydon" class="u-url mention" rel="nofollow noopener" target="_blank">@heydon</a>:<a href="https://heydonworks.com/article/poisoning-well/" rel="nofollow noopener" translate="no" target="_blank">https://heydonworks.com/article/poisoning-well/</a><a href="https://mas.to/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mas.to/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> <a href="https://mas.to/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://mas.to/tags/content" class="mention hashtag" rel="nofollow noopener" target="_blank">#content</a>

Frontend DogmaPlease Stop Externalizing Your Costs Directly Into My Face, by <a href="https://cmpwn.com/@sir" class="u-url mention" rel="nofollow noopener" target="_blank">@sir</a>:<a href="https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html" rel="nofollow noopener" translate="no" target="_blank">https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html</a><a href="https://mas.to/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mas.to/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> <a href="https://mas.to/tags/traffic" class="mention hashtag" rel="nofollow noopener" target="_blank">#traffic</a> <a href="https://mas.to/tags/economics" class="mention hashtag" rel="nofollow noopener" target="_blank">#economics</a>

DaLetra FrançaisDécouvrez les paroles de la chanson “Crawling” de Linkin Park <a href="https://flipboard.social/tags/LinkinPark" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinkinPark</a> <a href="https://flipboard.social/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://daletra.art/linkin-park/paroles/crawling.html" rel="nofollow noopener" translate="no" target="_blank">https://daletra.art/linkin-park/paroles/crawling.html</a>

Tobias KöngeterWho is familiar with <a href="https://sueden.social/tags/website" class="mention hashtag" rel="nofollow noopener" target="_blank">#website</a> <a href="https://sueden.social/tags/tracking" class="mention hashtag" rel="nofollow noopener" target="_blank">#tracking</a>? My website www.whatcolor.is currently has around 85,000 visitors per day, with most of them accessing all resources such as images, but also JSON documents and InDesign documents. That looks a lot like <a href="https://sueden.social/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> to me. I use <a href="https://sueden.social/tags/Matomo" class="mention hashtag" rel="nofollow noopener" target="_blank">#Matomo</a>, which has been configured in accordance with German data protection regulations, so I don't have a lot of information about these visitors. How can I find out more about these visitors? What could they be?<a href="https://sueden.social/tags/helpwanted" class="mention hashtag" rel="nofollow noopener" target="_blank">#helpwanted</a>

DaLetraConfira a letra da música “Crawling” de Linkin Park <a href="https://flipboard.social/tags/LinkinPark" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinkinPark</a> <a href="https://flipboard.social/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://daletra.com.br/linkin-park/letra/crawling.html" rel="nofollow noopener" translate="no" target="_blank">https://daletra.com.br/linkin-park/letra/crawling.html</a>

Max ResingIt looks like LLM-producing companies that are massively <a href="https://infosec.exchange/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> the <a href="https://infosec.exchange/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a> require the owners of a website to take action to opt out. Albeit I am not intrinsically against <a href="https://infosec.exchange/tags/generativeai" class="mention hashtag" rel="nofollow noopener" target="_blank">#generativeai</a> and the acquisition of <a href="https://infosec.exchange/tags/opendata" class="mention hashtag" rel="nofollow noopener" target="_blank">#opendata</a>, reading about hundreds of dollars of rising <a href="https://infosec.exchange/tags/cloud" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloud</a> costs for hobby projects is quite concerning. How is it accepted that hypergiants skyrocket the costs of tightly budgeted projects through massive spikes in egress traffic and increased processing requirements? Projects that run on a shoestring budget and are operated by volunteers who dedicated hundreds of hours without any reward other than believing in their mission?I am mostly concerned about the default of opting out. Are the owners of those projects required to take action? Seriously? As an <a href="https://infosec.exchange/tags/operator" class="mention hashtag" rel="nofollow noopener" target="_blank">#operator</a>, it would be my responsibility to methodically work myself through the crawling documentation of the hundreds of <a href="https://infosec.exchange/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a> <a href="https://infosec.exchange/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a> <a href="https://infosec.exchange/tags/crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawlers</a>? I am the one responsible for configuring a unique crawling specification in my robots.txt because hypergiants make it immanently hard to have generic <a href="https://infosec.exchange/tags/opt" class="mention hashtag" rel="nofollow noopener" target="_blank">#opt</a>-out configurations that tackle LLM projects specifically?I reject to accept that this is our new norm. A norm in which hypergiants are not only methodically exploiting the work of thousands of individuals for their own benefit and without returning a penny. But also a norm, in which the resource owner is required to prevent these crawlers from skyrocketing one's own operational costs?We require a new <a href="https://infosec.exchange/tags/opt" class="mention hashtag" rel="nofollow noopener" target="_blank">#opt</a>-in. Often, public and open projects are keen to share their data. They just don't like the idea of carrying the unpredictable, multitudinous financial burden of sharing the data without notice from said crawlers. Even <a href="https://infosec.exchange/tags/CommonCrawl" class="mention hashtag" rel="nofollow noopener" target="_blank">#CommonCrawl</a> has safe-fail mechanisms to reduce the burden on website owners. Why are LLM crawlers above the guidelines of good <a href="https://infosec.exchange/tags/Internet" class="mention hashtag" rel="nofollow noopener" target="_blank">#Internet</a> citizenship?To counter the most common argument already: Yes, you can deny-by-default in your robots.txt, but that excludes any non-mainstream browser, too.Some concerning <a href="https://infosec.exchange/tags/news" class="mention hashtag" rel="nofollow noopener" target="_blank">#news</a> articles on the topic:<ul><li><a href="https://archive.is/nQ6Gk" rel="nofollow noopener" translate="no" target="_blank">https://archive.is/nQ6Gk</a></li><li><a href="https://archive.is/CRwVs" rel="nofollow noopener" translate="no" target="_blank">https://archive.is/CRwVs</a></li></ul><a href="https://infosec.exchange/tags/webcrawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#webcrawling</a> <a href="https://infosec.exchange/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://infosec.exchange/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a> <a href="https://infosec.exchange/tags/opensource" class="mention hashtag" rel="nofollow noopener" target="_blank">#opensource</a>

Kevin Karhan :verified:<a href="https://mastodon.online/@zdl" class="u-url mention" rel="nofollow noopener" target="_blank">@zdl</a> <a href="https://hachyderm.io/@evacide" class="u-url mention" rel="nofollow noopener" target="_blank">@evacide</a> that any the fact that <a href="https://mastodon.world/@signalapp" class="u-url mention" rel="nofollow noopener" target="_blank">@signalapp</a> is incorportated in the <a href="https://infosec.space/tags/USA" class="mention hashtag" rel="nofollow noopener" target="_blank">#USA</a>, making them susceptible to <a href="https://infosec.space/tags/GDPR" class="mention hashtag" rel="nofollow noopener" target="_blank">#GDPR</a> & <a href="https://infosec.space/tags/BDSG" class="mention hashtag" rel="nofollow noopener" target="_blank">#BDSG</a>-incompatible <a href="https://infosec.space/tags/cyberfacist" class="mention hashtag" rel="nofollow noopener" target="_blank">#cyberfacist</a> bs like <a href="https://infosec.space/tags/CloudAct" class="mention hashtag" rel="nofollow noopener" target="_blank">#CloudAct</a>.<ul><li>If <a href="https://infosec.space/tags/Signal" class="mention hashtag" rel="nofollow noopener" target="_blank">#Signal</a> cared, they'd completely <a href="https://infosec.space/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenSource</a> <a href="https://infosec.space/tags/backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#backend</a> and <a href="https://infosec.space/tags/frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#frontend</a> as well as <a href="https://infosec.space/tags/decentralize" class="mention hashtag" rel="nofollow noopener" target="_blank">#decentralize</a> and refuse to collect any <a href="https://infosec.space/tags/PII" class="mention hashtag" rel="nofollow noopener" target="_blank">#PII</a> (like <a href="https://infosec.space/tags/PhoneNumers" class="mention hashtag" rel="nofollow noopener" target="_blank">#PhoneNumers</a>) at all!</li></ul>Remember: <a href="https://infosec.space/tags/KYC" class="mention hashtag" rel="nofollow noopener" target="_blank">#KYC</a> IS THE ILLICIT ACTIVITY when it comes to <a href="https://infosec.space/tags/Communication" class="mention hashtag" rel="nofollow noopener" target="_blank">#Communication</a>!<ul><li>To me Signal has a stench like <a href="https://infosec.space/tags/CryptoAG" class="mention hashtag" rel="nofollow noopener" target="_blank">#CryptoAG</a> (aka. <a href="https://infosec.space/tags/MINERVA" class="mention hashtag" rel="nofollow noopener" target="_blank">#MINERVA</a> / <a href="https://infosec.space/tags/RUBIKON" class="mention hashtag" rel="nofollow noopener" target="_blank">#RUBIKON</a>), <a href="https://infosec.space/tags/EncroChat" class="mention hashtag" rel="nofollow noopener" target="_blank">#EncroChat</a> and espechally <a href="https://infosec.space/tags/AN%C3%98M" class="mention hashtag" rel="nofollow noopener" target="_blank">#ANØM</a> (aka. <a href="https://infosec.space/tags/OperationIronside" class="mention hashtag" rel="nofollow noopener" target="_blank">#OperationIronside</a> / <a href="https://infosec.space/tags/OperationTr%C3%B8janShield" class="mention hashtag" rel="nofollow noopener" target="_blank">#OperationTrøjanShield</a>)...</li></ul>Compare that to <a href="https://monocles.social/@monocles" class="u-url mention" rel="nofollow noopener" target="_blank">@monocles</a> / <a href="https://infosec.space/tags/monoclesChat" class="mention hashtag" rel="nofollow noopener" target="_blank">#monoclesChat</a> which don't demand any PII or KYC and allow people to pay for their services with <a href="https://infosec.space/tags/Monero" class="mention hashtag" rel="nofollow noopener" target="_blank">#Monero</a> and <a href="https://infosec.space/tags/CashByMail" class="mention hashtag" rel="nofollow noopener" target="_blank">#CashByMail</a> besides <a href="https://infosec.space/tags/SEPA" class="mention hashtag" rel="nofollow noopener" target="_blank">#SEPA</a> <a href="https://infosec.space/tags/WireTransfer" class="mention hashtag" rel="nofollow noopener" target="_blank">#WireTransfer</a>, <a href="https://infosec.space/tags/Stripe" class="mention hashtag" rel="nofollow noopener" target="_blank">#Stripe</a> & <a href="https://infosec.space/tags/PayPal" class="mention hashtag" rel="nofollow noopener" target="_blank">#PayPal</a> whilst supporting both decentralization (<a href="https://infosec.space/tags/XMPP" class="mention hashtag" rel="nofollow noopener" target="_blank">#XMPP</a> is not a <a href="https://infosec.space/tags/SingleVendor" class="mention hashtag" rel="nofollow noopener" target="_blank">#SingleVendor</a> / <a href="https://infosec.space/tags/SingleProvider" class="mention hashtag" rel="nofollow noopener" target="_blank">#SingleProvider</a> solution!), implementing real <a href="https://infosec.space/tags/SelfCustody" class="mention hashtag" rel="nofollow noopener" target="_blank">#SelfCustody</a> (<a href="https://infosec.space/tags/OMEMO" class="mention hashtag" rel="nofollow noopener" target="_blank">#OMEMO</a>, <a href="https://infosec.space/tags/OTR" class="mention hashtag" rel="nofollow noopener" target="_blank">#OTR</a> & <a href="https://infosec.space/tags/PGP" class="mention hashtag" rel="nofollow noopener" target="_blank">#PGP</a> is supported out of the box) for all the keys, and proper <a href="https://infosec.space/tags/Anonymitiy" class="mention hashtag" rel="nofollow noopener" target="_blank">#Anonymitiy</a> (using <a href="https://mastodon.social/@torproject" class="u-url mention" rel="nofollow noopener" target="_blank">@torproject</a> / <a href="https://infosec.space/tags/Tor" class="mention hashtag" rel="nofollow noopener" target="_blank">#Tor</a> & <a href="https://social.librem.one/@guardianproject" class="u-url mention" rel="nofollow noopener" target="_blank">@guardianproject</a> <a href="https://infosec.space/tags/Orbot" class="mention hashtag" rel="nofollow noopener" target="_blank">#Orbot</a> for <a href="https://infosec.space/tags/privacy" class="mention hashtag" rel="nofollow noopener" target="_blank">#privacy</a>), so in case they ever get a duely sumitted warrant by a court they'd have to comply with, they'll most likely have no data whatsoever on clients that could allow identification.<ul><li>And that is a good thing, because whilst very unlikely, one cannot exclude the non-zero chance of i.e. <a href="https://infosec.space/tags/MLAT" class="mention hashtag" rel="nofollow noopener" target="_blank">#MLAT</a>|s being filed with knowingly false information by 3rd countries.</li></ul>Also having no PII is a matter of reducing <a href="https://infosec.space/tags/liability" class="mention hashtag" rel="nofollow noopener" target="_blank">#liability</a> in the sense of <a href="https://infosec.space/tags/DataProtection" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataProtection</a>: All data requested and by <a href="https://infosec.space/tags/monocles" class="mention hashtag" rel="nofollow noopener" target="_blank">#monocles</a> is the bare minimum mandated for <a href="https://infosec.space/tags/accounting" class="mention hashtag" rel="nofollow noopener" target="_blank">#accounting</a> (i.e. only linking a payment like a <a href="https://infosec.space/tags/TxID" class="mention hashtag" rel="nofollow noopener" target="_blank">#TxID</a> / Transaction-ID to an account and then adding up validity/activation period).<ul><li>And since running a <a href="https://infosec.space/tags/Service" class="mention hashtag" rel="nofollow noopener" target="_blank">#Service</a> costs money, the low <a href="https://infosec.space/tags/subscription" class="mention hashtag" rel="nofollow noopener" target="_blank">#subscription</a> to their <a href="https://infosec.space/tags/Services" class="mention hashtag" rel="nofollow noopener" target="_blank">#Services</a> makes them independent from <a href="https://infosec.space/tags/ads" class="mention hashtag" rel="nofollow noopener" target="_blank">#ads</a>, <a href="https://infosec.space/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> / <a href="https://infosec.space/tags/espionage" class="mention hashtag" rel="nofollow noopener" target="_blank">#espionage</a> against <a href="https://infosec.space/tags/customers" class="mention hashtag" rel="nofollow noopener" target="_blank">#customers</a> and depending on <a href="https://infosec.space/tags/grants" class="mention hashtag" rel="nofollow noopener" target="_blank">#grants</a> and <a href="https://infosec.space/tags/donations" class="mention hashtag" rel="nofollow noopener" target="_blank">#donations</a> to keep the lights on, making it a <a href="https://infosec.space/tags/sustainable" class="mention hashtag" rel="nofollow noopener" target="_blank">#sustainable</a> <a href="https://infosec.space/tags/business" class="mention hashtag" rel="nofollow noopener" target="_blank">#business</a>...</li></ul>

StarbeamrainbowlabsImplementation progress: got the crawler written & integrated with the indexes + thumbnailer (+`.gitignore` support) but not tested. The vector search persistence system is truly shocking..... I think I need to back it in an `njodb` instance.Next up:- testing the crawler - integrating with the rest of the app - HTTP API - starting working on the web interface(see also <<a href="https://www.npmjs.com/package/njodb" rel="nofollow noopener" translate="no" target="_blank">https://www.npmjs.com/package/njodb</a>>)<a href="https://fediscience.org/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenSource</a> <a href="https://fediscience.org/tags/Eventually" class="mention hashtag" rel="nofollow noopener" target="_blank">#Eventually</a> <a href="https://fediscience.org/tags/OnceIveFinishedAMinimumWorkingVersion" class="mention hashtag" rel="nofollow noopener" target="_blank">#OnceIveFinishedAMinimumWorkingVersion</a> <a href="https://fediscience.org/tags/Javascript" class="mention hashtag" rel="nofollow noopener" target="_blank">#Javascript</a> <a href="https://fediscience.org/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://fediscience.org/tags/Programming" class="mention hashtag" rel="nofollow noopener" target="_blank">#Programming</a> <a href="https://fediscience.org/tags/Chatter" class="mention hashtag" rel="nofollow noopener" target="_blank">#Chatter</a> <a href="https://fediscience.org/tags/ProgressUpdate" class="mention hashtag" rel="nofollow noopener" target="_blank">#ProgressUpdate</a> <a href="https://fediscience.org/tags/microblog" class="mention hashtag" rel="nofollow noopener" target="_blank">#microblog</a> <a href="https://fediscience.org/tags/IsThereATagForPhotoManagement" class="mention hashtag" rel="nofollow noopener" target="_blank">#IsThereATagForPhotoManagement</a> <a href="https://fediscience.org/tags/PhotoManagement" class="mention hashtag" rel="nofollow noopener" target="_blank">#PhotoManagement</a> <a href="https://fediscience.org/tags/WillTagWithPhotographyOnceTheresSomethingToSee" class="mention hashtag" rel="nofollow noopener" target="_blank">#WillTagWithPhotographyOnceTheresSomethingToSee</a>

DaLetra FrançaisVoir les paroles de la chanson “Crawling” de Linkin Park <a href="https://flipboard.social/tags/LinkinPark" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinkinPark</a> <a href="https://flipboard.social/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://daletra.art/linkin-park/paroles/crawling.html" rel="nofollow noopener" translate="no" target="_blank">https://daletra.art/linkin-park/paroles/crawling.html</a>

8 Ball SystemHey little <a href="https://8ballsystem.gay/tags/tip" class="mention hashtag" rel="nofollow noopener" target="_blank">#tip</a> for you all, if you want to <a href="https://8ballsystem.gay/tags/block" class="mention hashtag" rel="nofollow noopener" target="_blank">#block</a> <a href="https://8ballsystem.gay/tags/openAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#openAI</a> from <a href="https://8ballsystem.gay/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> your <a href="https://8ballsystem.gay/tags/server" class="mention hashtag" rel="nofollow noopener" target="_blank">#server</a> or <a href="https://8ballsystem.gay/tags/website" class="mention hashtag" rel="nofollow noopener" target="_blank">#website</a>, they public a list of IP addresses that they use for crawling.<a href="https://platform.openai.com/docs/bots" rel="nofollow noopener" translate="no" target="_blank">https://platform.openai.com/docs/bots</a>I plan on deploying firewall blocks on all these IP addresses later!If you know how <a href="https://8ballsystem.gay/tags/meta" class="mention hashtag" rel="nofollow noopener" target="_blank">#meta</a> and <a href="https://8ballsystem.gay/tags/google" class="mention hashtag" rel="nofollow noopener" target="_blank">#google</a>'s IP's they use for crawling please drop them here!! Keep yourself and your stuff safe!

DaLetra EnglishLyrics for the song “Crawling” by Linkin Park <a href="https://flipboard.social/tags/LinkinPark" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinkinPark</a> <a href="https://flipboard.social/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://daletra.com/linkin-park/lyrics/crawling.html" rel="nofollow noopener" translate="no" target="_blank">https://daletra.com/linkin-park/lyrics/crawling.html</a>

panigrc<a href="https://phpc.social/@otsch" class="u-url mention" rel="nofollow noopener" target="_blank">@otsch</a> Thanks for your contribution! 💙 The most difficult part about crawling is <a href="https://mastodon.social/tags/cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudflare</a> 😡 the only way I found to bypass it, but it doesn't work 100% of the time is <a href="https://mastodon.social/tags/flaresolverr" class="mention hashtag" rel="nofollow noopener" target="_blank">#flaresolverr</a> <a href="https://github.com/FlareSolverr/FlareSolverr" rel="nofollow noopener" translate="no" target="_blank">https://github.com/FlareSolverr/FlareSolverr</a>Since you started supporting structured data, here is an idea for you. Start supporting <a href="https://mastodon.social/tags/microformats" class="mention hashtag" rel="nofollow noopener" target="_blank">#microformats</a> <a href="https://microformats.org/" rel="nofollow noopener" translate="no" target="_blank">https://microformats.org/</a><a href="https://mastodon.social/tags/php" class="mention hashtag" rel="nofollow noopener" target="_blank">#php</a> <a href="https://mastodon.social/tags/webscraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#webscraping</a> <a href="https://mastodon.social/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a>

DaLetra EnglishCheck out the lyrics for the song “Crawling” by Linkin Park <a href="https://flipboard.social/tags/LinkinPark" class="mention hashtag" rel="nofollow noopener" target="_blank">#LinkinPark</a> <a href="https://flipboard.social/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://daletra.com/linkin-park/lyrics/crawling.html" rel="nofollow noopener" translate="no" target="_blank">https://daletra.com/linkin-park/lyrics/crawling.html</a>

Erik JonkerInteresting fact with regard to Meta External Agent (An AI crawler), as of August 2024, only about 2% of popular websites were blocking the Meta External Agent, compared to 25% blocking OpenAI's GPTBot. Personally i think trying to block AI webcrawlers is useless. If a human is allowed to read a website why shouldn't we let a machine do this. There are many legitimate uses for machines reading websites, not only AI. <a href="https://fortune.com/2024/08/20/meta-external-agent-new-web-crawler-bot-scrape-data-train-ai-models-llama/" rel="nofollow noopener" translate="no" target="_blank">https://fortune.com/2024/08/20/meta-external-agent-new-web-crawler-bot-scrape-data-train-ai-models-llama/</a> <a href="https://mastodon.social/tags/metaexternalagent" class="mention hashtag" rel="nofollow noopener" target="_blank">#metaexternalagent</a> <a href="https://mastodon.social/tags/meta" class="mention hashtag" rel="nofollow noopener" target="_blank">#meta</a> <a href="https://mastodon.social/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a>

Josef 'Jeff' SipekHoly crap! <a href="https://mastodon.radio/tags/Huawei" class="mention hashtag" rel="nofollow noopener" target="_blank">#Huawei</a> is *very* aggressively <a href="https://mastodon.radio/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawling</a> my <a href="https://mastodon.radio/tags/webserver" class="mention hashtag" rel="nofollow noopener" target="_blank">#webserver</a>. As in, 6+ requests/sec for many hours coming from quite a few IPs. Here are the subnets I blocked which kills most of the traffic so far:49.0.200.0/21 94.74.80.0/20 101.44.160.0/20 111.119.192.0/20 114.119.172.0/22 114.119.176.0/20 119.8.160.0/19 119.13.96.0/20 124.243.128.0/18 159.138.96.0/20 166.108.192.0/20 166.108.224.0/20 190.92.192.0/19The user agent string is the typical "every browser in existence". <a href="https://mastodon.radio/tags/webcrawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#webcrawler</a>

Lyrical Garfield 🎶 :garfield:<a href="https://masto.ai/tags/Garfield" class="mention hashtag" rel="nofollow noopener" target="_blank">#Garfield</a> <a href="https://masto.ai/tags/music" class="mention hashtag" rel="nofollow noopener" target="_blank">#music</a> <a href="https://masto.ai/tags/Crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawling</a> <a href="https://masto.ai/tags/bot" class="mention hashtag" rel="nofollow noopener" target="_blank">#bot</a> <a href="https://masto.ai/tags/lyrics" class="mention hashtag" rel="nofollow noopener" target="_blank">#lyrics</a>

Recent searches

Search options

Administered by:

Server stats:

#crawling