123
-=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- (c) WidthPadding Industries 1987 0|209|0 -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=-
Socoder -> Site & Server -> Spiderbots Descend

Fri, 11 Oct 2024, 05:42
cyangames
Just a word of warning, bytespider and claudebot are mass crawling again, if you run a website, block them via .htaccess

-=-=-
Web / Game Dev, occasionally finishes off coding games also!
Fri, 11 Oct 2024, 05:42
Jayenkai
Logs for the past hour suggest...

GPTBot : 7,193
Misc (Actual users, maybe?) : 2165
BingBog : 705
Meta : 219
Bytedance (TikTok) : 166
Google : 143
Claude : 40

-=-=-
''Load, Next List!''
Mon, 14 Oct 2024, 05:26
HoboBen
Dunno what your opinion is on ChatGPT is ethically.

Figure, if we post here, it's public, and ChatGPT has as much right as any body to lurk the forum.

But it's gonna package up what we say and regurgitate it without attribution and destroy the environment at the same time. And it's slurping up more data than anybody else.

Do you block it?

-=-=-
blog | work | code | more code
Mon, 14 Oct 2024, 06:09
Jayenkai
I don't block it. But discuss whether I should.
To be fair, all the various AI's would have scraped the site half-a-dozen times each, by now, even long before it was a "thing" to avoid.

Bing's ALWAYS scraping stuff, and they're chucking all that into CoPilot, and Google Google the Google Google, so the only "real" option is to pretty much block EVERYTHING.

-=-=-
''Load, Next List!''
Mon, 14 Oct 2024, 09:50
Jayenkai
Stats for the past hour

GPTBot : 8,741
Bing : 688
Probably actual users : 269
Google : 294
Bytedance : 167
CCBot : 58
Meta : 34
DotBot : 9
Claude : 5

-=-=-
''Load, Next List!''
Fri, 08 Nov 2024, 10:25
Jayenkai
Stats for the past hour

Google : 3,124
dataforseo : 521
Bing : 453
Probably actual users : 184
Bytedance : 97
Meta : 59
AmazonBot : 26
Claude : 8
DotBot : 4
GPTBot : 1

.. Google..
And I double checked the IPs, and they're definitely Google.
Half of them are listed as GoogleBot, and the other half "Google Other"...!?!
So, I presume their AI crap.

I mean.. What am I supposed to do about that!? Block Google!?!?!

-=-=-
''Load, Next List!''
Fri, 08 Nov 2024, 15:08
cyangames
I think you can change the crawl rate within webmaster tools...well search console nowadays

-=-=-
Web / Game Dev, occasionally finishes off coding games also!
Fri, 08 Nov 2024, 16:25
Jayenkai
Oh yeah, I forgot about "you have to fart about online".
Quite why it can't pay attention to the settings in robots.txt, I've no idea.

-=-=-
''Load, Next List!''