Again Firefox problem?!


2024-05-24
Out of nowhere I cannot access UVL with Firefox. It says:

"Forbidden

You don't have permission to access this resource.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.4.29 (Ubuntu) Server at uvlist.net Port 80"

All other browsers work. And its not a caching problem this time.

2024-05-25
Can confirm I have the same issue. Cleared cache etc, no joy.

2024-05-25
Well, what can I say. After having a have a good night's rest its working again.

2024-05-25
Thank you for the report, I haven't blocked Firefox explicitly, I was trying to block a scraper from completely blocking the website.
You might have noticed an exaggerated slowness yesterday.

I'm really thinking to take some extreme count measures to prevent "anyone" from monopolizing the site and make it unusable.

Please note that I don't care about the scrapers to get our data: Most of it is already public and we are not claiming any copyright, the only exception are your game reviews and anything personal we add here.

Here are some of the measures:

1 - Limit most content to logged-in users.

I've seen this done on other sites, is quite annoying for the casual users, and Google will love use even less, making UVL be less findable.


2- Host a sort of data-dump that anyone could fetch and use at will.

There is a LOT of data here, publishing a csv of all the games with most important information could stop scrapers for hogging the site? Maybe no.
Maybe some will.
Doing this will have to make editors discuss and accept this anyway.


3- Upgrade the server and/or use some anti-bot services

Upgrading will increase the limit when the site will feel slow, but bots that make as much requests as they can will cause issues anyway. (and will double the bill)
I'm also using the free services from Cloudflare, and when needed I can enable a "under attack mode" and any user will se the captcha that usually blocks the bots.
Bu I can't be always there checking for bad activities.
Paying users have this and many more services enabled automatically and could really make a difference, but this makes sense for sites that make money, and this site doesn't.


4- Improve code performance

Each page does a ot of database queries, and this is the most resource intensive activity.
A lot of page do some sort of caching (groups, companies, platforms) but not game searching as it needs live data to work.

The game info page also is not caching, fixing this could help a lot.
I'm working on it but it is the most complex page of all UVL, it fetches any kind of data and has a lot of embedded editor tools so it take time.



None of these points is the final solution, and some may not be acceptable for you.
I'd like to have your opinion on this.

2024-05-25

1 - Limit most content to logged-in users.

What do you mean by most content? If searching for games and browsing the game info pages and group info pages etc. are not usable for non registered users its not good. This would destroy the purpose of the site and it would be used much much less. And we all put work into it for others to enjoy. So no.

2- Host a sort of data-dump that anyone could fetch and use at will.

I do not know how long scrapers check a site manually (and maybe see that option). Or if its just an automatic process for them without doing any research. I have absolutely no insight into the world and work of scrapers.

3- Upgrade the server and/or use some anti-bot services

Unless someone wins the lottery, I think no. Having the typical Cloudfare message of having to make sure you are a human and not a bot would be acceptable and I see this on a lot of other sites recently, but if that costs money then likely no. And I doubt it would cost 5€-10€ per month extra which then I would say I could pay for it for a year and we see if its effective. That would be worthwhile. But it surely costs more.

4- Improve code performance


Improving code performance is always a good idea. But don't feel pressed into doing this fast.



None of these points is the final solution, and some may not be acceptable for you.
I'd like to have your opinion on this.


How bad is it with scrapers generally? And how long does the process take? Yes there are times when the site is slower than usual, but as long as it does not prevent me from browsing/editing completely I could live with it.

2024-05-27
About limiting content, Mobygames shows that first seven pages of games list for each platform to un-logged users, but I agree it's not a good idea to decrease games visibility.

Scrapers are just software instructed to navigate a site and fetch data. The people using them could be smart and check for easy to download data, but it depends of who and the reasons they are doing this, of which I have no idea.
But could be "nice" to have as a feature.
Handling the scrapers means logging into the server and watching the scrolling logs, looking for patterns.
It feels a lot like an 80's science fiction/hacker movie, it's fun at first, but it's not sustainable in the long run, and I really prefer doing something else.

I'm not going to ask any of you to give money, your time is more precious than that.
I suspect that the Cloudflare 200€ monthly tier would be the right option, but is way more than the actual hosting price.
While reading their documentation, I came across some nice idea for managing bots using the free tier. I'm going to give it a try.

I'll keep improving the performances, I want do badly to complete the game info page, because it really needs it.