Fail2ban and firewall problem [SOLVED]

[lmarcos]

My server has recently seen an abnormal increase in input and output network traffic (and increasing CPU usage and rendering the sites unusable) and when I was reviewing settings, I've seen that there's only one banned IP in the firewall and nothing on fail2ban, when last time I checked there was a bunch.

But now, Active jails and amount of banned IP addresses: kh-phpmyadmin (0x), sshd (0x)
- No entries available -

Any ideas what can I do for fail2ban to start banning ips again?

[Daniel]

Hello,

could you determine at first what type of traffic increased in general?
Do you recognize any hints at the logs?

[lmarcos]

Traffic in general, but unrelated to the main content, since it comes from several different countries (from China to Russia to Sweden) and they're also requesting pages that do not exist. I've shown the log, or part of it, to an acquaintance and he told me somebody might be scrapping the site to train an AI.

[Daniel]

Hello,

is there a potential similarity, like user-agent?
"ClaudeBot" e.g. is currently often around with aggressive crawls.

You could limit them or other "unwanted" user-agents using "rewrite" rules at the virtual host of apache. Or if its too much, implement a countryfilter for 80/443 tcp @iptables/ipset.

[lmarcos]

Claudebot, Bytespider and facebookexternalhit were the culprits. Thanks for pointing me in the right direction. Still, I don't get why these agressive bots are not jailed by fail2ban. But that is not keyhelp related.

[Daniel]

Hello,

there are no rules to turn down legitimate bots that are crawling websites. They are basically seen as normal usertraffic.
Of course its possible to block them with fail2ban, but its very inefficent. Best options are rules according to robots.txt or if they do not comply with, block the user-agents with re-write rules or throttle them with a rate-limit on application layer. WIth nginx done through "limit_req" analogous to apache mod_qos. mod_evasive is not useful since there is no user-agent matching option.

[Jolinar]

I don't get why these agressive bots are not jailed by fail2ba

You could customize the file /etc/fail2ban/filter.d/apache-badbots.conf to suit your needs.

[Ralph]

Claudebot, Bytespider and facebookexternalhit were the culprits. Thanks for pointing me in the right direction. Still, I don't get why these agressive bots are not jailed by fail2ban. But that is not keyhelp related.

You should use a custom filter and expand it accordingly ... will not be overwritten (F2B updates).
e.g.

Code: Select all

nano /etc/fail2ban/filter.d/my-badbots.conf
# add:

# Fail2Ban configuration file
#
# Regexp to catch specific spambots

[Definition]

failregex = ^<HOST>.*ahrefs\.com.*
            ^<HOST>.*PetalBot.*
            ^<HOST>.*ClaudeBot.*
            ^<HOST>.*claudebot.*
            ^<HOST>.*Bytespider.*
            ^<HOST>.*bytedance\.com.*
            ^<HOST>.*BLEXBot.*
            ^<HOST>.*facebookexternalhit.*

ignoreregex =

Code: Select all

nano /etc/fail2ban/jail.d/my-badbots.local
# add:

[my-badbots]

enabled  = true
port     = http,https
filter   = my-badbots
logpath = /home/users/*/logs/*/access.log
maxretry = 1
findtime  = 1800
bantime  = 48h

[lmarcos]

Thanks to all. I didn't see the last two replies until now. What I did after Daniel pointed me to Claudebot was to ban them in robots.txt. At least that's what I think I did.