www.newlisp.org site server problems

[Lutz]

Since two to three days used bandwidth on http://www.newlisp.org">www.newlisp.org (newlisp.nfshost.com) has grown 10 times, this allthough page hits, downloads and other statistics in the logs seem to be the same. I am already using the following .htaccess on Apache for a few months:

Code Select Expand

RewriteCond %{HTTP_USER_AGENT} MJ12bot
order allow,deny
deny from 220.181.157.164
deny from 220.181.157.165
deny from 220.181.157.166
deny from 220.181.157.196
deny from 220.181.157.197
deny from 220.181.55.55
deny from 218.30.117.160
allow from all


.. as I have multiple downloads requests daily for all of the IPs above everyday for all downloads on the site. This has brought downloads and pages statistics down, but in the last 3 days bandwidth suddenly is 10 fold and I don't know why. Download and page hit numbers stay low, but bandwidth is going through the roof as if .htaccess is not working correctly anymore? Does anybody have suggestions? Is there a more efficient way to suppress access from certain IPs ?

[TedWalther]

Lutz, are you operating a virtual server?  Perhaps the bandwidth usage isn't coming in over the web, but is a ping flood, or traffic going to some other internet port on your server?  I'm willing to log in and look around, contact me in private with details.



How are you measuring the bandwidth?  Google analytics?  Webalyzer?

[rrq]

how about iptables rules at the OS level? Well, assuming linux or mac, and that you "own" the host.

[Lutz]

This is not my own host but a site at the hosting company https://www.nearlyfreespeech.net">https://www.nearlyfreespeech.net . I like them because they charge purely by bandwidth usage. Before opening a paid support ticket with them, I am asking for help here at the forum, as there are so many system admin people among newLISP users.



It's a FreeBSD 9.2-RELEASE-p3 site running virtual servers and Apache - don't know which version. Tools installed are limited for users of virtual servers. I don't think, I have permissions to filter at the OS level. The .htaccess files where working fine, but suddenly bandwidth usage is 10-12 times what it was before with similar content of log files, generated with software by http://www.awstats.org">http://www.awstats.org, configuration of which cannot be done by me.

[TedWalther]

Thank you.  Does awstats have an option so you can see which IP and which file is using the most bandwidth?  Google analytics does.



If it is $30 or less, I'd open a support ticket.



I remember I colocated a server years ago with a large colo company.  I was watching my logs, at the web server level, and also at the packet level.  Suddenly the colo said bandwidth usage jumped, but my own logs said it didn't.  I protested, and they held my server hostage.  Ever since, I try to host my own servers.

[rickyboy]

Before opening a paid support ticket with them, I am asking for help here at the forum, ...

Please check all the "Support Options" (https://members.nearlyfreespeech.net/support/request">https://members.nearlyfreespeech.net/support/request), before you pay the $30 -- unless you are in a hurry.  One of the options is to post on the NFSN Member Forum.  Good luck!

[rickyboy]

I remember I colocated a server years ago with a large colo company.  I was watching my logs, at the web server level, and also at the packet level.  Suddenly the colo said bandwidth usage jumped, but my own logs said it didn't.  I protested, and they held my server hostage.  Ever since, I try to host my own servers.

I agree with that.  I like NFSN, but a $5/mo Digital Ocean box looks really good. :)

[rrq]

I haven't used them myself, but there are bandwidth limiting and throttling modules for apache; maybe something like the "mod_cband" would serve the purpose. I suppose you need something that would limit the amount of data a single user can get in a short time period, ideally without affecting a "normal" user.

[Lutz]

Thanks to everybody for all the good advice. I posted more details about the case on the nearlyfreespeech.net discussion forum.



All the bandwidth increase is reported for CGI pages, which is strange because the number of hits on these pages is the same as before and scripts haven't changed. Hopefully I will get some answers soon. Today the bandwidth increase is 20-fold, yesterday about 10-fold !?!?

[TedWalther]

Has the awstats software been updated?  How does it tell the size of a CGI transaction?  Is your CGI script accurately reporting the size of the data it is returning, in the MIME header?

[Lutz]

Looks like the problem is solved :)



One of the nfs guys on the nfshost.com user forum gave me a command to extract file sizes from the access log. That helped to identify a big text file in the newLISP wiki's pages directory and from more and more sources over time. It had nothing to do with the IPs locked out by .htaccess, which is still working well.



Thanks again to everybody for helping to shed light on this.