Developer forum

Forum » CMS - Standard features » Users Banned, but not in _BannedIPs.txt

Users Banned, but not in _BannedIPs.txt

Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi,

 

We have 2 customers (9.14.10 and 9.16.7) that have both reported a very odd experience wtih IP Banner.

 

  • They claim to be banned
  • Their IP is NOT in GeneralLog nor in _BannedIPs.txt
  • But whenever we completely clear the _BannedIPs.txt file, they are unblocked

 

The only similar thing between the two is that some IPv6 were banned, but they don't look right, storing only the first 4 digits of the IP.

 

Any thoughts on what could be causing this? If it's best, I can reach out through Care so that I can provide the customer's names and the banned IPs

 

Best Regards,

Nuno Aguiar


Replies

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Could they be use some sort of proxy on their net. x-forwarded-for header vs. the IP?

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Could they be use some sort of proxy on their net. x-forwarded-for header vs. the IP?

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Havin an IPv6 in their banned IPs is very odd as there should not be ipv6 on their webserver. That could indicate that they internally run ipv6, and that is forwarded in a header DW understands...

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

I'll see if I can meet with the customer and through screen sharing try to capture more information. If you have any other ideas on how I could even capture this, I'd appreciate it.

 

In regards to the IPv6 (which may or may not be related), here's what we're seeing

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Cloudflare configuration issue? ipBanner is looking at x-forwarded-for headers - and they could contain something that DW misunderstand...

I guess this kind of request is not coming from inside their organisation

/products?ProductType=2%27%29+AND+1%3D1+UNION+ALL+SELECT+1%2CNULL%2C%27%3Cscript%3Ealert%28%5C%22XSS%5C%22%29%3C%2Fscript%3E%27%2Ctable_name+FROM+information_schema.tables+WHERE+2%3E1--%2F%2A%2A%2F%3B+EXEC+xp_cmdshell%28%27cat+..%2F..%2F..%2Fetc%2Fpasswd%27%29%23

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

I haven't been able to meet with both customers to see what they are getting. Cloudflare is a possibility, but we'll have to see.

 

Clearly what you highlighted is not coming from within their organization (a clear SQL Injection and XSS attempt). I guess I wondered if there could be something in DW that once if finds a banned IP of type IPv6, the code acts up and does not validate them accurately (since our customers stop getting banned once we completetly delete everything in _BannedIps.txt despite not seeing their IP there - at least that's been how our Support Desk people reported it on both occasions). I'll get to the bottom of the mistery :) 

 

Thank you for your insights,

Nuno Aguiar

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

At least we figured why IP v6 are getting logged badly (it's coming up more and more)

        public static string GetClientIpAddress()
        {
            try
            {
                string userHostAddress = Context.Current.Request.UserHostAddress;
                // Could use TryParse instead, but I wanted to catch all exceptions
                IPAddress.Parse(userHostAddress);
                string forwardIp = Context.Current.Request?.Headers?["X-FORWARDED-FOR"];
                if (string.IsNullOrEmpty(forwardIp))
                {
                    forwardIp = Context.Current.Request?.Headers?["X-CLIENT-IP"];
                }
                if (string.IsNullOrEmpty(forwardIp))
                {
                    return userHostAddress;
                }
                // Get a list of public ip addresses in the X-FORWARDED-FOR and X-CLIENT-IP headers
                var publicForwardingIps = new List<string>();
                var array = forwardIp.Split(',');
                for (int i = 0, loopTo = array.Length - 1; i <= loopTo; i++)
                {
                    string ipString = array[i];
                    if (ipString.Contains(":"))
                    {
                        ipString = ipString.Split(':')[0];
                    }
                    if (!IsPrivateIpAddress(ipString) && !publicForwardingIps.Contains(ipString))
                    {
                        publicForwardingIps.Add(ipString);
                    }
                }
                // If we found any, return the last one, otherwise return the user host address
                return publicForwardingIps.Any() ? publicForwardingIps.First() : userHostAddress;
            }
            catch
            {
                return null;
            }
        }

 

If this could be changed, would at least help support them in future versions.

 

I found a few way to go about it. Here's a suggestion without a lot of refactoring

        public static string GetClientIpAddress()
        {
            try
            {
                string userHostAddress = Context.Current.Request.UserHostAddress;
                // Could use TryParse instead, but I wanted to catch all exceptions
                IPAddress.Parse(userHostAddress);
                string forwardIp = Context.Current.Request?.Headers?["X-FORWARDED-FOR"];
                if (string.IsNullOrEmpty(forwardIp))
                {
                    forwardIp = Context.Current.Request?.Headers?["X-CLIENT-IP"];
                }
                if (string.IsNullOrEmpty(forwardIp))
                {
                    return userHostAddress;
                }
                // Get a list of public ip addresses in the X-FORWARDED-FOR and X-CLIENT-IP headers
                var publicForwardingIps = new List<string>();
                var array = forwardIp.Split(',');
                for (int i = 0, loopTo = array.Length - 1; i <= loopTo; i++)
                {
                    IPAddress address;
                    if (IPAddress.TryParse(array[i], out address))
                    {
                        string ipString = array[i];
                        if (ipString.Contains(":") && address == System.Net.Sockets.AddressFamily.InterNetwork)
                        {
                            ipString = ipString.Split(':')[0];
                        }
                        if (!IsPrivateIpAddress(ipString) && !publicForwardingIps.Contains(ipString))
                        {
                            publicForwardingIps.Add(ipString);
                        }
                    }
                }
                // If we found any, return the last one, otherwise return the user host address
                return publicForwardingIps.Any() ? publicForwardingIps.First() : userHostAddress;
            }
            catch
            {
                return null;
            }
        }

 

Best Regards,

Nuno Aguiar

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

I suggest this:

public static string? GetClientIpAddress(this IRequest request)
{
    try
    {
        // Attempt to parse.
        if (IPAddress.TryParse(request.UserHostAddress, out var iPAddress))
            return iPAddress.ToString();

        return null;
    }
    catch
    {
        return null;
    }
}

Looking at headers like this is a security breach anyways...

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

Sure, I'm happy either way. Can this change be made to:

  • DW9 - /src/88 - Content/Dynamicweb/SystemTools/Security/IpBanner.cs
  • DW10 - /src/Features/Content/Dynamicweb/SystemTools/Security/IpBanner.cs
 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Registered as a feature change - devops#23002 - pull already made for DW10

 
Anouk van der Veer
Reply

Hi guys,

What was the culprit in your case Nuno? We are running into similar problems since about ten days and are still looking for a solution.

 

Kind regards,

Anouk

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Anouk,

 

Do you use a WAF much like Cloudflare? Our issue is that Cloudflare would move the client's IP to a new header (CF-Connecting-IP) and, most importantly, added it's own IP to the X-Forwarded-For.

 

The problem was that DW was using the X-Forwarded-For header's first IP it finds, which is what we were getting in Banned IPs. We assumed CloudFlare does a good job keeping those mappings (i.e. my real ip would always get the same CF IP), but I could never find that mapping anywhere, so I was stuck.

 

We tried to implement this https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/ without success

 

I know the IPBanner code has changed now, but I haven't tested it yet. I know it's not looking at specific headers anymore.

Our issues were also related to IPv6 (some devices only communicated with IPv6), that were not being properly logged. It considered only the first group (4 digits), so it was likely that thousands of IPs would be considered banned as well. This has also been fixed in recent versions.

 

What we did was to mimic (as much as possible) the regex's from DW as WAF rules in CloudFlare. That way, CF only blocks the requests (and never reaches DW). That reduced our issues, because what we were having were infected computers (or bots, or malicious attempts, or ethical hacking classes) from Universities, which blocked their IPs, and other departments were trying to purchase on the site and were Banned.

 

Hope this helps.

 

Best Regards,

Nuno Aguiar

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

We changed DW9 so it will not read from the header directly as that can go wrong - so now there is an explicit setting for which header to read from if any.

In Dynamicweb 10.13 we have implemented ForwardedHeadersMiddleware  https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.httpoverrides.forwardedheadersmiddleware?view=aspnetcore-9.0

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

I don't see any change in DW9 (looking at the History of IpBanner.cs for DW9). I also checked active PRs for DW9 that could suggest it.

 

Are you sure someone changed it for DW9? Or you meant DW10?

 

Best Regards,

Nuno Aguiar

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

Can you please confirm if this was changed for DW9? I don't see it, and we continue to have issues with some customers, and we'd like to get them on a DW9 version that would be less prone to banning valid users.

 

Currently we're having to leave "Do not ban IPs" checked, and that causes a lot more requests to the site (biolegend.com) that keeps getting constanly attacked.

 

Nuno

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Hi Nuno

No - I meant DW10. No changes has been made in DW9.

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

Could they be moved to DW9? We have a series of customers who have had legitimate IP banned, and I hope this improvement will help us capture things better (and upgrading to DW10 will not be an option for the foreseeable future, and leaving IP Banning disabled increases the attacks more than 300x (as those IPs keep trying to hack the system)

 

You might be aware we also use Cloudflare. With that in mind:

  • We got the multiple regex formulas' from IpBanner and added them to Cloudflare
    • Concept: Let Cloudflare block the request - doesn't ban the IP
  • But we're still struggling as it blocks some valid interactions (such as in the backend)

 

Best Regards,

Nuno 

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Hi Nuno

If you use Cloudflare, you should just not use ipbanner - cloudflare should take care of security instead.

I have made a pull request that does 2 things

  • You will now have to manually specify ip forward header string - e.g. 'X-FORWARDED-FOR' in system security settings.
    • That is a new setting, /Globalsettings/System/Security/ip-forward-header, that if not specified, DW will not try to read any forward headers due to security
    • Change of behavior - think it is needed as I consider the current implementation a security breach.
  • If the forwarded header is an IPv6, we no longer check for port. It uses IPAddress.TryParse instead and hence relies on .net logic.

Work item; https://dev.azure.com/dynamicwebsoftware/Dynamicweb/_workitems/edit/23979

 

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

That is great, thank you. It will certainly be of great help.

 

If you use Cloudflare, you should just not use ipbanner - cloudflare should take care of security instead.

I don't disagree, but that has been easier said than done:

  • In DW we have basically 2 checkboxes (SQL Injection + GetImage)
    • And under the hood that's basically 10 regex (roughly)
  • In Cloudflare, they have rulesets we can use
    • To get the same functionality we have in DW, we "expect" to enable 2 of them
    • But these 2 rulesets have more than 600 granular rules (between them)
    • Enabling them all broke normal behavior (backend and frontend)
    • Trying to manually create a Cloudflare rule that matches DW's regex's also broke some functionality (mostly in the backend in some cases)

 

While this happening, we're still having to deal with unhappy customers where customers are either blocked or DW is pounded by bots trying to hack the site. 

Maybe a happy medium, where Cloudflare takes most of the hits, but we still have DW handling the ones that come through will be our final conclusion. That's yet to be determined.

 

You can say I've been having "fun" :)

 

Thanks,

Nuno Aguiar

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Hi Nuno

You do not have to redo the DW rules. I am pretty sure Cloudflare knows what they are doing. 

DW rules only runs for frontend - and not backend. Maybe you can configure the same for Cloudflare. You can disable IP banning but still run DW security - that will at least take care that IPs does not get blocked.

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Nicolai,

 

Yeah, we did not get good results just by leaving Cloudflare settings and subscription:

  • You need the Advanced WAF subscription to check into the Request's Body (POST)
    • In our case we don't have that subscription, so we can only check Querystring parameters (GET) (among a few other things)
  • Cloudflare has what they call Rulesets, which may have rule groups of it's individual rules
    • That adds up to +700 individual rules
    • The simplified version can have a few dozen combinations
    • For each combination we sort of need to run through the site and do a few things to see if that impacts anything (upload files, submit forms, webapi requests, 3rd party libraries...)
    • I had actually googled for the proper defaults for B2B/B2C eCommerce websites, but those kept failing
  • Because of the limitations above, I tried to simply rebuild the DW rules in CF
    • Why?
      • Some customers were getting banned too often.
      • And leaving the IP banning off, increased the traffic by more than 75% to the VM
      • Blocking them in CF, greatly reduced traffic/computation to the VM (which in Azure translates to cost savings - depending on your settings)
    • How?
      • Created a rule to skip all backend requests (except /admin/public/*)
      • Created individual rules
      • Had to adapt a rule though, because CF's regex engine is limited and some PCRE features such as lookaheads/lookbehinds and some escape sequences may not work as expected
    • We will continue to monitor and tweak things.
       

So for now using CF to block the large majority of requests is prooving to be a good solution, while letting DW take care of the requests that come through.

 

This is all FYI and maybe to help someone else in the future who stumbles into this.

 

Best Regards,

Nuno Aguiar

 

You must be logged in to post in the forum