Exclusions in Web Application Firewall

When enabling Web Application Firewall on an Application Gateway in Azure you will probably run into problems where some legitimate requests are blocked. This post is an attempt to make it easier understand why these requests were blocked and how to ensure they are not blocked in the future. This post assumes that the Web Application Firewall is deployed with the OWASP rule set. I also assume that diagnostic settings has already been configured to make sure we log everything we need.

How the WAF works with rules

First of all we need to briefly touch on how the WAF works with rules and anomaly scores. As of this writing the OWASP rule set contains 186 rules. Each of these rules has an anomaly score. If a request matches a rule, that request will get a score, based on the severity of the matched rule. If multiple rules are matched the anomaly scores will be added together. After all rules have completed processing the request will have a total score based on all rules matched. If the total score is above the threshold, and the WAF is in prevention mode, the request will be blocked.

Each rule has a severity which decides its contribution to anomaly score.

The anomaly score threshold is 5, which means that one critical rule is enough to block a request.

Finding the requests in the logs

When a request has been blocked, you should be able to find the request in the AzureDiagnostics table. Finding blocked requests is easy:

AzureDiagnostics 
| where action_s == 'Blocked'
| project TimeGenerated, transactionId_g, requestUri_s, Message, clientIp_s, ruleSetType_s, ruleId_s, ruleGroup_s, action_s, details_message_s

Running this statement should produce some results where you can see blocked requests. Here is an example, converted to JSON for visibility.

{
    "TimeGenerated [UTC]": "12/13/2024, 10:56:13.628 AM",
    "transactionId_g": "6850b97f-19ac-4b49-b949-80e7795b8029",
    "requestUri_s": "/api/customer",
    "Message": "Inbound Anomaly Score Exceeded (Total Score: 7)",
    "clientIp_s": "35.13.204.26",
    "ruleSetType_s": "OWASP CRS",
    "ruleId_s": "949110",
    "ruleGroup_s": "REQUEST-949-BLOCKING-EVALUATION",
    "action_s": "Blocked",
    "details_message_s": "Greater and Equal to Tx:inbound_anomaly_score_threshold at TX:anomaly_score."
}

This request has been blocked by the firewall. However, we do not know why it has been blocked. The results show us that the request contained certain suspicios content which produced an anomaly score higher than the threshold (5). Now we need to figure out what rules caused the block. Thats what the transactionId_g is for.

AzureDiagnostics 
| where transactionId_g == '6850b97f-19ac-4b49-b949-80e7795b8029'
| project TimeGenerated, transactionId_g, requestUri_s, Message, clientIp_s, ruleSetType_s, ruleId_s, ruleGroup_s, action_s, details_message_s, details_data_s

{
    "TimeGenerated [UTC]": "12/13/2024, 10:56:13.628 AM",
    "transactionId_g": "6850b97f-19ac-4b49-b949-80e7795b8029",
    "requestUri_s": "/api/customer",
    "Message": "Missing User Agent Header",
    "clientIp_s": "35.13.204.26",
    "ruleSetType_s": "OWASP CRS",
    "ruleId_s": "920320",
    "ruleGroup_s": "REQUEST-920-PROTOCOL-ENFORCEMENT",
    "action_s": "Matched",
    "details_message_s": "Equal 0 at REQUEST_HEADERS:user-agent.",
    "details_data_s": "{ found within [REQUEST_HEADERS:0]}"
},
{
    "TimeGenerated [UTC]": "12/13/2024, 10:56:13.628 AM",
    "transactionId_g": "6850b97f-19ac-4b49-b949-80e7795b8029",
    "requestUri_s": "/api/customer",
    "Message": "SQL Comment Sequence Detected.",
    "clientIp_s": "35.13.204.26",
    "ruleSetType_s": "OWASP CRS",
    "ruleId_s": "942440",
    "ruleGroup_s": "REQUEST-942-APPLICATION-ATTACK-SQLI",
    "action_s": "Matched",
    "details_message_s": "Pattern match (?:/\\*!?|\\*/|[';]--|--[\\s\\r\\n\\v\\f]|--[^-]*?-|[^&-]#.*?[\\s\\r\\n\\v\\f]|;?\\x00) at ARGS.",
    "details_data_s": "{/* found within [ARGS:CustomerFolders:/customers/*cust1]}"
},
{
    "TimeGenerated [UTC]": "12/13/2024, 10:56:13.628 AM",
    "transactionId_g": "6850b97f-19ac-4b49-b949-80e7795b8029",
    "requestUri_s": "/api/customer",
    "Message": "Inbound Anomaly Score Exceeded (Total Score: 7)",
    "clientIp_s": "35.13.204.26",
    "ruleSetType_s": "OWASP CRS",
    "ruleId_s": "949110",
    "ruleGroup_s": "REQUEST-949-BLOCKING-EVALUATION",
    "action_s": "Blocked",
    "details_message_s": "Greater and Equal to Tx:inbound_anomaly_score_threshold at TX:anomaly_score.",
}

There are three entries in the log for this transaction. The last one is the one we found earlier which is blocking the request. The first two entries shows rules which are matched. The first one shows that the request is missing a User Agent Header. This is suspicious behaviour, but should be easy to fix by whoever did the request. Fixing the header might actually be enough for this request to be allowed.

The second one shows that there is a suspicios string that could be an indication of a SQL Injection attack. The string in this example is “/*” which is a comment in certain SQL dialects. In the details_data_s column it even shows you where it found it in the request body: ARGS:CustomerFolders:/customers/*cust1

So now we know why the request was blocked and its time to figure out what we can do with this information. But before we do that, its good to know what severity these rules have.

Rule definition and severity

When multiple rules are matched its often good to know the severity of those rules. If the second rule about SQL comments has critical severity it doesnt matter if you add headers to the request to fix the first rule, the request will still be blocked. Sometimes you can get lucky and the severity is shown in the Message column, but often its not. (Please Microsoft, always add the severity to the log)

Fortunately the OWASP rule set is available on GitHub: https://github.com/coreruleset/coreruleset

Here you can find the rule definition for the second rule which has rule id 942440.

# -=[ Detect MySQL in-line comments ]=-
#
# MySQL in-line comments can be used to bypass SQLi detection.
#
# Ref: https://dev.mysql.com/doc/refman/8.0/en/comments.html:
# SELECT /*! STRAIGHT_JOIN */ col1 FROM table1,table2 WHERE ...
# CREATE TABLE t1(a INT, KEY (a)) /*!50110 KEY_BLOCK_SIZE=1024 */;
# SELECT /*+ BKA(t1) */ FROM ... ;
#
# http://localhost/test.php?id=9999+or+{if+length((/*!5000select+username/*!50000from*/user+where+id=1))>0}
#
# The minimal string that triggers this regexp is: /*!*/ or /*+*/.
# The rule 942500 is related to 942440 which catches both /*! and */ independently.
#
# Regular expression generated from regex-assembly/942500.ra.
# To update the regular expression run the following shell script
# (consult https://coreruleset.org/docs/development/regex_assembly/ for details):
#   crs-toolchain regex update 942500
#
SecRule REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/* "@rx (?i)/\*[\s\x0b]*?[!\+](?:[\s\x0b\(\)\-0-9=A-Z_a-z]+)?\*/" \
    "id:942500,\
    phase:2,\
    block,\
    capture,\
    t:none,t:urlDecodeUni,\
    msg:'MySQL in-line comment detected',\
    logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-sqli',\
    tag:'paranoia-level/1',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/152/248/66',\
    tag:'PCI/6.5.2',\
    ver:'OWASP_CRS/4.10.0-dev',\
    severity:'CRITICAL',\
    multiMatch,\
    setvar:'tx.sql_injection_score=+%{tx.critical_anomaly_score}',\
    setvar:'tx.inbound_anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

Notice that severity is set to CRITICAL which, according to the table above, gives an anomaly score of 5 which is the same as the threshold. That means that any request that matches this rule will always be blocked, even if its the only rule matched.

While we look at the definition of the rule, notice that this rule checks both cookies and arguments (query string and body). That kind of information is really good to have when troubleshooting OWASP rules in general.

Options to solve the problem

So now we know that the SQL comment need to be handled somehow. To allow this kind of request we now have three options.

1. Change the request

This is probably the best approach to look into first. If you can change the behaviour of the request to not look suspicios and learn from that for future requests then you probably have most to gain. This is certainly not viable for every scenario as there might be business logic or technical reasons to why the request need to look like it does.

2. Create exclusion matching the request content

There are plenty of scenarios where you cant change how the request is sent. In those scenarios you can create exclusions. An exclusion is an exception of the rule. When the request looks in a certain way, its fine to skip the rule. This is generally what is recommended when the request cannot be changed.

3. Disable the rule

Disabling a rule should really be the last resort. If there is no other way, go ahead and disable it, but most of the time there is another way.

Create rule exclusion

When you create a rule exclusion you first need to select which rule the exclusion applies to. Then you select a match variable, operator and selector. I have always found the match variable to be confusing. Here are the options:

• RequestArgKeys
• RequestArgNames
• RequestArgValues
• RequestHeaderKeys
• RequestHeaderNames
• RequestHeaderValues
• RequestCookieKeys
• RequestCookieNames
• RequestCookieValues

When reading the documentation its sometimes hard to distinguish the differences between Keys, Names and Values. Some of the examples in the documentation does not show a clear difference between these types. The RequestArg variables has to do with the request body and query string in the url, so in this case we want to use one of those.

The request body we are trying to make an exclusion for looks like this:

{
    "CustomerName": "Customer1",
    "CustomerFolders": "/customers/*cust1",
    "CustomerCountry": "SE"
}

Remember, the request was blocked because the CustomerFolders contained “/*”. So we need to make an exclusion for this JSON property. If we then try to use RequestArgValues, set the Operator to Equals and then enter the key “CustomerFolders” from the JSON body. This tells the rules engine to skip the rule if the request contains a JSON body with the property CustomerFolders.

Here is the bicep code for it:

//...
exclusions: [
    {
        matchVariable: 'RequestArgValues'
        selectorMatchOperator: 'Equals'
        selector: 'CustomerFolders'
        exclusionManagedRuleSets: [
            {
            ruleSetType: 'OWASP'
            ruleSetVersion: '3.2'
            ruleGroups: [
                {
                ruleGroupName: 'REQUEST-942-APPLICATION-ATTACK-SQLI'
                rules: [
                    {
                        ruleId: '942440'
                    }
                ]
                }
            ]
            }
        ]
    }
]
//...

This rule would be enough to allow this request through the Web Application Firewall.

Success!

Nested JSON body

In the example above we had a flat JSON body. If you have a nested JSON body with child items, you simply “dot” your way through it. Here is an example body:

{
    "CustomerDetails": {
        "CustomerName": "Customer1",
        "CustomerCountry": "SE"
    },
    "CustomerConfig": {
        "CustomerFolders": "/customers/*cust1"
    }
}

Then if you want to use the selector to point at CustomerFolders which is located in CustomerConfig you simply write it like this:

//...
exclusions: [
    {
        matchVariable: 'RequestArgValues'
        selectorMatchOperator: 'Equals'
        selector: 'CustomerConfig.CustomerFolders'
        exclusionManagedRuleSets: [
            {
            ruleSetType: 'OWASP'
            ruleSetVersion: '3.2'
            ruleGroups: [
                {
                ruleGroupName: 'REQUEST-942-APPLICATION-ATTACK-SQLI'
                rules: [
                    {
                        ruleId: '942440'
                    }
                ]
                }
            ]
            }
        ]
    }
]
//...

More about rule exclusions

Here are a few more details that has been helpful to me when working with rule exclusions.

Restricted to JSON

It seems like the rule engine only interprets JSON, so if you are using XML for example you are much more restricted when configuring exclusions.

No URL filtering

If you create a custom rule you can use URL as a filter for that rule. URL filters are not supported when creating exclusions. This would have been really helpful to further narrow down the scope of the exclusion.

Be careful with custom rules

You should really think twize before adding a custom rule since the rule processing will change. From the documentation:

Custom rules are always applied before rules in the Core Rule Set are evaluated. If a request matches a custom rule, the corresponding rule action is applied. The request is either blocked or passed through to the back-end. No other custom rules or the rules in the Core Rule Set are processed.

Thats it for today. I hope you find this helpful.

- Written by John Roos on December 13, 2024