Limitations and Precautions for URL Filtering

Hardware Requirements

The URL filtering function is supported by all models.

Only the USG6510E/6510E-POE/6530E does not support safe search for HTTP search requests.

License Requirements

The URL filtering blacklist and whitelist functions, user-defined URL categories, and predefined URL categories based on the preset URL category database are not license-controlled.

The URL remote query function is URL remote query license-controlled. For details about the license control scopes, see the License Control Items.

Component package Requirements

To use the URL remote query function, you need to load the URL remote query component package. For details about the component package, see Dynamic Loading.

Limitations for URL Categories

• If the URL remote query license and component package are not loaded, the configuration items of the remote URL query function are unavailable on the web UI.
• User-defined URL categories have a higher priority than predefined URL categories.
• If the domain name in a URL request meets any of the following conditions, remote query-based URL category filtering is not supported. If the domain name does not match the URL blacklist/whitelist, user-defined URL categories, or predefined categories in the local cache, the device takes the action specified for the Others category (allow by default) and does not perform remote query.
  • The domain name length is less than four bytes.
  • The domain name starts and ends with a hyphen (-).
  • The domain name contains characters other than letters, digits, hyphens (-), colons (:), and periods (.).
  • Invalid IPv6 address
  • Other domain names that do not comply with RFC standards (such as RFC 1034 and RFC 1035)

Limitations for Safe Search

• The safe search function can be available only when a TCP proxy policy and an SSL-encrypted traffic detection policy are configured on the FW.
  • For HTTP search requests, you need to configure a TCP proxy policy on the FW.
  • For HTTPS search traffic, you need to configure an SSL-encrypted traffic detection policy on the FW.
• The safe search of the FW does not take effect when the SSL proxy is deployed between the browser and the browser proxy server.
• The URL safe search function is unavailable in the following scenarios:
  • Protocol traffic scenarios that are not supported by the FW, such as HTTP/2, HTTP/3, and QUIC
  • URL GET requests are segmented in cross-packet scenarios.
  • POST request scenario for the YouTube search engine
  • Scenario where encrypted traffic cannot be decrypted by the FW

  The URL safe search function is implemented based on some solutions disclosed by the search engine service provider. The actual filtering effect depends on the safe search capabilities of the service provider. The FW cannot 100% ensure the safe search filtering effect.

  URL safe search and DNS safe search must be used together.

Limitations for Google Account Control

• The google account control of the FW does not take effect when the SSL proxy is deployed between the browser and the browser proxy server.
• If google account control is required, SSL-encrypted traffic detection needs to be configured on the FW.
• Currently, the google account control function for mobile applications is not supported.
• The Google account control feature is unavailable in the following scenarios:
  • Protocol traffic scenarios that are not supported by the FW, such as HTTP/2, HTTP/3, and QUIC
  • URL GET requests are segmented in cross-packet scenarios.
  • Scenario where encrypted traffic cannot be decrypted by the FW

  The Google account control function is implemented based on some solutions disclosed by the search engine service provider. The actual effect depends on the capability of the service provider. The FW cannot 100% ensure the control effect.

Restrictions for Encrypted Traffic Filtering of HTTPS URL Filtering

• Encrypted traffic filtering of URL filtering supports TLS 1.0 and later versions.
• URL filtering supports only HTTP/1.0 and HTTP/1.1. To filter HTTPS URL requests, you also need to configure SSL-Encrypted Traffic Detection or encrypted traffic filtering function. If you want HTTPS URL filtering to be more accurate, you are advised to configure the SSL encrypted traffic detection function. However, the URL filtering implemented by the encrypted traffic filtering function is at the domain name level, which is not accurate enough. It obtains the domain name of the website that a user accesses from the Server Name Indication (SNI) field in the client's Client Hello packet and the Common Name (CN) and Subject Alternative Name (SAN) fields in the server's Certificate packet. Before using the encrypted traffic filtering function to implement HTTPS URL filtering, read the following restrictions:
  • When a browser has a proxy server, the domain name extracted by the FW from the SNI field in the client's Client Hello packet and the CN and SAN fields in the server's Certificate packet may be different from the actual domain name of the website. If this happens, URL filtering does not take effect. For example, when a user uses the UC browser on a mobile phone to access Youku, URL filtering does not take effect. To resolve this issue, disable the cloud acceleration function of the UC browser on the mobile phone.
  • If some browsers use proprietary protocols to transmit data, URL filtering may not take effect. For example, Google Chrome uses the proprietary protocol Quick UDP Internet Connection (QUIC) for data transmission by default. This protocol uses a dedicated encryption mode that the FW cannot decrypt. As a result, URL filtering cannot filter HTTPS websites accessed through Google Chrome. To resolve this issue, configure a security policy to block QUIC applications.
  • When a website server provides services for multiple domain names using the same IP address, the SNI field may contain multiple domain names. If only one domain name is configured in the URL filtering blacklist, not all traffic from this website cannot be blocked. For example, when a user accesses www.example.com through HTTPS, the website may have multiple domain names, such as 1.example.com and 2.example.com. If only 1.example.com is configured in the URL filtering blacklist, not all traffic of this website cannot be blocked. To resolve this issue, configure level-1 domain name *.example.com in the blacklist.
  • When a certificate contains multiple URLs (that is, multiple websites use one certificate), the SAN field may contain multiple URLs. If one URL is blocked, the traffic from this URL is blocked and other URLs cannot be accessed.
  • If the domain name of a website is added to the URL filtering whitelist, not all embedded pages of the website can be accessed. For example, if only www.example.com is added to the whitelist, the embedded pages whose domain names are not www.example.com cannot be accessed. To resolve this issue, add the embedded pages whose domain names are not www.example.com to the whitelist or configure the embedded whitelist function.
  • Encrypted traffic filtering implements URL filtering through SSL handshake packets. The client does not initiate an HTTP request. Therefore, sending URL push information is not supported. You can configure SSL-encrypted traffic detection to implement the function of sending URL push information.
  • If the secure DNS option is enabled on the user's browser, the Encrypted Client Hello (ECH) extended option may be used in the SSL handshake phase. In this case, the domain name extracted by the device from the SNI is inconsistent with the actual domain name accessed by the user, the URL filtering function does not take effect. To handle this, you need to enable SSL-encrypted traffic detection or disable the secure DNS option in the browser.
  • If the secure DNS option is enabled in the user browser, the ECH extension option may be used in the SSL handshake phase. In this case, the domain name extracted by the device from the SNI is inconsistent with the actual domain name accessed by the user. As a result, the URL filtering function does not take effect.

    You need to disable the ECH option in the browser or configure a security policy on the device to block encrypted DNS traffic.

    Configure a security policy to block encrypted DNS traffic, including encrypted protocol traffic, such as QUIC traffic (with the UDP service and destination port 443), HTTPS traffic (with the TCP service and destination port 443), and DNS_over_TLS traffic (with the TCP service and destination port 853), traffic with IP addresses of well-known DNS servers (1.0.0.3, 1.1.1.3, 8.8.4.4, and 8.8.8.8), and encrypted DNS application traffic (such as DNS_ECH and DNS_Over_HTTPS).

  • When TLS 1.3 ClientHello packets are segmented, URL filtering does not take effect for encrypted traffic in non-decryption mode.

  With the development of Internet technologies, more URLs that cannot be accurately filtered using the encrypted traffic filtering function may emerge. The preceding lists only some typical restrictions.

Other Limitations for URL Filtering

• In the exact match mode of the URL/HOST rule in URL filtering, the length of URLs that can be matched cannot contain more than 1279 characters. If the URL length exceeds this limit, exact match cannot be performed. In this case, you need to change the match of the URL/HOST rule to fuzzy match (with the wildcard *).
• The default HTTP port number is 80, and the default HTTPS port number is 443. If the server uses the default port, you do not need to configure the port number in a URL filtering rule. If the server uses a non-default port, the port number is mandatory in a URL filtering rule. For example, to block the access of 10.1.1.1, if the server uses the default port 80, configure 10.1.1.1 instead of 10.1.1.1:80 in the URL filtering blacklist; if the server uses port 8080, configure 10.1.1.1:8080 in the URL filtering blacklist.
• The URL filtering function is unavailable in the networking where the forward and return packet paths are different.
• The URL filtering function does not support filtering online-proxied URL requests.
• URL filtering supports IPv4 and IPv6.
• The port mapping function supports only IPv4.
• If the browser (for example, Google Chrome) caches a web page, the browser does not request the web page but refreshes the sub-pages on the web page when the web page is accessed again. This may result in the failure to display the URL filtering push page. Therefore, you are advised to clear the browser cache before using the URL filtering push page.
• The TCP proxy and SSL-encrypted traffic detection features do not support IPv6. Therefore, the following situations occur:
  • IPv6-based HTTPS traffic cannot be decrypted during SSL encrypted traffic detection.
  • The IPv6-based safe search function does not take effect.
• The FW preferentially ensures network connectivity based on the service preference principle. URL filtering is a functional component, and its effect is affected by multiple factors. The URL filtering effect cannot be 100% guaranteed.

  The factors include but are not limited to device performance overload (traffic burst and traffic threshold-crossing), insufficient resources (such as CPU, memory, and queue channel resources), performance first mode, escape mechanism in resilience scenarios where URL filtering is bypassed. (Examples of such scenarios include integrated policy matching timeouts, protocol decoder exceptions, and domain names with special characters.)

  In scenarios where strict domain name control (such as monitoring) is required, a single FW cannot 100% ensure the filtering effect. In this case, you are advised to select a proper end-to-end solution.

Precautions for URL Filtering

• When the URL filtering is used to perform content security detection on traffic, the performance of the device is affected. Therefore, configure the function as required.
• The URL filtering function takes effect for all URL requests, including the web pages accessible to users and all website links on a web page. Generally, URL filtering rules take effect only for the URLs of web pages. To limit the website links on web pages, configure separate URL filtering rules.
• If a URL rule contains a number sign (#), # and the following string do not apply to rule matching. If the URL that a user accesses contains #, # and the following string will not be sent to the URL module for URL matching.
• If a session contains multiple URLs, the FW performs URL filtering on each URL and blocks the entire session as long as any one of the URLs is blocked.
• If the FW is deployed between two routers, and the routers detect each other through BFD, you are advised to properly prolong the BFD time (longer than 100 ms is recommended) to prevent BFD flapping resulting from occasional network congestion.
• To ensure that functions such as the URL blacklist and whitelist take effect, you need to configure the fuzzy match method for HTTPS URL filtering. For example, you can set the fuzzy match mode of www.huawei.com to *huawei*.
• The embedded URL filtering whitelist function solves the problem that users cannot access some embedded web pages in the whitelist. This function needs to obtain the Referer field (identifying the web page from which a user is directed) in an HTTP request for whitelist matching. Pay attention to the following when using this function:
  • For HTTPS requests, you need to configure SSL-encrypted traffic detection for decryption so that the device can obtain the Referer field information and match against the embedded whitelist.
  • Generally, when you access an embedded web page, the browser sets the HTTP Referer field information.
  • If the embedded page request does not contain the HTTP Referer field due to special browser settings, the embedded whitelist function is unavailable.
  • When a user attempts to access an embedded web page in the whitelist, if the resources (such as images, CSS files, JS files, and subpages) contained in the embedded web page are not in the whitelist, URL filtering blocks the access.
  • Therefore, only some static content can be displayed on the embedded web page, and the URL filtering blocking page is displayed for the blocked content.