CVE-2021-41124: Vulnerability in scrapy-splash

Background

scrapy-splash is a library that integrates Scrapy and JavaScript, allowing users to combine the two for web scraping purposes. It provides functionality for handling authentication with Splash, a headless browser. However, in certain versions of scrapy-splash, there is a vulnerability related to the usage of HttpAuthMiddleware for Splash authentication. This vulnerability exposes user credentials to the request target when non-Splash requests are made, including robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is enabled.

Vulnerability Detail

The vulnerability in scrapy-splash arises from the use of HttpAuthMiddleware for Splash authentication. When this middleware is used with the http_user and http_pass spider attributes, non-Splash requests can expose the user's credentials to the request target. This includes requests for robots.txt files, which are automatically made by Scrapy when the ROBOTSTXT_OBEY setting is enabled. The impact of this vulnerability is that an attacker who has access to the request target can potentially obtain the user's credentials. This can lead to unauthorized access to sensitive information or accounts.

pip install --upgrade "scrapy-splash==2b253e57fe64ec575079c8cdc99fe2013502ea31"

Workaround

If you are unable to upgrade to scrapy-splash 0.8.0 or implement the recommended changes, you can use the following workaround to mitigate the vulnerability. Instead of using HttpAuthMiddleware for authentication, you can set the Splash request credentials on a per-request basis using the splash_headers parameter. Here is an example of how to do this in a Scrapy spider:

import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'

    def start_requests(self):
        yield scrapy.Request(
            url='http://example.com',
            meta={
                'splash': {
                    'args': {
                        'splash_headers': {
                            'Authorization': 'Basic base64encodedcredentials'
                        }
                    }
                }
            }
        )

    def parse(self, response):
        # Parse the response
        pass

In this example, the Splash request credentials are set in the splash_headers parameter within the meta field of the Request object.

Conclusion

The vulnerability in scrapy-splash exposes user credentials to the request target when non-Splash requests are made. To mitigate this vulnerability, it is crucial to upgrade to scrapy-splash 0.8.0 or later and use the new SPLASH_USER and SPLASH_PASS settings for Splash authentication. If upgrading is not possible, the workaround of setting the Splash request credentials on a per-request basis using the splash_headers parameter can be employed. For further assistance with vulnerability remediation, please contact DevNack.com.

See every vulnerability recorded for scrapy-splash →

FAQ