CVE-2021-41125: HTTP authentication credentials potentially leaked to target websites in scrapy

Background

Scrapy is an open-source web crawling framework written in Python. It provides a set of tools and libraries for extracting data from websites. Scrapy allows users to define spiders that navigate websites and extract structured data. It supports various features like handling cookies, handling redirects, and HTTP authentication. However, a vulnerability has been identified in Scrapy that can potentially leak HTTP authentication credentials to target websites.

Vulnerability Detail

The vulnerability arises when using the HttpAuthMiddleware in Scrapy for HTTP authentication. If the spider attributes http_user and http_pass are used for authentication, all requests made by Scrapy, including robots.txt requests and requests reached through redirects, will expose the credentials to the target website. This can lead to unauthorized access to sensitive information or misuse of the credentials by malicious actors.

pip install --upgrade "scrapy==2.19.0"

Workaround

If upgrading Scrapy is not feasible, a workaround is to set the HTTP authentication credentials on a per-request basis. This can be done by using the w3lib.http.basic_auth_header function to convert the credentials into a value that can be assigned to the Authorization header of each request. Here's an example of how to implement this workaround in Python:

import scrapy
from w3lib.http import basic_auth_header

class MySpider(scrapy.Spider):
    name = 'myspider'
    start_urls = ['http://example.com']

    def start_requests(self):
        credentials = (self.http_user, self.http_pass)
        auth_header = basic_auth_header(*credentials)
        headers = {'Authorization': auth_header}
        for url in self.start_urls:
            yield scrapy.Request(url, headers=headers)

    def parse(self, response):
        # Parse the response here
        pass

Conclusion

The vulnerability in Scrapy allows HTTP authentication credentials to be potentially leaked to target websites. To mitigate this issue, it is crucial to upgrade to Scrapy 2.5.1 or higher and use the new http_auth_domain spider attribute. If upgrading is not feasible, setting the credentials on a per-request basis using the w3lib.http.basic_auth_header function can serve as a workaround. For further assistance on vulnerability remediation, please contact DevNack.com.

See every vulnerability recorded for scrapy →

FAQ