Sensitive Information Exposure in Scrapy Due to Improper Authorization Header Handling
Summary
CVE-2024-1968: In scrapy/scrapy, an issue was identified where the Authorization header is not removed during redirects that only change the scheme (e.g., HTTPS to HTTP) but remain within the same domain. Affects python-scrapy versions < 1d0502f25bbe55a22899af915623fda1aaeb9dd8. No fix is published. How to fix ↓
Check if this vulnerability affects your application
Upload a manifest or SBOM and we'll show whether you're affected, plus every other risky package in the same scan.
No signup required · Results in about 60 seconds
Background
Scrapy is an open-source web crawling framework written in Python, designed for extracting data from websites. It provides a robust set of tools for web scraping, including handling requests, parsing responses, and managing data storage. The framework is widely used for data mining and web scraping tasks due to its flexibility and ease of use. However, it is vulnerable to a significant issue where the Authorization header is not removed during certain redirects, potentially exposing sensitive information.
Vulnerability Detail
The vulnerability, identified as CVE-2024-1968, arises from the improper handling of the Authorization header during redirects that change the scheme from HTTPS to HTTP while remaining within the same domain. This behavior violates the Fetch standard, which requires the removal of Authorization headers in cross-origin requests when the scheme, host, or port changes. As a result, sensitive information contained in the Authorization header may be exposed in plaintext, allowing unauthorized actors to access confidential data.
How To Fix
No fixed version is published for this vulnerability.
Tip — Not sure whether your app is affected? Vulert can check the versions your app actually runs. Scan free →
Workaround
As a temporary workaround, you can manually remove the Authorization header from requests before performing redirects. This can be done by modifying the _build_redirect_request function in the redirect middleware of Scrapy. Here is a sample code snippet:
class CustomRedirectMiddleware:
def _build_redirect_request(self, request, response, target_url):
new_request = request.replace(url=target_url)
new_request.headers.pop('Authorization', None) # Remove Authorization header
return new_request
Conclusion
CVE-2024-1968 highlights a critical vulnerability in the Scrapy framework that can lead to the exposure of sensitive information through improper handling of the Authorization header during redirects. It is essential to update to the latest version of Scrapy to mitigate this risk and ensure the security of your applications.
See every vulnerability recorded for python-scrapy →
FAQ
CVE-2024-1968 is a vulnerability in the Scrapy framework that allows the Authorization header to be exposed during certain redirects, potentially leading to sensitive information disclosure.
If your application uses Scrapy and performs redirects from HTTPS to HTTP, sensitive information in the Authorization header may be exposed to unauthorized users.
All versions of Scrapy prior to the fix in version 2.11.2-1 for Debian 13 are affected by this vulnerability.
You can use Vulert's playground at Vulert Scanner to check if your application is affected by this vulnerability without needing to sign up.
If you cannot update Scrapy right away, implement the provided workaround to remove the Authorization header during redirects until you can apply the fix.
Related vulnerabilities
Recent vulnerabilities recorded for python-scrapy.
