Sensitive Information Exposure in python-scrapy Due to Insecure Redirect Handling
Summary
DEBIAN-CVE-2024-1968: In scrapy/scrapy, an issue was identified where the Authorization header is not removed during redirects that only change the scheme (e.g., HTTPS to HTTP) but remain within the same domain. Affects python-scrapy versions < 2.11.2-1. Fixed in 2.11.2-1. How to fix ↓
Check if this vulnerability affects your application
Upload a manifest or SBOM and we'll show whether you're affected, plus every other risky package in the same scan.
No signup required · Results in about 60 seconds
Background
python-scrapy is a powerful web scraping framework for Python, designed to extract data from websites efficiently. It provides tools for handling requests, parsing responses, and managing data pipelines. However, a vulnerability has been identified in its redirect handling mechanism, which could expose sensitive information during insecure redirects.
Vulnerability Detail
In scrapy/scrapy, an issue was identified where the Authorization header is not removed during redirects that only change the scheme (e.g., HTTPS to HTTP) but remain within the same domain. This behavior contravenes the Fetch standard, which mandates the removal of Authorization headers in cross-origin requests when the scheme, host, or port changes. Consequently, when a redirect downgrades from HTTPS to HTTP, the Authorization header may be inadvertently exposed in plaintext, leading to potential sensitive information disclosure to unauthorized actors. The flaw is located in the _build_redirect_request function of the redirect middleware.
How To Fix
Minimum fixed version
2.11.2-1
This is the first release that contains the fix.
sudo apt-get install --only-upgrade python-scrapyTip — Not sure which version your app runs? Vulert can match this fix to it. Scan free →
Workaround
As a temporary workaround, consider implementing a custom redirect middleware that explicitly removes the Authorization header for redirects that change the scheme from HTTPS to HTTP. This can be done by overriding the _build_redirect_request function to include logic for header removal.
Conclusion
The vulnerability in python-scrapy poses a significant risk by exposing sensitive information through insecure redirect handling. It is crucial for users to update to the latest patched version to mitigate this risk. Additionally, implementing workarounds can provide temporary relief until the update is applied.
See every vulnerability recorded for python-scrapy →
FAQ
The vulnerability allows the Authorization header to be exposed during insecure redirects, potentially leaking sensitive information.
You can use the Vulert playground at Vulert Scanner to check if your application is affected by this vulnerability without any signup.
If exploited, unauthorized actors could gain access to sensitive information that is transmitted in the Authorization header, compromising user security.
Yes, a patch is available in version 2.11.2-1 of python-scrapy. It is recommended to update to this version or later.
If immediate updating is not possible, consider implementing a custom middleware to remove the Authorization header during insecure redirects as a temporary measure.
References
Related vulnerabilities
Recent vulnerabilities recorded for python-scrapy.
