CVE-2021-41125: Scrapy - Exposing HTTP Authentication Credentials
Summary
CVE-2021-41125: Scrapy is a high-level web crawling and scraping framework for Python. Affects scrapy across 3 version ranges.
< 1.8.1
>= 2.0.0, < 2.5.1
< b01d69a1bf48060daec8f751368622352d8b85a6
Fixed in 1.8.1 or 2.5.1, depending on the release line you run. How to fix ↓
Also tracked for
- python-scrapy · Debian:11
The same CVE-2021-41125 is tracked separately for each ecosystem and vendor. Its fix information is the same; the package and affected versions differ.
Check if this vulnerability affects your application
Upload a manifest or SBOM and we'll show whether you're affected, plus every other risky package in the same scan.
No signup required · Results in about 60 seconds
Background
Scrapy is a powerful Python framework used for web crawling and scraping. It provides a high-level interface for developers to extract data from websites. Scrapy supports various features and components to facilitate the scraping process, including the ability to handle HTTP authentication. However, a vulnerability has been identified in Scrapy versions prior to 2.5.1, where the use of HttpAuthMiddleware can lead to the exposure of HTTP authentication credentials. This vulnerability affects the security of requests made by Scrapy components, such as robots.txt requests and requests reached through redirects.
Vulnerability Detail
The vulnerability (CVE-2021-41125) in Scrapy arises from the use of HttpAuthMiddleware for HTTP authentication. When using this middleware, all requests made by Scrapy, including those generated by Scrapy components, will expose the configured credentials to the request target. This means that the credentials can be intercepted by malicious actors, potentially leading to unauthorized access or other security breaches. The vulnerability affects Scrapy versions prior to 2.5.1 and can have serious implications for applications relying on HTTP authentication.
How To Fix
Latest published version: 2.19.0 as of 22 Oct 2026
pip install --upgrade "scrapy==2.19.0"Upgrading to it may include breaking changes; the versions below are the minimum fixed release for this vulnerability.
If you can't move to the latest
The minimum fixed version depends on the release line you run.
-
< 1.8.11.8.1pip install --upgrade "scrapy==1.8.1" -
>= 2.0.0, < 2.5.12.5.1pip install --upgrade "scrapy==2.5.1"
Tip — Not sure which release line you're on? Vulert can find the fix for the version you are using. Scan free →
Workaround
If you are unable to upgrade Scrapy or set credentials on a per-request basis, there is no specific workaround provided by the advisory. However, it is crucial to avoid exposing sensitive credentials in requests sent by Scrapy. Developers should carefully review their code and ensure that HTTP authentication credentials are handled securely, considering alternative authentication methods if necessary.
Conclusion
The vulnerability in Scrapy versions prior to 2.5.1 exposes HTTP authentication credentials, potentially leading to unauthorized access or security breaches. It is crucial to upgrade to Scrapy 2.5.1 and utilize the new http_auth_domain attribute to control the domains that receive the credentials. If upgrading is not possible, consider upgrading to Scrapy 1.8.1 or implementing per-request credential assignment. For further assistance and support in addressing this vulnerability, please contact DevNack.com.
See every vulnerability recorded for scrapy →
FAQ
Scrapy is a Python framework used for web crawling and scraping. It provides a high-level interface for developers to extract data from websites efficiently.
The vulnerability in Scrapy can lead to the exposure of HTTP authentication credentials. This can potentially result in unauthorized access to sensitive information or other security breaches.
Scrapy versions prior to 2.5.1 are affected by this vulnerability. This includes Scrapy 1.8 and lower versions.
The recommended fix is to upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains receive the HTTP authentication credentials. Alternatively, you can upgrade to Scrapy 1.8.1 or set the credentials on a per-request basis.
While no specific workaround is provided, it is crucial to handle HTTP authentication credentials securely and avoid exposing them in requests sent by Scrapy. Review your code and consider alternative authentication methods if necessary.
Related vulnerabilities
Recent vulnerabilities recorded for scrapy.
