Scrapy is a great tool I agree. I also used this spider in one of my webinars. However when you need to bypass strict anti-bot systems like Cloudflare Turnstile you might need to utilize some trick: Use an headless or anti-detect browser to get the cf_clearance cookie and inject it to your Scrapy requests. This cookie will work as a "pass" and the anti-bot system won't stop you. See how: https://kameleo.io/blog/how-to-bypass-cloudflare-turnstile-with-scrapy
Scrapy is a great tool I agree. I also used this spider in one of my webinars. However when you need to bypass strict anti-bot systems like Cloudflare Turnstile you might need to utilize some trick: Use an headless or anti-detect browser to get the cf_clearance cookie and inject it to your Scrapy requests. This cookie will work as a "pass" and the anti-bot system won't stop you. See how: https://kameleo.io/blog/how-to-bypass-cloudflare-turnstile-with-scrapy
Great article. I think you have a small typo on the urls.
target_urls = [
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2",
"https://httpbin.org/html",
"https://httpbin.org/status/404",
"https://httpbin.org/delay/0.5",
]