I recently discovered Wanderio and wanted to commend you on the great work you're doing. With ten years in web scraping, I’d like to share my thoughts.
I agree that LLMs are transforming web scraping, leading to two significant shifts. First, productivity is increasing; teams can manage more scrapers without expanding headcount. Second, AI companies' demand for data is pushing them to scrape in less conventional ways, prompting websites to enhance bot protections, complicating scraping efforts.
To adapt, we require more sophisticated tools, whether self-hosted or third-party. I interviewed Or Lenchner, CEO of Bright Data, for my YouTube channel (videos coming in September). He predicts a rise in "web unblocker" usage due to this growing complexity.
As you noted, data is also available for purchase on marketplaces. With rising complexity, not everyone needs to scrape the same sites for basic data like product prices. Recently, we launched Databoutique.com, a marketplace for web data, allowing users to purchase data easily.
Despite past stigmas, web scraping is gaining momentum due to the AI boom, so long live web scraping!
This is a great article, I have done some web scraping myself. I would say that it is definitely good to use Linux for this purpose. I have found it easiest to set up servers on that operating system. If you would like to know more, read this: https://swiftenterprises.substack.com/p/live-free-with-linux
Thanks for your blog! I'm using scrapped data to build my own GenAI app. This helps a lot!
Happy to know! Thank you Yvett 🙏
Nothing to add, except my compliments for your article. Great read. Top notch.
Thank you so much Demetrio!
Great article, loved it. Thank you for sharing it as free.
🙇♂️ 🙏
Hi Luca,
I recently discovered Wanderio and wanted to commend you on the great work you're doing. With ten years in web scraping, I’d like to share my thoughts.
I agree that LLMs are transforming web scraping, leading to two significant shifts. First, productivity is increasing; teams can manage more scrapers without expanding headcount. Second, AI companies' demand for data is pushing them to scrape in less conventional ways, prompting websites to enhance bot protections, complicating scraping efforts.
To adapt, we require more sophisticated tools, whether self-hosted or third-party. I interviewed Or Lenchner, CEO of Bright Data, for my YouTube channel (videos coming in September). He predicts a rise in "web unblocker" usage due to this growing complexity.
As you noted, data is also available for purchase on marketplaces. With rising complexity, not everyone needs to scrape the same sites for basic data like product prices. Recently, we launched Databoutique.com, a marketplace for web data, allowing users to purchase data easily.
Despite past stigmas, web scraping is gaining momentum due to the AI boom, so long live web scraping!
Hey Pierluigi, thank you for your insightful comment! Databoutique looks awesome, congrats!
This is a great article, I have done some web scraping myself. I would say that it is definitely good to use Linux for this purpose. I have found it easiest to set up servers on that operating system. If you would like to know more, read this: https://swiftenterprises.substack.com/p/live-free-with-linux
Thank you Tom this is useful!