They are pay per usage and are easy to develop! Puppeteer, TypeScript and NodeJs Logic Apps are great for defining and running workflows and look like a perfect fit. But then on a bright day I figured I would use Azure Logic Apps instead. A good start point for running puppeteer containers in Azure is this blog post.įor orchestrating the scraper I was thinking about using Azure Functions again. I can spin up and tear down the container with some orchestration and thereby limit my costs. So what are the alternatives? Well containers seems like a reasonable solution. My initial idea was to run puppeteer inside an Azure Function, however after some research I came to the conclusion that running a headless browser on Azure PaaS or Serverless is not going to happen. A headless Chrome API build by Google itself, very promising. After some research I stumbled upon `puppeteer`. Some are build as an SPA and that requires per definition a browser based approach. Also because lots of ecommerce sites rely on alot on JavaScript.
Since I want to scrape different ecommerce sites spinning up an actual browser looked like the way to go. Others spin up and entire (headless) browser and perform actual DOM operations. Some packages just perform Http calls and evaluate the response. Web scraping comes in different shapes and sizes. Later on I will have to develop some UI which discloses the information for ecommerce traders. Lastly the output of the scraper has to be stored in a database. I’m scraping ecommerce sites and the pages that need to be scraped depend on a list of id’s comming from a database. This frequency might change in the future so I don’t want to have it build in hard coded. I want to scrape certain websites twice a day. Secondly I only want to pay for actual usage and not for a VM thats idle. I don’t want to pay for a VM and just deploy the scraper on it because I need the solution to be scalable.
Utilizing Serverless and PaaS services is challenging. Xamarinwatchos xamarinwatchos was computed.Building a web scraper is pretty hard. netstandard2.1 netstandard2.1 was computed. Netstandard2.0 netstandard2.0 is compatible. netcoreapp3.1 netcoreapp3.1 was computed. netcoreapp3.0 netcoreapp3.0 was computed. netcoreapp2.2 netcoreapp2.2 was computed. netcoreapp2.1 netcoreapp2.1 was computed. Netcoreapp2.0 netcoreapp2.0 was computed. net7.0-windows net7.0-windows was computed. net7.0-maccatalyst net7.0-maccatalyst was computed. net7.0-android net7.0-android was computed. net6.0-windows net6.0-windows was computed. net6.0-maccatalyst net6.0-maccatalyst was computed. net6.0-android net6.0-android was computed. net5.0-windows net5.0-windows was computed. Versions Compatible and additional computed target framework versions.
0 kommentar(er)
