Browser closed unexpectedly

Question:
I’m trying to web scrap a JS site using:

from requests_html import HTMLSession

On my main computer this code works, but when I try to host it on replit it doesn’t work, why?

I think something can be wrong with chromium path. ChatGPT didn’t help to resolve this problem.

I want an universal solution, because in future I want to host it anywhere.
I post below the simplest html_requests code.

Error:
pyppeteer.errors.BrowserError: Browser closed unexpectedly
Repl link:
https://replit.com/@DesTuuuu/chromium#main.py

from requests_html import HTMLSession

url = 'https://konwenty-poludniowe.pl/konwenty/kalendarz'
session = HTMLSession()
response = session.get(url)

response.html.render()

print(response.html.html)

This happens because pyppeteer (which is used by requests_html) won’t install all the required things for Chromium. Unfortunately, a universal solution isn’t possible because this hugely depends on the environment, and Replit (unfortunately) suffers from limitations when it comes to running headless browsers like Chromium (as an environment with its own restrictions), which is required by both pyppeteer and requests_html for advanced rendering.

Did you tried to use selenium with a headless browser setup?

not yet, i will try it later :slight_smile:

2 Likes