How Do I Get Past Loading Screen on Web Scraper?

I’m using Python to web scrape Spotify, but when I submit the login form, there is a brief loading animation on the user’s side. As I web scrape Spotify, I need to login but cannot due to the loading screen. How can I get around this?

Here is my code:

import requests
import os
from bs4 import BeautifulSoup

spotify = BeautifulSoup(
    requests.get("https://accounts.spotify.com/en/login").content, features="html5lib"
)

response = requests.post(
    "https://accounts.spotify.com/en/login",
    data={
        "login-username": os.environ["SPOTIFY_USERNAME"],
        "login-password": os.environ["SPOTIFY_PASSWORD"],
    },
)

print(response)

What are you trying to scrape.

@Sky I want to put the login info into this:
https://accounts.spotify.com/en/login

I want to scrape this:

1 Like

Yes but what exactly are you trying to scrape from that page.

@Sky Well, I want to scrape data such as music info and profile info, but that’s not what I need help on. It’s that I cannot login because their is a loading screen and I can’t get around it, because the requests library will only pull the HTML of the loading animation.

1 Like

That’s because you’re sending the wrong request or wrong JSON data.

@Sky How would I send the right JSON data?

I can probably create a mini library for this.

@Sky That’d be awesome, if possible. How long would it take you?

It depends. Do you want to obtain profile information and music-related details? When you mention “music information”, are you referring to searching for a song or something else?

@Sky Here are some features I need:

  • Sign In/Log out
  • Search (with filters)
  • View the logged in user’s information
  • View a artist/song/album information
  • Some way to play music selected by the user

I basically want a way to make a music portal via Spotify in Python.

1 Like

I can do the first four options, but I’m not sure about the fifth one. I mean, I could do it, but at that point, I might as well create an entire Spotify wrapper for Python.

That’s pretty much what I need (no easy task) :laughing:

@Sky Do you think that you can do that? If so, how long may it take? It’s okay if you can’t.

Do you have a Discord by any chance?

@Sky No, sorry :frowning:
Any reason why?

1 Like

It’s easier for me to contact you.

1 Like

Yeah, sorry ‘bout that.

It’s fine, not everyone uses Discord. Anyway, I will work on features like signing in, logging out, searching, viewing user information, and finding songs, artists, or albums right now. It might take me some time, but I’m on it.

Why don’t you just use Spotify’s official API?

There’s also a python wrapper

2 Likes

Yeah, just use the public API instead. I remember that Spotify uses reCAPTCHA, and you would need a solver for that, so using their public API is a better option.