"400 Client Error: Bad Request for url: https://kv.replit.com/..." when using ReplitDB

Problem description

I have a data pipeline I have been using for a while without issue. Currently there is a specific issue with ReplitDB when I attempt to store information in the DB. Running in shell, I get this error:

INFO|(Scrape)|Saving to DB...
ERROR|(Scrape)|Error in get_new_matches: 400 Client Error: Bad Request for url: https://kv.replit.com/v0/eyJhbGciOiJIUzUxMiIsImlzcyI6ImNvbm1hbiIsImtpZCI6InByb2Q6MSIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJjb25tYW4iLCJleHAiOjE3MTA0NzE4NjUsImlhdCI6MTcxMDM2MDI2NSwiZGF0YWJhc2VfaWQiOiI2ZjdiZGIyYy1iZDFhLTQ5YTUtODc4Mi0yNGYyNjI4ZTk5ZDIiLCJ1c2VyIjoic29odW1rYWppIiwic2x1ZyI6IkZvb3R5U3RhdHMtMiJ9.2ilbCRcgPQI-bKNG3k0Ddy7-fnfrW7SGZTTI96O1xClCbo-cqZN_qwIdWUPJ2iO81AatyusWodYMlz7CW_HlxA

I found a related bug report but no solution:

I did some of my own testing below.

Expected behavior

Store information in ReplitDB as I have been for months. Here is the specific code that has an error:

    logging.info('Saving to DB...')
    db['table'][key1] = json.dumps(vars(x))          #: Convert to json and store in db
    logging.info('Saved!')

x is an object that is converted to dictionary and then json and stored in db. All of the lead up steps to “store in db” work.

Actual behavior

I have a data pipeline I have been using for a while without issue. Currently there is a specific issue with ReplitDB when I attempt to store information in the DB. Running in shell, I get this error:

INFO|(Scrape)|Saving to DB...
ERROR|(Scrape)|Error in get_new_matches: 400 Client Error: Bad Request for url: <replit link>

Attempting to manually test the issue, I tried to copy the value of a different key (that had successfully been populated by the pipeline earlier today) to the key ‘temp’

>>> len(db['table']['key1'])
6437
>>> type(db['table']['key1'])
<class 'str'>
>>> 'temp' in db['table']
False
>>> db['table']['temp'] = db['table']['key1']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 346, in __setitem__
    self.on_mutate()
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 330, in on_mutate
    self._on_mutate_handler(self.value)
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 387, in cb
    db[k] = val
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 514, in __setitem__
    self.set(key, value)
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 523, in set
    self.set_raw(key, _dumps(value))
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 532, in set_raw
    self.set_bulk_raw({key: value})
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 550, in set_bulk_raw
    r.raise_for_status()
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <replit link>

Lastly I tried to see if making the string smaller worked! It did! (but not consistently)

>>> db['table']['temp'] = db['table']['key1'][0:1000]
>>> len(db['table']['temp'])
1000

In view of this, I tried to minimize the string size and rerun my pipeline - it failed with the same error as at first.

Steps to reproduce

For me, run my data pipeline or try to store something “too large” in db. I had this error in the past but more briefly and it resolved itself. Currently, I have had it all day.

Browser

Chrome

OS

Mac OS

Device if mobile

N/A

Plan

Core membership

I’m not sure if this is the cause, but Replit DB can only store up to 50 MB (IIRC?), and if you are storing too much info the request might fail.

2 Likes

And 5k key-value pairs, and some other restrictions:

Yup! Forgot to mention, I checked that: I am at 8/50 MiB|11 keys according to the DB tab.

Wrote back to the other commenter:

“Yup! Forgot to mention, I checked that: I am at 8/50 MiB|11 keys according to the DB tab.”

Not sure about bytes per key or MiB per value - either way this seems to be a 400 error. If it is bytes per key or MiB per value - I should be able to create a new key/split up the values to make it work. Will try that.

I think the issue is with the auto-generated urls for replit db and this part of the long error message:

  db[k] = val
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 514, in __setitem__
    self.set(key, value)
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 523, in set
    self.set_raw(key, _dumps(value))
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 532, in set_raw
    self.set_bulk_raw({key: value})
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 550, in set_bulk_raw
    r.raise_for_status()
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

I don’t think so, given all DB urls are about that long, and last I checked mine worked fine.

Per-key should be just 1 byte per one character, same for values. So if you have a key over 1024 characters, or a value over… 5,242,880 characters, I think it would fail.

8 MiB for 11 keys seems like you might have a few, big keys that are causing the issue.

1 Like