"400 Client Error: Bad Request for url: https://kv.replit.com/..." when using ReplitDB

Support Bug Reports
1

Problem description

I have a data pipeline I have been using for a while without issue. Currently there is a specific issue with ReplitDB when I attempt to store information in the DB. Running in shell, I get this error:

INFO|(Scrape)|Saving to DB...
ERROR|(Scrape)|Error in get_new_matches: 400 Client Error: Bad Request for url: https://kv.replit.com/v0/eyJhbGciOiJIUzUxMiIsImlzcyI6ImNvbm1hbiIsImtpZCI6InByb2Q6MSIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJjb25tYW4iLCJleHAiOjE3MTA0NzE4NjUsImlhdCI6MTcxMDM2MDI2NSwiZGF0YWJhc2VfaWQiOiI2ZjdiZGIyYy1iZDFhLTQ5YTUtODc4Mi0yNGYyNjI4ZTk5ZDIiLCJ1c2VyIjoic29odW1rYWppIiwic2x1ZyI6IkZvb3R5U3RhdHMtMiJ9.2ilbCRcgPQI-bKNG3k0Ddy7-fnfrW7SGZTTI96O1xClCbo-cqZN_qwIdWUPJ2iO81AatyusWodYMlz7CW_HlxA

I found a related bug report but no solution:

I did some of my own testing below.

Expected behavior

Store information in ReplitDB as I have been for months. Here is the specific code that has an error:

    logging.info('Saving to DB...')
    db['table'][key1] = json.dumps(vars(x))          #: Convert to json and store in db
    logging.info('Saved!')

x is an object that is converted to dictionary and then json and stored in db. All of the lead up steps to ā€œstore in dbā€ work.

Actual behavior

I have a data pipeline I have been using for a while without issue. Currently there is a specific issue with ReplitDB when I attempt to store information in the DB. Running in shell, I get this error:

INFO|(Scrape)|Saving to DB...
ERROR|(Scrape)|Error in get_new_matches: 400 Client Error: Bad Request for url: <replit link>

Attempting to manually test the issue, I tried to copy the value of a different key (that had successfully been populated by the pipeline earlier today) to the key ā€˜tempā€™

>>> len(db['table']['key1'])
6437
>>> type(db['table']['key1'])
<class 'str'>
>>> 'temp' in db['table']
False
>>> db['table']['temp'] = db['table']['key1']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 346, in __setitem__
    self.on_mutate()
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 330, in on_mutate
    self._on_mutate_handler(self.value)
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 387, in cb
    db[k] = val
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 514, in __setitem__
    self.set(key, value)
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 523, in set
    self.set_raw(key, _dumps(value))
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 532, in set_raw
    self.set_bulk_raw({key: value})
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 550, in set_bulk_raw
    r.raise_for_status()
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <replit link>

Lastly I tried to see if making the string smaller worked! It did! (but not consistently)

>>> db['table']['temp'] = db['table']['key1'][0:1000]
>>> len(db['table']['temp'])
1000

In view of this, I tried to minimize the string size and rerun my pipeline - it failed with the same error as at first.

Steps to reproduce

For me, run my data pipeline or try to store something ā€œtoo largeā€ in db. I had this error in the past but more briefly and it resolved itself. Currently, I have had it all day.

Browser

Chrome

OS

Mac OS

Device if mobile

N/A

Plan

Core membership

2

Iā€™m not sure if this is the cause, but Replit DB can only store up to 50 MB (IIRC?), and if you are storing too much info the request might fail.

2 Likes
3

And 5k key-value pairs, and some other restrictions:

4

Yup! Forgot to mention, I checked that: I am at 8/50 MiB|11 keys according to the DB tab.

5

Wrote back to the other commenter:

ā€œYup! Forgot to mention, I checked that: I am at 8/50 MiB|11 keys according to the DB tab.ā€

Not sure about bytes per key or MiB per value - either way this seems to be a 400 error. If it is bytes per key or MiB per value - I should be able to create a new key/split up the values to make it work. Will try that.

I think the issue is with the auto-generated urls for replit db and this part of the long error message:

  db[k] = val
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 514, in __setitem__
    self.set(key, value)
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 523, in set
    self.set_raw(key, _dumps(value))
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 532, in set_raw
    self.set_bulk_raw({key: value})
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/replit/database/database.py", line 550, in set_bulk_raw
    r.raise_for_status()
  File "/home/runner/FootyStats-2/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
6

I donā€™t think so, given all DB urls are about that long, and last I checked mine worked fine.

Per-key should be just 1 byte per one character, same for values. So if you have a key over 1024 characters, or a value overā€¦ 5,242,880 characters, I think it would fail.

8 MiB for 11 keys seems like you might have a few, big keys that are causing the issue.

1 Like