Logging Methods

Code Help Python
1

Question:

For a Python Flask project, I record website requests and other pertinent details to my log in this manner:

def add_to_log(text):
  with open('logs/log.txt', 'a') as f:
    f.write(get_timestamp() + ' ' + text + '\n')

Also, I retrieve the log and review it using something like this:

def get_log():
  with open('logs/log.txt', 'r') as f:
    lines = f.read().splitlines()[::-1]
  return '<br>'.join(lines[0:500])

I understand as we transition to Deployments, we will not have a persistent filesystem, so I need to learn a new way to record and access logs. I have heard that one should use ‘postgres’ or ‘replit db’ advised in the forums, but I am not an experienced programmer and have no experience or familiarity with these concepts.

What does this mean? And very practically, how I would implement this in a Replit app with Python?

Thank you!

2

You can simply use Replit DB as a Python dictionary. Note that keys and values cannot have slashes (/) and need to be JSON serializable.

Here’s a Python Flask example app that uses Replit DB to keep logs:

from flask import Flask, request
from replit import db

app = Flask(__name__)
# create the logs list if it doesn't already exist
db['logs'] = db.get('logs', [])


@app.route('/')
def index():
    return '''<form method="post" action="/post">
    <input type="text" name="input" />
</form>'''


@app.route('/post')
def post():
    # do something...
    # and log it
    db['logs'].append(request.form['input'])


@app.route('/logs')
def logs():
    # I would use jinja for this, but since this app is only using one file
    page = str(['<p>' + log + '</p>' for log in db['logs']])
    return page


if __name__ == '__main__':
    app.run('0.0.0.0')
2 Likes
3

Thank you. A few followup questions:

Essentially, when we say from replit import db, db[] is a persistent dictionary whose content will not change between runs?

They single key you created, ‘logs’, then holds a list. I can think of this as the equivalent of my former logs.txt, but now a list of strings where new entries are appended() ?

Do I need to use threading locks before writing to this dictionary to avoid concurrent writes?

What does JSON serializable mean?

2 Likes
5

Yes, I believe that is correct

My guess is that it means converting JSON (aka a dictionary) to a string. Serializing it into a string format. I’m not 100% on that though

2 Likes
6

Also, if the deployed app appends items to db[‘log’], will the changes be seen when I run the non-deployed version of the app from the Replit interface (i.e. just click the Run button)? They see the same db?

1 Like
7

It should, since the database itself is essentially a JWT-encoded URL containing data such as the REPL ID, your username, timestamp, etc., all encoded with a secret key. Essentially, this means that your deployed and non-deployed applications should be using the same database, and if they aren’t, I don’t think that is normal at all.

1 Like
8

When I tried the following code:

from replit import db

I received the following error:

> ./start.sh
Traceback (most recent call last):
  File "/home/runner/LL/src/main.py", line 4, in <module>
    from shared import add_to_log
  File "/home/runner/LL/src/shared.py", line 12, in <module>
    from replit import db
ModuleNotFoundError: No module named 'replit'
exit status 1

This Python Flask app is in a NixOS Replit, which might be the issue. I also tried the code in a new Python Replit, and it worked. Perhaps I need to explicitly configure NixOS for the Replit database?

1 Like
9 
pip install replit
1 Like
10

Thank you. I typed that in and it produced an error, here is the output:

~/LL$ pip install replit
Collecting replit
  Using cached replit-3.3.2-py3-none-any.whl (34 kB)
Collecting typing_extensions<5,>=4
  Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Requirement already satisfied: Flask<3.0.0,>=2.0.0 in /nix/store/66i18zqy7yhh7vd1xlgv3527ndn074sn-python3.9-Flask-2.0.2/lib/python3.9/site-packages (from replit) (2.0.2)
Requirement already satisfied: Werkzeug<3.0.0,>=2.0.0 in /nix/store/1gnl7sqrxs3q7sn9yp8zp8ix8kfvzb7d-python3.9-werkzeug-2.0.1/lib/python3.9/site-packages (from replit) (2.0.1)
Collecting urllib3==1.26.15
  Using cached urllib3-1.26.15-py2.py3-none-any.whl
Requirement already satisfied: requests<3.0.0,>=2.25.1 in /nix/store/dg7lp42zcj8a73kbsa91ndaf5ngm4j28-python3.9-requests-2.26.0/lib/python3.9/site-packages (from replit) (2.26.0)
Collecting pyseto<2.0.0,>=1.6.11
  Using cached pyseto-1.7.4-py3-none-any.whl (62 kB)
Collecting protobuf<5.0.0,>=4.21.8
  Using cached protobuf-4.24.3-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
Collecting aiohttp-retry<3.0.0,>=2.8.3
  Using cached aiohttp_retry-2.8.3-py3-none-any.whl
Collecting aiohttp<4.0.0,>=3.6.2
  Using cached aiohttp-3.8.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.4.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (228 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl
Collecting attrs>=17.3.0
  Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /nix/store/fzq8nm7dwink4a5326yl8zpy8zkc7dvh-python3.9-async-timeout-4.0.1/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.6.2->replit) (4.0.1)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /nix/store/sfgf0yf2r7d2sfmf2ckjlrirr0sqdr6f-python3.9-charset-normalizer-2.0.5/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.6.2->replit) (2.0.5)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (269 kB)
Requirement already satisfied: click>=7.1.2 in /nix/store/3z6kn6khmrvcchmlaq1kf4dazn7xjfxd-python3.9-click-8.0.3/lib/python3.9/site-packages (from Flask<3.0.0,>=2.0.0->replit) (8.0.3)
Requirement already satisfied: itsdangerous>=2.0 in /nix/store/h2llfysmswkh3swz1mpsp58hpmjv8fvd-python3.9-itsdangerous-2.0.1/lib/python3.9/site-packages (from Flask<3.0.0,>=2.0.0->replit) (2.0.1)
Requirement already satisfied: Jinja2>=3.0 in /nix/store/rjxbnz23hz80gqxqb2p98g7m2749s2x8-python3.9-Jinja2-3.0.2/lib/python3.9/site-packages (from Flask<3.0.0,>=2.0.0->replit) (3.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /nix/store/625pmj7qqxbdh7gwqkmnf7q9nnimck6y-python3.9-markupsafe-2.0.1/lib/python3.9/site-packages (from Jinja2>=3.0->Flask<3.0.0,>=2.0.0->replit) (2.0.1)
Collecting pycryptodomex<4.0.0,>=3.18.0
  Using cached pycryptodomex-3.19.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Collecting passlib[argon2]<2.0.0,>=1.7.4
  Using cached passlib-1.7.4-py2.py3-none-any.whl
Collecting iso8601<3.0.0,>=1.0.2
  Using cached iso8601-2.0.0-py3-none-any.whl (7.5 kB)
Collecting cryptography==41.0.3
  Using cached cryptography-41.0.3-cp37-abi3-manylinux_2_28_x86_64.whl (4.3 MB)
Requirement already satisfied: cffi>=1.12 in /nix/store/py7c0mw8dgz9m00nsliglsbw5ns0f0pf-python3.9-cffi-1.14.6/lib/python3.9/site-packages (from cryptography==41.0.3->pyseto<2.0.0,>=1.6.11->replit) (1.14.6)
Requirement already satisfied: pycparser in /nix/store/0hgwshx1mb9v68rm1jrxclvww6hlqcrw-python3.9-pycparser-2.20/lib/python3.9/site-packages (from cffi>=1.12->cryptography==41.0.3->pyseto<2.0.0,>=1.6.11->replit) (2.20)
Collecting argon2-cffi>=18.2.0
  Using cached argon2_cffi-23.1.0-py3-none-any.whl (15 kB)
Collecting argon2-cffi-bindings
  Using cached argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Requirement already satisfied: idna<4,>=2.5 in /nix/store/1fyn6w022l3mr8qapz2j2pxk4r9qaf46-python3.9-idna-3.2/lib/python3.9/site-packages (from requests<3.0.0,>=2.25.1->replit) (3.2)
Requirement already satisfied: certifi>=2017.4.17 in /nix/store/rkl4hjlw4jn0m2n3dp8x29siimx33z26-python3.9-certifi-2021.10.08/lib/python3.9/site-packages (from requests<3.0.0,>=2.25.1->replit) (2021.10.8)
Installing collected packages: typing-extensions, multidict, frozenlist, argon2-cffi-bindings, yarl, passlib, attrs, argon2-cffi, aiosignal, urllib3, pycryptodomex, iso8601, cryptography, aiohttp, pyseto, protobuf, aiohttp-retry, replit
  Attempting uninstall: typing-extensions
    Found existing installation: typing-extensions 3.10.0.2
    Uninstalling typing-extensions-3.10.0.2:
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'METADATA'
Consider using the `--user` option or check the permissions.

WARNING: You are using pip version 21.1.3; however, version 23.2.1 is available.
You should consider upgrading via the '/nix/store/cgxc3jz7idrb1wnb2lard9rvcx6aw2si-python3-3.9.6/bin/python3.9 -m pip install --upgrade pip' command.
1 Like
11 
pip install --user replit
12

That line executes from the command line without error. However, in my code, I still receive this error:

> ./start.sh
Traceback (most recent call last):
  File "/home/runner/LL/src/main.py", line 4, in <module>
    from shared import add_to_log
  File "/home/runner/LL/src/shared.py", line 12, in <module>
    from replit import db
ModuleNotFoundError: No module named 'replit'
exit status 1
>
13

Try poetry add replit just to be sure, but I suspect this is likely because of a recent issue todo with Repls’ file systems becoming read only. You can get up-to-date information on this issue on the status page.

2 Likes
14

Thank you. I tried poetry add replit and received an error:

~/LL$ poetry add replit

  RuntimeError

  Poetry could not find a pyproject.toml file in /home/runner/LL or its parents

  at /nix/store/w4kbm54rn59xfz9xx0ah52m1dwb72g4k-python3.9-poetry-core-1.0.7/lib/python3.9/site-packages/poetry/core/factory.py:369 in locate
      365│             if poetry_file.exists():
      366│                 return poetry_file
      367│ 
      368│         else:
    → 369│             raise RuntimeError(
      370│                 "Poetry could not find a pyproject.toml file in {} or its parents".format(
      371│                     cwd
      372│                 )
      373│             )
15

Do you see a pyproject.toml file in the packages section in the file hierachy (you may have to show hidden files to see this, click the three dots at the top of the file hierarchy and click show hidden files)? If you do, then it is because of the issue I mentioned above.

16

I clicked on ‘show hidden files’ and the two file named in the Config files section are:

.replit
replit.nix

There is no pyproject.toml.

17

Ok, that’s just because poetry is not set up in your Repl.

Do you see a folder name something like .pythonlibs? If so, try running the below in your shell:

if [ -d "/home/runner/$REPL_SLUG/.pythonlibs/lib/python3.10/site-packages/replit" ]; then echo "directory exists"; else echo "directory not found"; fi

If you see directory exists, then you know the package is installed and it’s because of the issue I mentioned above, if not, it’s something else.

18

It says ‘directory not found’. Do I need to create that?

19

Interesting, either the package hasn’t been installed or it’s using the older Python file system, is there a venv folder? Actualy, any chance of a screenshot of the file hierarchy or even a link to the Repl if it’s public? Edit: nvm, I found it: https://replit.com/@latinlens/LL?v=1

20

Huh, it seems it runs without error for me (in a fork). Very weird.

LL
LL1911×923 137 KB

21

Thats just because I commented out the line. Go to shared.py, uncomment:

#from replit import db

and let me know what happens.