Outage: readonly filesystems, connection failures, and other errors

I’m getting a similar issue: my error is:

nix error: building nix env: problem reading '.replit' file: unable to read .replit: read .replit: input/output error

I ran mount as requested – here’s the output:

 mount
overlay on / type overlay (ro,relatime,lowerdir=/var/lib/docker/10000.10000/overlay2/l/HVRDQWWDBKACOJETHDZMZLZUOV:/var/lib/docker/10000.10000/overlay2/l/B5EKMOUWSNVACBCXRC4PENTDZM:/var/lib/docker/10000.10000/overlay2/l/L3UN3AL67BHDESE7BOH5XC6N52:/var/lib/docker/10000.10000/overlay2/l/EHKFG4TZSJCORQR6Q22BGEZV42:/var/lib/docker/10000.10000/overlay2/l/34EAOQACXDMKNY473GLVEMWNPS:/var/lib/docker/10000.10000/overlay2/l/B6N64SEQ54NPLOQ6TDHB3EHKDE,upperdir=/var/lib/docker/10000.10000/overlay2/b9642cb09d3cdc50c6f3f3fee4cb30ac4d43971c56a1683f38f1f0e8bf76fbc0/diff,workdir=/var/lib/docker/10000.10000/overlay2/b9642cb09d3cdc50c6f3f3fee4cb30ac4d43971c56a1683f38f1f0e8bf76fbc0/work,nouserxattr)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,uid=10000,gid=10000,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=10005,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /io type tmpfs (rw,nosuid,nodev,size=13167192k,nr_inodes=819200,mode=755,inode64)
/dev/nbd17 on /repl type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/root on /nix type ext4 (ro,relatime,discard,errors=remount-ro)
/dev/root on /usr/bin/fusermount type ext4 (rw,relatime,discard,errors=remount-ro)
/dev/mapper/docker on /etc/resolv.conf type ext4 (ro,relatime)
tmpfs on /mnt/ruse type tmpfs (rw,nosuid,nodev,size=13167192k,nr_inodes=819200,mode=755,inode64)
/dev/nbd17 on /mnt/scratch type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/root on /mnt/cacache type ext4 (ro,relatime,discard,errors=remount-ro)
/dev/disk/by-id/google-nixmodules-a79e023-us-west1-a on /etc/nixmodules type ext4 (ro,nosuid,nodev,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
/dev/mapper/docker on /etc/hosts type ext4 (ro,relatime)
/dev/mapper/docker on /etc/hostname type ext4 (ro,relatime)
/dev/disk/by-id/google-nixmodules-a79e023-us-west1-a on /mnt/nixmodules type ext4 (ro,nosuid,nodev,relatime)
/dev/root on /usr/bin/fusermount type ext4 (ro,relatime,discard,errors=remount-ro)
/dev/disk/by-id/google-cacache-1693253768289-us-west1-a on /mnt/cacache/pip type ext4 (ro,nosuid,nodev,relatime)
/dev/disk/by-id/google-cacache-1693253768289-us-west1-a on /mnt/cacache/nix type ext4 (ro,nosuid,nodev,relatime)
tmpfs on /etc/ssl/certs/ca-certificates.crt type tmpfs (ro,nosuid,nodev,size=13167192k,nr_inodes=819200,mode=755,inode64)
devtmpfs on /dev/null type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /dev/random type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /dev/full type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /dev/tty type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /dev/zero type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /dev/urandom type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,uid=10000,gid=10000,inode64)
devtmpfs on /proc/kcore type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /proc/keys type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
devtmpfs on /proc/timer_list type devtmpfs (rw,nosuid,noexec,relatime,size=32913792k,nr_inodes=8228448,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,uid=10000,gid=10000,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,uid=10000,gid=10000,inode64)
/dev/nbd17 on /tmp type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/nbd17 on /run type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/nbd17 on /run/replit/socks type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/root on /run/wrappers/bin/fusermount type ext4 (ro,relatime,discard,errors=remount-ro)
overlay on /home/runner type overlay (rw,relatime,lowerdir=/home/runner,upperdir=/mnt/scratch/home/upper,workdir=/mnt/scratch/home/work,xino=off,volatile,nouserxattr)
overlay on /home/runner/.cargo/registry type overlay (rw,relatime,lowerdir=/mnt/cacache/cratesio-registry,upperdir=/mnt/scratch/cacache_cratesio_registry/upper,workdir=/mnt/scratch/cacache_cratesio_registry/work,volatile,nouserxattr)
overlay on /home/runner/.m2/repository type overlay (rw,relatime,lowerdir=/mnt/cacache/mvn,upperdir=/mnt/scratch/cacache_mvn/upper,workdir=/mnt/scratch/cacache_mvn/work,volatile,nouserxattr)
overlay on /home/runner/.npm type overlay (rw,relatime,lowerdir=/mnt/cacache/npm,upperdir=/mnt/scratch/cacache_npm/upper,workdir=/mnt/scratch/cacache_npm/work,volatile,nouserxattr)
overlay on /home/runner/.cache/pip type overlay (rw,relatime,lowerdir=/mnt/cacache/pip,upperdir=/mnt/scratch/cacache_pip/upper,workdir=/mnt/scratch/cacache_pip/work,volatile,nouserxattr)
overlay on /nix type overlay (rw,relatime,lowerdir=/mnt/cacache/nix:/mnt/nixmodules/nix:/nix,upperdir=/mnt/scratch/nix/upper,workdir=/mnt/scratch/nix/work,volatile,nouserxattr)
overlay on /store type overlay (rw,relatime,lowerdir=/mnt/cacache/nix:/mnt/nixmodules/nix:/nix,upperdir=/mnt/scratch/nix/upper,workdir=/mnt/scratch/nix/work,volatile,nouserxattr)
/dev/nbd45 on /home/runner/Day69100Days type btrfs (rw,noatime,ssd,flushoncommit,discard=async,space_cache=v2,commit=60,subvolid=257,subvol=/working_subv)
1 Like

i think anyone in repl has the same problem but not all repls effected

1 Like

my mount…

 mount
overlay on / type overlay (ro,relatime,lowerdir=/var/lib/docker/10000.10000/overlay2/l/XC44NUWRYOOGKQXMQO7H4CEWL4:/var/lib/docker/10000.10000/overlay2/l/B5EKMOUWSNVACBCXRC4PENTDZM:/var/lib/docker/10000.10000/overlay2/l/L3UN3AL67BHDESE7BOH5XC6N52:/var/lib/docker/10000.10000/overlay2/l/EHKFG4TZSJCORQR6Q22BGEZV42:/var/lib/docker/10000.10000/overlay2/l/34EAOQACXDMKNY473GLVEMWNPS:/var/lib/docker/10000.10000/overlay2/l/B6N64SEQ54NPLOQ6TDHB3EHKDE,upperdir=/var/lib/docker/10000.10000/overlay2/23ce1116395935151abaa151d1b6f8bace09e712091fe8c78c93054d9dd36113/diff,workdir=/var/lib/docker/10000.10000/overlay2/23ce1116395935151abaa151d1b6f8bace09e712091fe8c78c93054d9dd36113/work,nouserxattr)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,uid=10000,gid=10000,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=10005,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/nbd71 on /repl type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/root on /nix type ext4 (ro,relatime,discard,errors=remount-ro)
tmpfs on /io type tmpfs (rw,nosuid,nodev,size=13166780k,nr_inodes=819200,mode=755,inode64)
tmpfs on /mnt/ruse type tmpfs (rw,nosuid,nodev,size=13166780k,nr_inodes=819200,mode=755,inode64)
/dev/mapper/docker on /etc/resolv.conf type ext4 (ro,relatime)
/dev/nbd71 on /mnt/scratch type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/root on /mnt/cacache type ext4 (ro,relatime,discard,errors=remount-ro)
/dev/disk/by-id/google-nixmodules-a79e023-us-central1-f on /mnt/nixmodules type ext4 (ro,nosuid,nodev,relatime)
/dev/root on /usr/bin/fusermount type ext4 (rw,relatime,discard,errors=remount-ro)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
/dev/mapper/docker on /etc/hosts type ext4 (ro,relatime)
/dev/mapper/docker on /etc/hostname type ext4 (ro,relatime)
/dev/disk/by-id/google-nixmodules-a79e023-us-central1-f on /etc/nixmodules type ext4 (ro,nosuid,nodev,relatime)
/dev/disk/by-id/google-cacache-1693253768289-us-central1-f on /mnt/cacache/nix type ext4 (ro,nosuid,nodev,relatime)
/dev/disk/by-id/google-cacache-1693253768289-us-central1-f on /mnt/cacache/pip type ext4 (ro,nosuid,nodev,relatime)
/dev/root on /usr/bin/fusermount type ext4 (ro,relatime,discard,errors=remount-ro)
tmpfs on /etc/ssl/certs/ca-certificates.crt type tmpfs (ro,nosuid,nodev,size=13166780k,nr_inodes=819200,mode=755,inode64)
devtmpfs on /dev/null type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /dev/random type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /dev/full type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /dev/tty type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /dev/zero type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /dev/urandom type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime,uid=10000,gid=10000,inode64)
devtmpfs on /proc/kcore type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /proc/keys type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
devtmpfs on /proc/timer_list type devtmpfs (rw,nosuid,noexec,relatime,size=32912764k,nr_inodes=8228191,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,uid=10000,gid=10000,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,uid=10000,gid=10000,inode64)
/dev/nbd71 on /tmp type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/nbd71 on /run type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/nbd71 on /run/replit/socks type btrfs (rw,noatime,nobarrier,ssd,space_cache=v2,subvolid=256,subvol=/working_subv)
/dev/root on /run/wrappers/bin/fusermount type ext4 (ro,relatime,discard,errors=remount-ro)
overlay on /home/runner type overlay (rw,relatime,lowerdir=/home/runner,upperdir=/mnt/scratch/home/upper,workdir=/mnt/scratch/home/work,xino=off,volatile,nouserxattr)
overlay on /home/runner/.cargo/registry type overlay (rw,relatime,lowerdir=/mnt/cacache/cratesio-registry,upperdir=/mnt/scratch/cacache_cratesio_registry/upper,workdir=/mnt/scratch/cacache_cratesio_registry/work,volatile,nouserxattr)
overlay on /home/runner/.m2/repository type overlay (rw,relatime,lowerdir=/mnt/cacache/mvn,upperdir=/mnt/scratch/cacache_mvn/upper,workdir=/mnt/scratch/cacache_mvn/work,volatile,nouserxattr)
overlay on /home/runner/.npm type overlay (rw,relatime,lowerdir=/mnt/cacache/npm,upperdir=/mnt/scratch/cacache_npm/upper,workdir=/mnt/scratch/cacache_npm/work,volatile,nouserxattr)
overlay on /home/runner/.cache/pip type overlay (rw,relatime,lowerdir=/mnt/cacache/pip,upperdir=/mnt/scratch/cacache_pip/upper,workdir=/mnt/scratch/cacache_pip/work,volatile,nouserxattr)
overlay on /nix type overlay (rw,relatime,lowerdir=/mnt/cacache/nix:/mnt/nixmodules/nix:/nix,upperdir=/mnt/scratch/nix/upper,workdir=/mnt/scratch/nix/work,volatile,nouserxattr)
overlay on /store type overlay (rw,relatime,lowerdir=/mnt/cacache/nix:/mnt/nixmodules/nix:/nix,upperdir=/mnt/scratch/nix/upper,workdir=/mnt/scratch/nix/work,volatile,nouserxattr)
/dev/nbd12 on /home/runner/Apesss type btrfs (ro,noatime,ssd,flushoncommit,discard=async,space_cache=v2,commit=60,subvolid=257,subvol=/working_subv)
1 Like

2 Likes

Try running kill 1 in your shell (your repl’s storage device is messed up somehow, so Linux automatically remounted it as read-only)

2 Likes

So from what I can gather, today replit has been having a ton of server problems causing IO errors and repls not loading. Especially, the .replit file is not readable.
The repls experiencing the problem are all repls (because .replit file), though the specific file corrupted seems arbitrary. Some older repls are affected.
In the files tab, there’ll be an error message in place of the file or folder contents. A while ago I looked into this problem for the site-packages folder, I couldn’t even delete or rename it.
No known solutions for users.
Related topics (some closed):


Input/output error (+ModuleNotFoundError)


Repl not loading

2 Likes

@Firepup650 These might be a bunch of potential duplicates

1 Like

done but doesn’t work

1 Like

Does the repl fail to start now?

1 Like

Most of those are closed into this, I’ll review the rest.

2 Likes

We are having the same issue. Some of our team repls are affected some are not. Please help.

2 Likes

Just checked, they seem to be the same bug. Closed and linked to here.

1 Like

we are in the same boat

1 Like

Seriously, Replit has become a quagmire that must be gotten out of

4 Likes

the heart broken is that i can’t also download it so i can’t run :smiley: that’s sad

2 Likes

so we just have to wait?

1 Like

it doesnt affect my repl that isnt nix based

2 Likes

Downloading repls requires a working repl (the program used to download repls actually runs inside of the repl you download), so it won’t work, unfortunately.

2 Likes

The Replit team has been notified of this issue, hopefully they’ll be able to resolve it quickly :smile:

4 Likes

100% agree.

For the discussion, can I assume that my issues are related to what all these others are experiencing?

Left to go eat dinner, came back, hit Run to test out what I had done before stepping away, and now I get this:

image

2 Likes