Issue with migrate.py preventing me from running 4cat or accessing web interface

[robbydigital]

Hello, thanks for making this tool available. I'd be grateful for any tips: I'm getting an 'EOFError: EOF when reading a line' message when I run docker-compose up. I'm using Windows 10 Home. I initially tried to install 4cat manually to scrape 4chan, but I couldn't get it to work so I uninstalled and then tried to install through Docker.

I'm using Windows Powershell to run the command because when I run docker-compose up in Ubuntu 20.04 LTS I'm getting this message:

'The command 'docker-compose' could not be found in this WSL 2 distro.
We recommend to activate the WSL integration in Docker Desktop settings.

See https://docs.docker.com/desktop/windows/wsl/ for details.'

The WSL integration is activated in Docker Desktop settings by default. Could it be because I didn't bind-mount the folder I'm storing 4cat in to the Linux file system? I skipped that step and just stored 4cat in /c/users/myusername/ on Windows.

This is the message I get when I run docker-compose up command from Powershell:

PS C:\users\myusername\4cat> docker-compose up
[+] Running 2/2

• Container cat_db_1 Running 0.0s
• Container api Recreated 0.7s
  Attaching to api, db_1
  api | Waiting for postgres...
  api | PostgreSQL started
  api | 1
  api | Seed present
  api | Starting app
  api | Running migrations
  api |
  api | 4CAT migration agent
  api | ------------------------------------------
  api | Current 4CAT version: 1.9
  api | Checked out version: 1.16
  api | The following migration scripts will be run:
  api | migrate-1.9-1.10.py
  api | migrate-1.10-1.11.py
  api | migrate-1.11-1.12.py
  api | migrate-1.12-1.13.py
  api | migrate-1.13-1.14.py
  api | migrate-1.14-1.15.py
  api | WARNING: Migration can take quite a while. 4CAT will not be available during migration.
  api | If 4CAT is still running, it will be shut down now.
  api | Do you want to continue [y/n]? Traceback (most recent call last):
  api | File "helper-scripts/migrate.py", line 142, in
  api | if not args.yes and input("").lower() != "y":
  api | EOFError: EOF when reading a line
  api exited with code 1

[dale-wahl]

Hey robbydigital, my guess is that you are using an old version of 4CAT. That line in helper-scripts/migrate.py was changed in May with this commit.

Try cloning the repo again or doing a git pull to update your repository and then running docker-compose up --build.

You also should note that the docker version of 4CAT does not fully support 4chan out of the box (you can see a breakdown here of what is supported where). It is possible to install Sphinx in a docker container, but that is still in development so you may wish to install directly. That link should provide you with additional help in doing so.

[robbydigital]

ah, thanks, I somehow installed v1.17. I've now updated to 1.21 but embarrassingly I'm now having a problem running docker-compose up. I get this error message:

• frontend Error 17.4s
• db Error 17.4s
  Error response from daemon: pull access denied for 4cat, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

This seems like a simple one but I'm still confused .

[dale-wahl]

Is there additional information to those error messages? It's fine and helpful to copy the output in your posts. 4CAT wouldn't be asking for a docker login so I am guessing the issue is with Docker, but I can't really tell from what you provided.

I would run docker-compose up --build which tells docker to rebuild the image from the current information. It may be trying to run it from the older version.

[robbydigital]

There's not much more information, unfortunately. Here is the command and the full output:

'PS C:\users\myusername\4cat> docker-compose up --build
[+] Running 0/2

• db Error 17.9s
• frontend Error 17.9s
  Error response from daemon: pull access denied for 4cat, repository does not exist or may require 'docker login': denied: requested access to the resource is denied'

I checked Docker again and there was a message saying no containers running, so it gave me a command to run. I did that and then re-ran docker-compose up --build but got the same error message.

I ran the Docker command in Ubuntu first and got this:

'~$docker run -d -p 80:80 docker/getting-started
Unable to find image 'docker/getting-started:latest' locally
latest: Pulling from docker/getting-started
Digest: sha256:10555bb0c50e13fc4dd965ddb5f00e948ffa53c13ff15dcdc85b7ab65e1f240b
Status: Image is up to date for docker/getting-started:latest
3a423ca481423b7d8c645ff3e358e948d52ddfd124f4c947b6ef27a82a5e8aa6
docker: Error response from daemon: driver failed programming external connectivity on endpoint focused_lalande (3a856375e49f0b11b3ff386ad557e84d663538738c66ff05e1e44eeac3f491d1): Bind for 0.0.0.0:80 failed: port is already allocated.'

So I tried it in Powershell and got something slightly different. It does seem to have opened a charming_hellman container. It's created focused_lalande container but it doesn't seem to be working. This is the output from the command on Powershell:

'PS C:\users\myusername\4cat> docker run -d -p 80:80 docker/getting-started
Unable to find image 'docker/getting-started:latest' locally
latest: Pulling from docker/getting-started
540db60ca938: Pull complete
0ae30075c5da: Pull complete
9da81141e74e: Pull complete
b2e41dd2ded0: Pull complete
7f40e809fb2d: Pull complete
758848c48411: Pull complete
23ded5c3e3fe: Pull complete
38a847d4d941: Pull complete
Digest: sha256:10555bb0c50e13fc4dd965ddb5f00e948ffa53c13ff15dcdc85b7ab65e1f240b
Status: Downloaded newer image for docker/getting-started:latest
c90e400d10ba2d99c3d460cc12e8522739cdabea97097ab44cfcaf2e4d71ed8b'

[dale-wahl]

Is C:\users\myusername\4cat\ where you unpacked all the 4cat files? Specifically the same folder as the docker-compose.yml file is located? You have to run the docker commands from inside the same folder as the docker-compose.yml file.

If you run dir, you should see a list of lots of files like so:

09/28/2021  05:52 PM    <DIR>          .
09/28/2021  05:52 PM    <DIR>          ..
09/28/2021  05:51 PM               244 .env
09/28/2021  05:51 PM                89 .gitattributes
09/28/2021  05:51 PM    <DIR>          .github
09/28/2021  05:51 PM               923 .gitignore
09/28/2021  05:51 PM             1,064 .zenodo.json
09/28/2021  05:51 PM             8,294 4cat-daemon.py
09/28/2021  05:51 PM    <DIR>          backend
09/28/2021  05:51 PM    <DIR>          common
09/28/2021  05:51 PM             7,076 config.py-example
09/28/2021  05:51 PM    <DIR>          datasources
09/28/2021  05:51 PM    <DIR>          docker
09/28/2021  05:51 PM             1,168 docker-compose.yml
09/28/2021  05:51 PM    <DIR>          helper-scripts
09/28/2021  05:51 PM            17,179 LICENSE
09/28/2021  05:51 PM            42,589 LICENSE-3DPARTY
09/28/2021  05:52 PM    <DIR>          processors
09/28/2021  05:51 PM             4,760 README.md
09/28/2021  05:51 PM                49 requirements.txt
09/28/2021  05:52 PM    <DIR>          sessions
09/28/2021  05:51 PM             1,756 setup.py
09/28/2021  05:51 PM               416 VERSION
09/28/2021  05:52 PM    <DIR>          webtool
              13 File(s)         85,607 bytes

For some reason, when you are running docker-compose up --build, it is trying to download (pull) 4cat from docker. It says it does not exist there and that is true. But it should already be downloaded onto your computer. Did you install with git? By running something like git clone git@github.com:digitalmethodsinitiative/4cat.git?

[robbydigital]

I downloaded the source code from the list of releases here and unzipped it into C:\users\myusername\4cat. Should I try installing with git instead?

I ran dir and this is the output - it does seem like I'm missing some things:

Directory: C:\users\myusername\4cat

Mode LastWriteTime Length Name

---

d----- 28/09/2021 14:58 .github
d----- 28/09/2021 14:58 backend
d----- 28/09/2021 14:58 common
d----- 28/09/2021 14:58 datasources
d----- 28/09/2021 14:58 docker
d----- 28/09/2021 14:58 helper-scripts
d----- 28/09/2021 14:58 processors
d----- 28/09/2021 14:58 sessions
d----- 28/09/2021 14:59 webtool
-a---- 28/09/2021 14:58 244 .env
-a---- 28/09/2021 14:58 89 .gitattributes
-a---- 28/09/2021 14:58 923 .gitignore
-a---- 28/09/2021 14:58 1064 .zenodo.json
-a---- 28/09/2021 14:58 8294 4cat-daemon.py
-a---- 28/09/2021 14:58 7076 config.py-example
-a---- 28/09/2021 14:58 1168 docker-compose.yml
-a---- 28/09/2021 14:58 17179 LICENSE
-a---- 28/09/2021 14:58 42589 LICENSE-3DPARTY
-a---- 28/09/2021 14:58 4760 README.md
-a---- 28/09/2021 14:58 49 requirements.txt
-a---- 28/09/2021 14:58 1756 setup.py
-a---- 28/09/2021 14:58 416 VERSION

[robbydigital]

Just a quick follow up - I tried running git clone git@github.com:digitalmethodsinitiative/4cat.git and got this error:

C:\users\myusername\4cat> git clone git@github.com:digitalmethodsinitiative/4cat.git
Cloning into '4cat'...
The authenticity of host 'github.com (140.82.121.3)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (RSA) to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[dale-wahl]

That last one means you haven’t set up your GitHub account to use ssh keys.

I haven’t been able to reproduce your error. Tomorrow I’ll try to use the direct download/release that you used and see if I encounter anything. I suspect it has something to do with how Docker is set up on your machine but do not have any idea what could be causing it yet.

[dale-wahl]

Unfortunately, when downloading the release and using docker-compose on my Windows 10 machine I did not run into any errors so I still cannot truly troubleshoot your issue. You could perhaps attempt to force docker to recreate the images using docker-compose up --force-recreate, but I am not really sure why Docker is trying to pull a non-existent image.

I am using Docker engine v20.10.8. When searching for similar errors to yours, I did find that docker-compose does not always respect local images (and instead tries to pull/download them from their registry like you are currently experiencing). You could also try checking your Docker engine and using v20.10.8 or installing directly per our instructions.

[robbydigital]

Thanks so much for all your help with this. Unfortunately I get the same error message with docker-compose up --force-recreate. I am using Docker engine v20.10.8. I found people discussing a similar issue here but I'm not sure the workarounds suggested apply in this case. I originally tried installing directly but ran into several issues so decided to try with Docker. It's probably something I've messed up along the way. Think it's worth uninstalling everything and starting over?

[dale-wahl]

I think I have learned that we need to look at deploying a version on 4cat onto Docker itself, so thank you for that.

Three things you can try:

First

I saw that same link from the docker-compose group. You could try the docker-compose up --ignore-pull-failures flag like they mention. It should ignore that error, but I do not know what will happen.

Second

I am curious if changing your docker-compose.yml file like so will help.

1. delete image: 4cat from the backend section
2. delete image: 4cat from the frontend section AND replace it with:

    build:
      context: .
      dockerfile: docker/Dockerfile

3. run docker-compose up --build

That has been tested and works. It will take longer because it builds the same image twice, but without the image: 4cat I don't think Docker can get confused and try to pull an image with that name.

Third

Otherwise, we know Docker is your problem (at least currently). So, you can try again to install directly and troubleshoot any errors you encounter there.

[robbydigital]

Edit: Ok, so I think editing docker-compose.yml and running docker-compose up --build worked (mostly)! The 'cat' container now appears on Docker and I can access the login page on the web interface.

I did have to rerun the command several times after getting error messages; first because I hadn't generated a random salt as required in config.py and 4cat-daemon.py (I got ...error while starting 4CAT Backend Daemon (lockfile not found) and second because I hadn't put my email and SMTP server information in config.py and so "localhost" was undefined.

I think I solved those issues, however when I submit the username request form via the web interface I'm getting the message 'The form could not be submitted; the e-mail server is unreachable.' Does that mean my email server is unreachable, and if so that I've made a mistake with the SMTP server information I've included?

This is the current output I'm getting from docker-compose up --build :

PS C:\users\robto\4cat> docker-compose up --build
[+] Building 16.7s (29/41)
=> [cat_backend internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 32B 0.0s
=> [cat_backend internal] load .dockerignore 0.5s
=> => transferring context: 2B 0.0s
=> [cat_frontend internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 32B 0.0s
=> [cat_frontend internal] load .dockerignore 0.2s
=> => transferring context: 2B 0.0s
=> [cat_backend internal] load metadata for docker.io/library/debian:buster 12.5s
=> [auth] library/debian:pull token for registry-1.docker.io 0.0s
=> [cat_frontend internal] load build context 1.3s
=> => transferring context: 168.94kB 1.2s
=> [cat_backend 1/20] FROM docker.io/library/debian:buster@sha256:f4b36c72b25772bd4f83e939d9b17dfe804977e455338 0.0s
=> [cat_backend internal] load build context 1.3s
=> => transferring context: 168.94kB 1.2s
=> CACHED [cat_frontend 2/20] RUN apt-get update && apt install -y python3-pip gawk curl libpq- 0.0s
=> CACHED [cat_frontend 3/20] WORKDIR /usr/src/app 0.0s
=> CACHED [cat_frontend 4/20] RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.de 0.0s
=> CACHED [cat_frontend 5/20] RUN dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install 0.0s
=> CACHED [cat_frontend 6/20] RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`cur 0.0s
=> CACHED [cat_frontend 7/20] RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/ 0.0s
=> CACHED [cat_frontend 8/20] RUN pip3 install --upgrade pip 0.0s
=> CACHED [cat_frontend 9/20] COPY ./requirements.txt /usr/src/app/requirements.txt 0.0s
=> CACHED [cat_frontend 10/20] COPY ./setup.py /usr/src/app/setup.py 0.0s
=> CACHED [cat_frontend 11/20] COPY ./VERSION /usr/src/app/VERSION 0.0s
=> CACHED [cat_frontend 12/20] COPY ./README.md /usr/src/app/README.md 0.0s
=> CACHED [cat_frontend 13/20] RUN mkdir /usr/src/app/backend 0.0s
=> CACHED [cat_frontend 14/20] RUN mkdir /usr/src/app/webtool 0.0s
=> CACHED [cat_frontend 15/20] RUN mkdir /usr/src/app/datasources 0.0s
=> CACHED [cat_frontend 16/20] RUN pip3 install -r requirements.txt 0.0s
=> CACHED [cat_frontend 17/20] RUN pip3 install gunicorn 0.0s
=> [cat_frontend 18/20] COPY ./config.py-example /usr/src/app/config.py 0.1s
=> [cat_frontend 19/20] COPY ./VERSION /usr/src/app/.current-version 0.2s
=> [cat_frontend 20/20] COPY . /usr/src/app/ 0.7s
=> [cat_frontend] exporting to image 0.9s
=> => exporting layers 0.7s
=> => writing image sha256:2f002e735f6e5251e2a617378fa32e84108b24274a85f8468ff7707bf8d9697f 0.0s
=> => naming to docker.io/library/cat_backend 0.0s
=> => naming to docker.io/library/cat_frontend 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 3/3

• Container db Running 0.0s
• Container 4cat_backend Recreated 15.2s
• Container 4cat_frontend Recreated 0.5s
  Attaching to 4cat_backend, 4cat_frontend, db
  4cat_backend | Waiting for postgres...
  4cat_backend | PostgreSQL started
  4cat_backend | 1
  4cat_backend | Seed present
  4cat_backend | Starting app
  4cat_backend | Version is up to date
  4cat_backend | 4CAT is accessible at:
  4cat_backend | http://localhost:80
  4cat_backend |
  4cat_backend | Starting 4CAT Backend Daemon...
  4cat_backend | ...4CAT Backend Daemon started.
  4cat_frontend | [2021-09-29 14:47:11 +0000] [1] [INFO] Starting gunicorn 20.1.0
  4cat_frontend | [2021-09-29 14:47:11 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
  4cat_frontend | [2021-09-29 14:47:11 +0000] [1] [INFO] Using worker: gthread
  4cat_frontend | [2021-09-29 14:47:11 +0000] [9] [INFO] Booting worker with pid: 9
  4cat_frontend | [2021-09-29 14:47:11 +0000] [10] [INFO] Booting worker with pid: 10
  4cat_frontend | [2021-09-29 14:47:42 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:9)
  4cat_frontend | [2021-09-29 14:47:42 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:10)
  4cat_frontend | [2021-09-29 14:47:42 +0000] [10] [INFO] Worker exiting (pid: 10)
  4cat_frontend | [2021-09-29 14:47:42 +0000] [9] [INFO] Worker exiting (pid: 9)
  4cat_frontend | [2021-09-29 14:47:42 +0000] [41] [INFO] Booting worker with pid: 41
  4cat_frontend | [2021-09-29 14:47:42 +0000] [1] [WARNING] Worker with pid 10 was terminated due to signal 9
  4cat_frontend | [2021-09-29 14:47:42 +0000] [42] [INFO] Booting worker with pid: 42

[dale-wahl]

OH. So, you... already set it up. And, well, the first time it worked it provided you with a username and password. Now when you build it, it knows a user is already there! Try this command to retrieve it. Note that you may wish to delete that file afterwards depending on who you give access to your computer.

docker exec -it 4cat_backend cat docker/shared/login.txt

Also, I am guessing you created your own config.py file, probably when you originally tried to install it manually. That is slightly unnecessary if you are using the docker install as it will create it's own if you had not already done so.

Also, in the future, you only need to run docker-compose up to start the service. It will be faster. You can also start it in the Docker Desktop app with the play button.

[robbydigital]

Ah, fantastic - I'm in! Thank you so much for your help. I think I made it more complicated than it had to be...but we got there in the end.

[dale-wahl]

Glad to hear it! Feel free to open up an issue with any other questions.