Immich server does not start after update to v1.119.1 #4202

Author

Owner

@alextran1502 commented on GitHub (Nov 1, 2024):

@werefkin Can you try restart the stack with docker compose down, docker compose up?

@lupoalberto12 your same problem might be from different issues with the setup, so you should create a different discussion thread with your setup and logs we can help

@alextran1502 commented on GitHub (Nov 1, 2024): @werefkin Can you try restart the stack with docker compose down, docker compose up? @lupoalberto12 your same problem might be from different issues with the setup, so you should create a different discussion thread with your setup and logs we can help

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 1, 2024):

@werefkin Can you try restart the stack with docker compose down, docker compose up?

Hm.. It is the same, it fails to start (there are some failure counts at the beginning and unhealthy status then), but apparently after some random time it does start indeed, cannot distingush a pattern yet.
I also redeployed the container and restored a snapshot from library and database from some days ago, it is the same

@werefkin commented on GitHub (Nov 1, 2024): > @werefkin Can you try restart the stack with docker compose down, docker compose up? Hm.. It is the same, it fails to start (there are some failure counts at the beginning and unhealthy status then), but apparently after some random time it does start indeed, cannot distingush a pattern yet. I also redeployed the container and restored a snapshot from library and database from some days ago, it is the same

deekerman commented

Author

Owner

@alextran1502 commented on GitHub (Nov 1, 2024):

~~Can you try bring down the containers, remove the images, then repull and restart?~~

Nvm it started, so it was just downloading and ingesting the geocoding data

@alextran1502 commented on GitHub (Nov 1, 2024): ~~Can you try bring down the containers, remove the images, then repull and restart?~~ Nvm it started, so it was just downloading and ingesting the geocoding data

deekerman commented

Author

Owner

@yunasc commented on GitHub (Nov 1, 2024):

How long did it took for your instance to start? I tried waiting for 2 hours - no luck.

@yunasc commented on GitHub (Nov 1, 2024): How long did it took for your instance to start? I tried waiting for 2 hours - no luck.

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 1, 2024):

How long did it took for your instance to start? I tried waiting for 2 hours - no luck.

not that long, a few minutes, I still see the error and starting attempts though. I completety remove the stack and restored with my database backup.

@werefkin commented on GitHub (Nov 1, 2024): > How long did it took for your instance to start? I tried waiting for 2 hours - no luck. not that long, a few minutes, I still see the error and starting attempts though. I completety remove the stack and restored with my database backup.

deekerman commented

Author

Owner

@yunasc commented on GitHub (Nov 1, 2024):

Yikes! I know that Immich is for 3-2-1 setup, but still from v1.x.x I would expect some backwards compatibility, especially since changelog mentioned nothing about this.

@yunasc commented on GitHub (Nov 1, 2024): Yikes! I know that Immich is for 3-2-1 setup, but still from v1.x.x I would expect *some* backwards compatibility, especially since changelog mentioned nothing about this.

deekerman commented

Author

Owner

@bo0tzz commented on GitHub (Nov 1, 2024):

To note - Immich is not following semver yet. I'm not sure what you're referring to with the changelog/compat comment though?

@bo0tzz commented on GitHub (Nov 1, 2024): To note - Immich is not following semver yet. I'm not sure what you're referring to with the changelog/compat comment though?

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 1, 2024):

How long did it took for your instance to start? I tried waiting for 2 hours - no luck.

I needed 7 minutes with 1 failed trial and abovementioned TypeError issue. I will see how it performs, might be I think about downgrading it..

@werefkin commented on GitHub (Nov 1, 2024): > How long did it took for your instance to start? I tried waiting for 2 hours - no luck. I needed 7 minutes with 1 failed trial and abovementioned `TypeError` issue. I will see how it performs, might be I think about downgrading it..

deekerman commented

Author

Owner

@alextran1502 commented on GitHub (Nov 1, 2024):

@zackpollard I wonder if there is any log we can add in this process to give users some feedback?

@alextran1502 commented on GitHub (Nov 1, 2024): @zackpollard I wonder if there is any log we can add in this process to give users some feedback?

deekerman commented

Author

Owner

@zackpollard commented on GitHub (Nov 1, 2024):

The geodata really should take minutes at maximum, for most people it's like 10 seconds. I'm surprised it's taking this long but I can take a look into it at some point

@zackpollard commented on GitHub (Nov 1, 2024): The geodata really should take minutes at maximum, for most people it's like 10 seconds. I'm surprised it's taking this long but I can take a look into it at some point

deekerman commented

Author

Owner

@avtarsingh1122 commented on GitHub (Nov 2, 2024):

for me it took approx 16 minutes to be healthy, but working fine afterwards

@avtarsingh1122 commented on GitHub (Nov 2, 2024): for me it took approx 16 minutes to be healthy, but working fine afterwards

deekerman commented

Author

Owner

@yunasc commented on GitHub (Nov 3, 2024):

changelog/compat comment though?

nvm.

As for the issue, I can see (after plugging in main image from 13 hours ago), that the slow part is pgsql.
Which is created from the helm chart with the following resources (default ones):

        limits:
          cpu: 150m
          ephemeral-storage: 2Gi
          memory: 192Mi
        requests:
          cpu: 100m
          ephemeral-storage: 50Mi
          memory: 128Mi

After uncapping pgsql and restarting both immich and pgsql now it's done pretty fast:

[Nest] 17  - 11/03/2024, 4:43:38 PM     LOG [Api:EventRepository] Initialized websocket server
[Nest] 7  - 11/03/2024, 4:44:20 PM     LOG [Microservices:MapRepository] Geodata import completed

@yunasc commented on GitHub (Nov 3, 2024): > changelog/compat comment though? nvm. As for the issue, I can see (after plugging in main image from 13 hours ago), that the slow part is pgsql. Which is created from the helm chart with the following resources (default ones): ``` resources: limits: cpu: 150m ephemeral-storage: 2Gi memory: 192Mi requests: cpu: 100m ephemeral-storage: 50Mi memory: 128Mi ``` After uncapping pgsql and restarting both immich and pgsql now it's done pretty fast: ``` [Nest] 17 - 11/03/2024, 4:43:38 PM LOG [Api:EventRepository] Initialized websocket server [Nest] 7 - 11/03/2024, 4:44:20 PM LOG [Microservices:MapRepository] Geodata import completed ```

deekerman commented

Author

Owner

@JvdMaat commented on GitHub (Nov 3, 2024):

Came here after an hour of troubleshooting figuring I had messed up the port 3001->2283 conversion (after Immich had already been down for 4 hours). Turns out it took 5 hours for postgres to fix itself and allow immich to boot. This would have been good to know before upgrading.

@JvdMaat commented on GitHub (Nov 3, 2024): Came here after an hour of troubleshooting figuring I had messed up the port 3001->2283 conversion (after Immich had already been down for 4 hours). Turns out it took 5 hours for postgres to fix itself and allow immich to boot. This would have been good to know before upgrading. ![image](https://github.com/user-attachments/assets/9dc1e78f-fbdc-4e67-ae91-decaa3e620e4)

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 3, 2024):

Came here after an hour of troubleshooting figuring I had messed up the port 3001->2283 conversion (after Immich had already been down for 4 hours). Turns out it took 5 hours for postgres to fix itself and allow immich to boot. This would have been good to know before upgrading.

Ha, actually, I did change the port during the troubleshooting as well (completely reinstalled immich and restored the backup), maybe this explains that it started to boot after that, just checked the previous docker compose file.

@werefkin commented on GitHub (Nov 3, 2024): > Came here after an hour of troubleshooting figuring I had messed up the port 3001->2283 conversion (after Immich had already been down for 4 hours). Turns out it took 5 hours for postgres to fix itself and allow immich to boot. This would have been good to know before upgrading. > ![image](https://github.com/user-attachments/assets/9dc1e78f-fbdc-4e67-ae91-decaa3e620e4) > Ha, actually, I did change the port during the troubleshooting as well (completely reinstalled immich and restored the backup), maybe this explains that it started to boot after that, just checked the previous docker compose file.

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 4, 2024):

@zackpollard

I am wondering, the issue is closed, was there any solution? Apparently the problem is common

@werefkin commented on GitHub (Nov 4, 2024): @zackpollard I am wondering, the issue is closed, was there any solution? Apparently the problem is common

deekerman commented

Author

Owner

@disconnect5852 commented on GitHub (Nov 5, 2024):

also me having this problem immich_server and immich_microservices are crashing. Reverted ti v114, where i upgraded from, and it works.

@disconnect5852 commented on GitHub (Nov 5, 2024): also me having this problem immich_server and immich_microservices are crashing. Reverted ti v114, where i upgraded from, and it works.

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 5, 2024):

also me having this problem immich_server and immich_microservices are crashing. Reverted ti v114, where i upgraded from, and it works.

Ok, I believe it is the same bug. The microservices do have indeed the same behaviour as server and fail to start. Did you modify version in the env file to downgrade?

@werefkin commented on GitHub (Nov 5, 2024): > also me having this problem immich_server and immich_microservices are crashing. Reverted ti v114, where i upgraded from, and it works. Ok, I believe it is the same bug. The microservices do have indeed the same behaviour as server and fail to start. Did you modify version in the env file to downgrade?

deekerman commented

Author

Owner

@zackpollard commented on GitHub (Nov 5, 2024):

If people who are commenting they have the same issue please include what hardware you're running on and confirm that you don't have any resource limits set on your containers. I did a bunch of testing yesterday and couldn't get the geocoding import process to take any more than 45 seconds, except in the case that postgres resources were severely restricted for RAM.

@zackpollard commented on GitHub (Nov 5, 2024): If people who are commenting they have the same issue please include what hardware you're running on and confirm that you don't have any resource limits set on your containers. I did a bunch of testing yesterday and couldn't get the geocoding import process to take any more than 45 seconds, except in the case that postgres resources were severely restricted for RAM.

deekerman commented

Author

Owner

@disconnect5852 commented on GitHub (Nov 5, 2024):

also me having this problem immich_server and immich_microservices are crashing. Reverted ti v114, where i upgraded from, and it works.

Ok, I believe it is the same bug. The microservices do have indeed the same behaviour as server and fail to start. Did you modify version in the env file to downgrade?

Yes, i set the version in .env.

My hardware is a 2 core celeron N2807 with 4 gb ram. Nothing ran out of resources, swap is just on 16%

@disconnect5852 commented on GitHub (Nov 5, 2024): > > also me having this problem immich_server and immich_microservices are crashing. Reverted ti v114, where i upgraded from, and it works. > > Ok, I believe it is the same bug. The microservices do have indeed the same behaviour as server and fail to start. Did you modify version in the env file to downgrade? Yes, i set the version in .env. My hardware is a 2 core celeron N2807 with 4 gb ram. Nothing ran out of resources, swap is just on 16%

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 5, 2024):

If people who are commenting they have the same issue please include what hardware you're running on and confirm that you don't have any resource limits set on your containers. I did a bunch of testing yesterday and couldn't get the geocoding import process to take any more than 45 seconds, except in the case that postgres resources were severely restricted for RAM.

I run it (together with portrainer and nextcloud) on Raspberry Pi 4 with 4GB RAM. No limits are set, the issue was not present on the previous versions. As mentioned, it did not start at all right after the update, I made a completely new deployment of the stack and restored the backup after that it starts, though takes more time with some startup failures.

@werefkin commented on GitHub (Nov 5, 2024): > If people who are commenting they have the same issue please include what hardware you're running on and confirm that you don't have any resource limits set on your containers. I did a bunch of testing yesterday and couldn't get the geocoding import process to take any more than 45 seconds, except in the case that postgres resources were severely restricted for RAM. I run it (together with portrainer and nextcloud) on Raspberry Pi 4 with 4GB RAM. No limits are set, the issue was not present on the previous versions. As mentioned, it did not start at all right after the update, I made a completely new deployment of the stack and restored the backup after that it starts, though takes more time with some startup failures.

deekerman commented

Author

Owner

@bo0tzz commented on GitHub (Nov 5, 2024):

2 core celeron N2807 with 4 gb ram.

That's a seriously underpowered system. I don't think we can do much about Immich running poorly on that.

@bo0tzz commented on GitHub (Nov 5, 2024): > 2 core celeron N2807 with 4 gb ram. That's a seriously underpowered system. I don't think we can do much about Immich running poorly on that.

deekerman commented

Author

Owner

@zackpollard commented on GitHub (Nov 5, 2024):

I run it (together with portrainer and nextcloud) on Raspberry Pi 4 with 4GB RAM. No limits are set, the issue was not present on the previous versions. As mentioned, it did not start at all right after the update, I made a completely new deployment of the stack and restored the backup after that it starts, though takes more time with some startup failures.

Could you expand more what you mean by "some startup failures"?

@zackpollard commented on GitHub (Nov 5, 2024): > I run it (together with portrainer and nextcloud) on Raspberry Pi 4 with 4GB RAM. No limits are set, the issue was not present on the previous versions. As mentioned, it did not start at all right after the update, I made a completely new deployment of the stack and restored the backup after that it starts, though takes more time with some startup failures. Could you expand more what you mean by "some startup failures"?

deekerman commented

Author

Owner

@JvdMaat commented on GitHub (Nov 5, 2024):

That's a seriously underpowered system. I don't think we can do much about Immich running poorly on that.

I think a good chunk of the userbase is running it on underpowered systems (I'm running mine (along with 30 other containers) on an old 2014 laptop with 4 cores and 16GB RAM.)
I don't think the issue is that it takes a while. I think it would help if that was communicated. If we know that for a certain upgrade it'll take time to come back up due to DB work, we can wait for that (and even look for that in our DB CPU utilization, which clearly showed postgres working hard for 5 hours).
All I saw when I upgraded (from 117 to 119) was that the immich_server container kept failing healthcheck, crashing and restarting and giving errors on the port 2283. (which coincided with the v118 port change). So I tried troubleshooting an issue that wasn't actually there.

On a side note, I also just noticed that the laptop's CPU is pegged at 100% (and 15m load is at 20) now because I started a missing face detection last night. Which is fine. I don't care if that takes a few days. The other containers are still responsive, and the faces will get detected eventually without requiring further intervention from me.

@JvdMaat commented on GitHub (Nov 5, 2024): > That's a seriously underpowered system. I don't think we can do much about Immich running poorly on that. I think a good chunk of the userbase is running it on underpowered systems (I'm running mine (along with 30 other containers) on an old 2014 laptop with 4 cores and 16GB RAM.) I don't think the issue is that it takes a while. I think it would help if that was communicated. If we know that for a certain upgrade it'll take time to come back up due to DB work, we can wait for that (and even look for that in our DB CPU utilization, which clearly showed postgres working hard for 5 hours). All I saw when I upgraded (from 117 to 119) was that the immich_server container kept failing healthcheck, crashing and restarting and giving errors on the port 2283. (which coincided with the v118 port change). So I tried troubleshooting an issue that wasn't actually there. On a side note, I also just noticed that the laptop's CPU is pegged at 100% (and 15m load is at 20) now because I started a missing face detection last night. Which is fine. I don't care if that takes a few days. The other containers are still responsive, and the faces will get detected eventually without requiring further intervention from me.

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 5, 2024):

Could you expand more what you mean by "some startup failures"?
@zackpollard
Unhealthy status with several failure counts and following output:
TypeError: fetch failed at node:internal/deps/undici/undici:13185:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async main (/usr/src/app/dist/bin/healthcheck.js:14:26) { [cause]: Error: connect ECONNREFUSED 127.0.0.1:2283 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 2283 } }

@werefkin commented on GitHub (Nov 5, 2024): > Could you expand more what you mean by "some startup failures"? @zackpollard Unhealthy status with several failure counts and following output: TypeError: fetch failed at node:internal/deps/undici/undici:13185:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async main (/usr/src/app/dist/bin/healthcheck.js:14:26) { [cause]: Error: connect ECONNREFUSED 127.0.0.1:2283 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 2283 } } ![Screenshot From 2024-11-05 15-05-05](https://github.com/user-attachments/assets/8b007231-ba9c-4646-bbe0-ab4d9125c1f1)

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 5, 2024):

OK, I first wrote that the start was successful after the reinstallation of the entire stack, apparently it is still unstable. It is not running currently and is in starting phase continuously.

@werefkin commented on GitHub (Nov 5, 2024): OK, I first wrote that the start was successful after the reinstallation of the entire stack, apparently it is still unstable. It is not running currently and is in starting phase continuously.

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 5, 2024):

this is the latest output I have

@werefkin commented on GitHub (Nov 5, 2024): ![Screenshot From 2024-11-05 15-31-19](https://github.com/user-attachments/assets/3c0a22ec-b995-4968-b011-386c419f89cf) this is the latest output I have

deekerman commented

Author

Owner

@werefkin commented on GitHub (Nov 5, 2024):

That's a seriously underpowered system. I don't think we can do much about Immich running poorly on that.

I think a good chunk of the userbase is running it on underpowered systems (I'm running mine (along with 30 other containers) on an old 2014 laptop with 4 cores and 16GB RAM.) I don't think the issue is that it takes a while. I think it would help if that was communicated. If we know that for a certain upgrade it'll take time to come back up due to DB work, we can wait for that (and even look for that in our DB CPU utilization, which clearly showed postgres working hard for 5 hours). All I saw when I upgraded (from 117 to 119) was that the immich_server container kept failing healthcheck, crashing and restarting and giving errors on the port 2283. (which coincided with the v118 port change). So I tried troubleshooting an issue that wasn't actually there.

On a side note, I also just noticed that the laptop's CPU is pegged at 100% (and 15m load is at 20) now because I started a missing face detection last night. Which is fine. I don't care if that takes a few days. The other containers are still responsive, and the faces will get detected eventually without requiring further intervention from me.

You seem to experience exactly the same symptoms. Did you downgrade it and if yes, to what version?

@werefkin commented on GitHub (Nov 5, 2024): > > That's a seriously underpowered system. I don't think we can do much about Immich running poorly on that. > > I think a good chunk of the userbase is running it on underpowered systems (I'm running mine (along with 30 other containers) on an old 2014 laptop with 4 cores and 16GB RAM.) I don't think the issue is that it takes a while. I think it would help if that was communicated. If we know that for a certain upgrade it'll take time to come back up due to DB work, we can wait for that (and even look for that in our DB CPU utilization, which clearly showed postgres working hard for 5 hours). All I saw when I upgraded (from 117 to 119) was that the immich_server container kept failing healthcheck, crashing and restarting and giving errors on the port 2283. (which coincided with the v118 port change). So I tried troubleshooting an issue that wasn't actually there. > > On a side note, I also just noticed that the laptop's CPU is pegged at 100% (and 15m load is at 20) now because I started a missing face detection last night. Which is fine. I don't care if that takes a few days. The other containers are still responsive, and the faces will get detected eventually without requiring further intervention from me. You seem to experience exactly the same symptoms. Did you downgrade it and if yes, to what version?

deekerman commented

Author

Owner

@JvdMaat commented on GitHub (Nov 5, 2024):

You seem to experience exactly the same symptoms. Did you downgrade it and if yes, to what version?

I upgraded (from 117 to 119), let it sit for 4 hours (yardwork), came back, noticed it was down and spent an hour troubleshooting (updating ports in docker compose & swag, going back to 117, etc, etc), finally back up to 119, and then it suddenly pulled out of it and worked. Check the CPU utilization on the postgres container. This is what mine looked like during the upgrade to 119:

The dip at 16:15 is where I downgraded to 117, and then back to 119 at 16:30. And eventually once that CPU peak settled at 17:15, immich-server suddenly came online and stopped crashing.

@JvdMaat commented on GitHub (Nov 5, 2024): > You seem to experience exactly the same symptoms. Did you downgrade it and if yes, to what version? I upgraded (from 117 to 119), let it sit for 4 hours (yardwork), came back, noticed it was down and spent an hour troubleshooting (updating ports in docker compose & swag, going back to 117, etc, etc), finally back up to 119, and then it suddenly pulled out of it and worked. Check the CPU utilization on the postgres container. This is what mine looked like during the upgrade to 119: ![image](https://github.com/user-attachments/assets/e79ac4fd-bdbc-4c45-9842-47aa1c897064) The dip at 16:15 is where I downgraded to 117, and then back to 119 at 16:30. And eventually once that CPU peak settled at 17:15, immich-server suddenly came online and stopped crashing.

deekerman commented

Author

Owner

@zackpollard commented on GitHub (Nov 5, 2024):

this is the latest output I have

This is the output of the healthcheck, we would need the output from the logs

@zackpollard commented on GitHub (Nov 5, 2024): > ![Screenshot From 2024-11-05 15-31-19](https://private-user-images.githubusercontent.com/40002124/383188008-3c0a22ec-b995-4968-b011-386c419f89cf.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzA4MjIzMDEsIm5iZiI6MTczMDgyMjAwMSwicGF0aCI6Ii80MDAwMjEyNC8zODMxODgwMDgtM2MwYTIyZWMtYjk5NS00OTY4LWIwMTEtMzg2YzQxOWY4OWNmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMDUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTA1VDE1NTMyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQwYjlmOTNlMDdjYjAxMzJlNmIzOGEwMGVhZTRiZWVmMDllYzRiMWYzYWYxZmFmMTQ1NGE0NTRjZTkzYmViZDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.SMakKyDmtjdL8uu9hTdaxrS1DnPf40xA9zFl-2X0l54) this is the latest output I have This is the output of the healthcheck, we would need the output from the logs

deekerman commented