starred/immich

Fork 0

mirror of https://github.com/immich-app/immich.git synced 2026-03-02 22:57:45 -05:00

Server: Storage layer refactor #442

New issue

Closed

opened 2026-02-19 23:09:22 -05:00 by deekerman · 9 comments

deekerman commented

2026-02-19 23:09:22 -05:00

Owner

Originally created by @bo0tzz on GitHub (Nov 23, 2022).

Feature detail

At the moment, Immich does not have an abstracted storage layer. On upload, files are stored in the semi-hardcoded library path with a randomly generated filename, their path is stored in the database, and in any future (read) operations this stored path is used (file serving, thumbnail generation, etc).
For several of the features we're meaning to (potentially) implement in the future (eg #34, #418, #451), it will be very helpful to refactor and abstract the storage layer. For some of them, like supporting multiple storage backends, it will be necessary entirely. In this issue I want to propose a design, although it will need some more discussion and refinement before it will be complete.

As mentioned above, currently the storage path for a file is generated once and stored in the database. I propose that we instead move to a model where storage paths are built on the fly based on the data we have for an asset. We already use some of that data to build the path on upload right now:

const originalUploadFolder = join(basePath, req.user.id, 'original', sanitizedDeviceId);

Instead, when trying to write or read an asset, the storage layer would expose a function for that which accepts the AssetEntity (or a more limited set of data, if desired). The storage implementation then uses that internally, together with some configuration, to build the actual path. That way, things like the storage path become an implementation detail that does not need to be exposed to the rest of Immich.

I think it would be good to keep the storage providers as self-contained as we can, and avoid having it do things like access the database. Instead, it would take in a configuration when initializing (eg, the root path where to store files, S3 access credentials, or a template for the filename). That configuration can of course be read from the database by whatever code initializes the provider.

This will allow for a multitude of nice things:

Storage backends become swappable. We can then relatively easily add support for storing files on S3, etc.
We can use multiple storage backends for different file types (original file, thumbnail, etc), even on different mediums
(eg thumbnails on disk, originals on S3)
Code for things like path templating can be cleanly contained inside a single storage provider
Migrations between storage types! Just instantiate both the old and the new configuration, and copy between them. No need to even update the database.
Probably other stuff :)

tbd:

What interface does a storage provider need? Probably at least create, delete and stat. How about something like S3, which might be able to provide URLs for direct access (bypassing immich)?
The description here (partly) covers multiple features. What is the exact scope of the initial refactor (and how do we anticipate those future features in it)?
???

Platform

Server

Originally created by @bo0tzz on GitHub (Nov 23, 2022). ### Feature detail At the moment, Immich does not have an abstracted storage layer. On upload, [files are stored](https://github.com/immich-app/immich/blob/main/server/apps/immich/src/config/asset-upload.config.ts) in the semi-hardcoded library path with a randomly generated filename, their path is stored in the database, and in any future (read) operations this stored path is used ([file serving](https://github.com/immich-app/immich/blob/main/server/apps/immich/src/api-v1/asset/asset.service.ts#L388), [thumbnail generation](https://github.com/immich-app/immich/blob/main/server/apps/microservices/src/processors/thumbnail.processor.ts#L68), etc). For several of the features we're meaning to (potentially) implement in the future (eg #34, #418, #451), it will be very helpful to refactor and abstract the storage layer. For some of them, like supporting multiple storage backends, it will be necessary entirely. In this issue I want to propose a design, although it will need some more discussion and refinement before it will be complete. As mentioned above, currently the storage path for a file is generated once and stored in the database. I propose that we instead move to a model where storage paths are built on the fly based on the data we have for an asset. We already use some of that data to build the path on upload right now: ```typescript const originalUploadFolder = join(basePath, req.user.id, 'original', sanitizedDeviceId); ``` Instead, when trying to write or read an asset, the storage layer would expose a function for that which accepts the `AssetEntity` (or a more limited set of data, if desired). The storage implementation then uses that internally, together with some configuration, to build the actual path. That way, things like the storage path become an implementation detail that does not need to be exposed to the rest of Immich. I think it would be good to keep the storage providers as self-contained as we can, and avoid having it do things like access the database. Instead, it would take in a configuration when initializing (eg, the root path where to store files, S3 access credentials, or a template for the filename). That configuration can of course be read from the database by whatever code initializes the provider. This will allow for a multitude of nice things: * Storage backends become swappable. We can then relatively easily add support for storing files on S3, etc. * We can use multiple storage backends for different file types (original file, thumbnail, etc), even on different mediums (eg thumbnails on disk, originals on S3) * Code for things like path templating can be cleanly contained inside a single storage provider * Migrations between storage types! Just instantiate both the old and the new configuration, and copy between them. No need to even update the database. * Probably other stuff :) tbd: 1. What interface does a storage provider need? Probably at least `create`, `delete` and `stat`. How about something like S3, which might be able to provide URLs for direct access (bypassing immich)? 2. The description here (partly) covers multiple features. What is the exact scope of the initial refactor (and how do we anticipate those future features in it)? 3. ??? ### Platform Server

deekerman

2026-02-19 23:09:22 -05:00

closed this issue
added the
🗄️server
label

deekerman commented

2026-02-19 23:09:23 -05:00

Author

Owner

@Cellivar commented on GitHub (Dec 7, 2022):

If I might be so bold as to add some unsolicited advice..

You're very close to the concept of a generic blob storage interface, where filesystem storage just a slightly weird looking blob storage API. Blob storage for large files, like images, is a very common design pattern for modern networked systems. Though my link hasn't been updated for a few years it's a good example of the way you may want to head in with your implementation. I suspect you can find more modern options for TypeScript out there, searching from my phone is difficult.

Your abstraction layer you described becomes blob operations, which then translate into actual blob API calls (filesystem write, s3 write, NFS share write, etc).

@Cellivar commented on GitHub (Dec 7, 2022): If I might be so bold as to add some unsolicited advice.. You're very close to the concept of [a generic blob storage interface](https://github.com/maxogden/abstract-blob-store), where filesystem storage just a slightly weird looking blob storage API. Blob storage for large files, like images, is a very common design pattern for modern networked systems. Though my link hasn't been updated for a few years it's a good example of the way you may want to head in with your implementation. I suspect you can find more modern options for TypeScript out there, searching from my phone is difficult. Your abstraction layer you described becomes blob operations, which then translate into actual blob API calls (filesystem write, s3 write, NFS share write, etc).

deekerman commented

2026-02-19 23:09:23 -05:00

Author

Owner

@bo0tzz commented on GitHub (Dec 8, 2022):

That's very helpful, thank you!

@bo0tzz commented on GitHub (Dec 8, 2022): That's very helpful, thank you!

deekerman commented

2026-02-19 23:09:23 -05:00

Author

Owner

@pinpox commented on GitHub (Jul 11, 2023):

Does immich support S3 Storage currently? I saw a related pr was merged some time ago, but can't figure out how to set it up.

@pinpox commented on GitHub (Jul 11, 2023): Does immich support S3 Storage currently? I saw a related pr was merged some time ago, but can't figure out how to set it up.

deekerman commented

2026-02-19 23:09:23 -05:00

Author

Owner

@jrasm91 commented on GitHub (Jul 11, 2023):

This is probably what you are thinking of: https://github.com/immich-app/immich/discussions/1683#discussioncomment-6206105

@jrasm91 commented on GitHub (Jul 11, 2023): This is probably what you are thinking of: https://github.com/immich-app/immich/discussions/1683#discussioncomment-6206105

deekerman commented

2026-02-19 23:09:23 -05:00

Author

Owner

@ibotty commented on GitHub (Aug 28, 2024):

I just wanted to point to Apache OpenDAL which is used in the big data ecosystem quiet a bit. It is a unified storage layer supporting many different storage systems, among it s3 and local posix file systems.

It also has node bindings.

https://opendal.apache.org/docs/nodejs/

@ibotty commented on GitHub (Aug 28, 2024): I just wanted to point to Apache OpenDAL which is used in the big data ecosystem quiet a bit. It is a unified storage layer supporting many different storage systems, among it s3 and local posix file systems. It also has node bindings. https://opendal.apache.org/docs/nodejs/

deekerman commented

2026-02-19 23:09:23 -05:00

Author

Owner

@algora-pbc commented on GitHub (Aug 2, 2025):

💎 nycterent is offering a $100 bounty for this issue. View and reward the bounty at algora.io/immich-app/immich/issues/1011

👉 Got a pull request resolving this? Claim the bounty by commenting /claim #1011 in your PR and joining algora.io

@algora-pbc commented on GitHub (Aug 2, 2025): 💎 **nycterent** is offering a **$100** bounty for this issue. View and reward the bounty at `algora.io/immich-app/immich/issues/1011` 👉 Got a pull request resolving this? Claim the bounty by commenting `/claim #1011` in your PR and joining `algora.io`

deekerman commented

2026-02-19 23:09:24 -05:00

Author

Owner

@bo0tzz commented on GitHub (Aug 3, 2025):

We don't want open bounties like this, so I've hidden the above comment. The zig project explains it better than I could at https://ziglang.org/news/bounties-damage-open-source-projects/.

@bo0tzz commented on GitHub (Aug 3, 2025): We don't want open bounties like this, so I've hidden the above comment. The zig project explains it better than I could at https://ziglang.org/news/bounties-damage-open-source-projects/.

deekerman commented

2026-02-19 23:09:24 -05:00

Author

Owner

@mikes1991gh commented on GitHub (Aug 7, 2025):

💯

It's been about a year since any update here or for https://github.com/immich-app/immich/issues/4445 so I think a lot of people would appreciate any kind of update. Not necessarily that it is happening tomorrow but some sign if this (new storage layer and S3/backblaze/etc compat in particular) is something that might be considered in the medium term.

@mikes1991gh commented on GitHub (Aug 7, 2025): 💯 It's been about a year since any update here or for https://github.com/immich-app/immich/issues/4445 so I think a lot of people would appreciate any kind of update. Not necessarily that it is happening tomorrow but some sign if this (new storage layer and S3/backblaze/etc compat in particular) is something that might be considered in the medium term.

deekerman commented

2026-02-19 23:09:24 -05:00

Author

Owner

@unedited-despair commented on GitHub (Aug 19, 2025):

The ownCloud OCIS server comes with a pretty advanced thougt model / architecture / implementation of a storage layer.
It's a different technology stack, but could still serve as good inspiration:

Functional concept - Federated storage: This could be directly applicable to Immich
Runtime concept - Cloud Storage Services for Synchronization and Sharing (CS3): The OCIS implementation revolves around REVA, the reference client for CS3. REVA supports local filesystems as well as S3 for example. See for example: github.com/owncloud/ocis@20febffaa7/services/storage-users/pkg/revaconfig/drivers.go (L271)
Example: Storage config for S3: https://doc.owncloud.com/ocis/next/deployment/storage/s3.html
Full list of REVA's storage drivers incl. explanatory remarks / upsides / downsides: https://github.com/cs3org/reva/wiki/storages#storage-driver-implementations-in-reva

I second @mikes1991gh's feedback, would be nice to get some hint about the plans in regard to the storage layer refactor.

@unedited-despair commented on GitHub (Aug 19, 2025): The ownCloud OCIS server comes with a pretty advanced thougt model / architecture / implementation of a storage layer. It's a different technology stack, but could still serve as good inspiration: 1. Functional concept - [Federated storage](https://doc.owncloud.com/ocis/next/architecture/architecture.html#federated-storage): _This could be directly applicable to Immich_ 2. Runtime concept - [Cloud Storage Services for Synchronization and Sharing (CS3)](https://doc.owncloud.com/ocis/next/architecture/architecture.html#reva-and-cs3): The OCIS implementation revolves around REVA, the reference client for CS3. REVA supports local filesystems as well as S3 for example. See for example: https://github.com/owncloud/ocis/blob/20febffaa7f68bfc0ced1ca76b5414df8afc6bf7/services/storage-users/pkg/revaconfig/drivers.go#L271 3. Example: Storage config for S3: https://doc.owncloud.com/ocis/next/deployment/storage/s3.html 4. Full list of REVA's storage drivers incl. explanatory remarks / upsides / downsides: https://github.com/cs3org/reva/wiki/storages#storage-driver-implementations-in-reva I second @mikes1991gh's feedback, would be nice to get some hint about the plans in regard to the storage layer refactor.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/immich#442

No description provided.

Rows
Columns