Index: Option to automatically create albums from files paths (including duplicates), e.g. for album exports from other tools #950

Open
opened 2026-02-20 00:02:24 -05:00 by deekerman · 14 comments
Owner

Originally created by @blaineam on GitHub (May 20, 2021).

As a user, I want to migrate albums from other tools like Google Photos or Apple Photos to PhotoPrism without having duplicates in my filesystem, so that I do not need to create albums again.

When exporting albums from e.g. Google Photos you get one JSON file with metadata of the album, like name or description. But there is no relation to the photos that belong to this album in the JSON.

Additionally you get a folder for each album e.g. "Holiday 2020". You might also get additional folders for each day, e.g. "2020-04-01". One photo might be located in multiple folders.

Acceptance Criteria:

  • Find a way to recreate only the relevant albums in PhotoPrism without having duplicates

I know that the index option creates folders in the side nav for where the foldername is in the Originals folder.

I would like a similar feature for when a folder is created and used when uploading files via WEBDAV to the import folder.

This would allow you to sync a folder hierarchy of a library of album exports that may result in duplicated photos and videos because the same media is in two folders on a fileystem.

The import would still de dupe the files based on hash, however it would use the folders that the media was uploaded to, to create or add to existing albums of the same name.

This would allow for quick upload and immediate optimization by deduping the files saving in overall storage space.

I think it would be a great addition that can even help with importing photo library exports in an efficient manner all while allowing albums to be kept when moving from a different platform to photoprism. Google take exports the files in this manner and Apple Photos can export albums like this with the help of third party scripts.

I am hesitant to use the Originals folder because photoprism does not mutate those files which means there will be a rather large percentage of duplicate files for albums that share a lot of the same media.

Originally created by @blaineam on GitHub (May 20, 2021). **As a user, I want to migrate albums from other tools like Google Photos or Apple Photos to PhotoPrism without having duplicates in my filesystem, so that I do not need to create albums again.** When exporting albums from e.g. Google Photos you get one JSON file with metadata of the album, like name or description. But there is no relation to the photos that belong to this album in the JSON. Additionally you get a folder for each album e.g. "Holiday 2020". You might also get additional folders for each day, e.g. "2020-04-01". One photo might be located in multiple folders. Acceptance Criteria: * Find a way to recreate only the relevant albums in PhotoPrism without having duplicates ---------------------------------------------------------------------------------------------------------------------- I know that the index option creates folders in the side nav for where the foldername is in the Originals folder. I would like a similar feature for when a folder is created and used when uploading files via WEBDAV to the import folder. This would allow you to sync a folder hierarchy of a library of album exports that may result in duplicated photos and videos because the same media is in two folders on a fileystem. The import would still de dupe the files based on hash, however it would use the folders that the media was uploaded to, to create or add to existing albums of the same name. This would allow for quick upload and immediate optimization by deduping the files saving in overall storage space. I think it would be a great addition that can even help with importing photo library exports in an efficient manner all while allowing albums to be kept when moving from a different platform to photoprism. Google take exports the files in this manner and Apple Photos can export albums like this with the help of third party scripts. I am hesitant to use the Originals folder because photoprism does not mutate those files which means there will be a rather large percentage of duplicate files for albums that share a lot of the same media.
Author
Owner

@graciousgrey commented on GitHub (May 20, 2021):

If you want to keep your folder structure you should add your files to originals not to import.
Duplicates are detected in originals as well. At the moment PhotoPrism displays the first one it indexes in the UI, so your views won't be polluted by duplicates. You can find all duplicate files in the database as well.

Advanced handling of duplicates in the UI is planned as well, see https://github.com/photoprism/photoprism/issues/1308 .
With the advanced handling you could see which folders in originals contain a duplicated photo.
E.g. if you see you have the same file in "original/pets" and in "originals/family". You could delete one file and add the remaining photo to albums named "pets" and "family".

@graciousgrey commented on GitHub (May 20, 2021): If you want to keep your folder structure you should add your files to originals not to import. Duplicates are detected in originals as well. At the moment PhotoPrism displays the first one it indexes in the UI, so your views won't be polluted by duplicates. You can find all duplicate files in the database as well. Advanced handling of duplicates in the UI is planned as well, see https://github.com/photoprism/photoprism/issues/1308 . With the advanced handling you could see which folders in originals contain a duplicated photo. E.g. if you see you have the same file in "original/pets" and in "originals/family". You could delete one file and add the remaining photo to albums named "pets" and "family".
Author
Owner

@blaineam commented on GitHub (May 20, 2021):

Will this UI improvement allows for a single or few button clicks to remove duplicates across all of the duplicates?

@blaineam commented on GitHub (May 20, 2021): Will this UI improvement allows for a single or few button clicks to remove duplicates across all of the duplicates?
Author
Owner

@graciousgrey commented on GitHub (May 21, 2021):

What exactly do you mean with "remove duplicates across all of the duplicates"?

@graciousgrey commented on GitHub (May 21, 2021): What exactly do you mean with "remove duplicates across all of the duplicates"?
Author
Owner

@blaineam commented on GitHub (May 21, 2021):

What exactly do you mean with "remove duplicates across all of the duplicates"?

I just meant that from what I understand about how the app handles duplicates from the originals folder is it creates a "stack" of paths to images that are the same image. With that said I would imagine this ui would allow you to review duplicates and squash the "stacks" down to just the original image and then it would create albums for the other directories that the image used to reside in.

I have some issues with this approach which depends on a few things:

  • If you have to manually go through each duplicate stack one by one and squash them I don't think that will work for how I wanted to use this app.
  • If albums don't get updated and can only ever get created new by default this one help my intended use case either. I would like if an album has the same name it would update the existing album instead of creating a duplicate album. Maybe that's how this would work.
  • will the deduplication and or import create an album for the original image at the top of the stack that does not get removed? From what I understand about the original folder is that it only creates a entry in the folders menu and will not create or update the albums under the albums menu?

Basically what I really could use is a way to drop files into a folder that have a folder structure let the app crunch through the folder structure creating albums with de duplicated photos. End result is minimal storage usage and maximum usability for how the files were before moving to photoprism.

@blaineam commented on GitHub (May 21, 2021): > What exactly do you mean with "remove duplicates across all of the duplicates"? I just meant that from what I understand about how the app handles duplicates from the originals folder is it creates a "stack" of paths to images that are the same image. With that said I would imagine this ui would allow you to review duplicates and squash the "stacks" down to just the original image and then it would create albums for the other directories that the image used to reside in. I have some issues with this approach which depends on a few things: - If you have to manually go through each duplicate stack one by one and squash them I don't think that will work for how I wanted to use this app. - If albums don't get updated and can only ever get created new by default this one help my intended use case either. I would like if an album has the same name it would update the existing album instead of creating a duplicate album. Maybe that's how this would work. - will the deduplication and or import create an album for the original image at the top of the stack that does not get removed? From what I understand about the original folder is that it only creates a entry in the folders menu and will not create or update the albums under the albums menu? Basically what I really could use is a way to drop files into a folder that have a folder structure let the app crunch through the folder structure creating albums with de duplicated photos. End result is minimal storage usage and maximum usability for how the files were before moving to photoprism.
Author
Owner

@lastzero commented on GitHub (May 21, 2021):

Excact duplicates don't get indexed at all, so they won't get stacked as well. Stacks are for related or similar files, like different variants or formats.

@lastzero commented on GitHub (May 21, 2021): Excact duplicates don't get indexed at all, so they won't get stacked as well. Stacks are for related or similar files, like different variants or formats.
Author
Owner

@graciousgrey commented on GitHub (May 24, 2021):

Stacks and duplicates are different things:

PhotoPrism has different album types:

In the PhotoPrism UI you will not find any duplicates at the moment.

Example Use Case: You have a file with the same hash in the folders /family and /vacation.

Import:
If the importer encounters a file that has the same hash as a file that is already in originals it will be skipped. Any information on the path of the skipped file is lost.
The intention of the importer is to sort files after date taken and not folder structure before import (we have index for this).
For the use case above the importer would import the file from /family, the file would be moved to the respective folder in originals e.g. /2020/07 and it will get the keyword "family". When the importer comes to the same file in /vacation it will just skip the file. There will be no entry in the duplicates table.

Index:
If the indexer indexes a file with the same hash as an already indexed file it skips indexing this file.
Assumed the indexer would first index /originals/family, so the photo appears in the folder family in the UI. If the indexer comes to the same file in /originals/vacation it will be skipped. So it will not appear in the folder vacation in the UI. It is still present in the filesystem and you find it in the duplicates table in the database.

The issue I mentioned #1308 will do the following: Provide a UI for files in the duplicates table, so that the user can see duplicates in the UI and decide which one to delete in the filesystem. If this is implemented you could see in the UI that the file exists in family and vacation. You could decide to delete one of those in the filesystem and add the file to albums (not folders) named vacation and family.

So I guess what you want on top is an option for the indexer to automatically create albums (not folders) from the paths of files including duplicates. This might be a bit tricky as one might not want to have albums for all folders. And albums must be created before one of the duplicates is deleted in the filesystem.

From which tools are you exporting albums?

@graciousgrey commented on GitHub (May 24, 2021): Stacks and duplicates are different things: * Photos in PhotoPrism can consist of multiple files: e.g. a jpg + a raw + an xmp or json file. The files that belong to a photo are listed on the files tab (https://docs.photoprism.org/user-guide/organize/edit/#edit-files) * Stacks are different photos that are grouped together because they are related (see https://docs.photoprism.org/user-guide/organize/stacks/) e.g. the original version of a photo and a edited one --> those are not duplicates PhotoPrism has different album types: * Folders: Display the folder structure from the originals folder on filesystem. Those are updated in case folder structure in originals changes. (https://docs.photoprism.org/user-guide/organize/folders/) * Albums: Those are created by the user and have no representation within the filesystem because one photo can be part of multiple albums (https://docs.photoprism.org/user-guide/organize/albums/) * PhotoPrism has an option to create albums from folders (https://docs.photoprism.org/user-guide/organize/folders/#create-album-from-folder) In the PhotoPrism UI you will not find any duplicates at the moment. Example Use Case: You have a file with the same hash in the folders /family and /vacation. **Import:** If the importer encounters a file that has the same hash as a file that is already in originals it will be skipped. Any information on the path of the skipped file is lost. The intention of the importer is to sort files after date taken and not folder structure before import (we have index for this). For the use case above the importer would import the file from /family, the file would be moved to the respective folder in originals e.g. /2020/07 and it will get the keyword "family". When the importer comes to the same file in /vacation it will just skip the file. There will be no entry in the duplicates table. **Index:** If the indexer indexes a file with the same hash as an already indexed file it skips indexing this file. Assumed the indexer would first index /originals/family, so the photo appears in the folder family in the UI. If the indexer comes to the same file in /originals/vacation it will be skipped. So it will not appear in the folder vacation in the UI. It is still present in the filesystem and you find it in the duplicates table in the database. The issue I mentioned #1308 will do the following: Provide a UI for files in the duplicates table, so that the user can see duplicates in the UI and decide which one to delete in the filesystem. If this is implemented you could see in the UI that the file exists in family and vacation. You could decide to delete one of those in the filesystem and add the file to albums (not folders) named vacation and family. So I guess what you want on top is an option for the indexer to automatically create albums (not folders) from the paths of files including duplicates. This might be a bit tricky as one might not want to have albums for all folders. And albums must be created before one of the duplicates is deleted in the filesystem. From which tools are you exporting albums?
Author
Owner

@cabe1214 commented on GitHub (Jul 11, 2021):

Hi all, before creating a new issue I'd like to ask here if this is the use case i'm looking for.
ATM I've organized albums in folders (Folder1, Folder2, Folder3) and I would like Photoprism to automatically create/update an album associated to each folder (Folder1=Album1, Folder2=Album2, ...).

So far I can do this with 2 clicks, the point is that every time new files are added to "Folder1" I'd need to manually trigger the update of Album1 by clicking "Add Folder1 to Album1" (which will add to Album1 the new files).
An automatic update of albums would be my use case.
I can imagine that each folder can have an "associated album" and that every N Minutes a trigger executes the "Add Folder1 to Album1" action.

Is this issue covering my use case? If not I would create a new issue.
Thanks in advance!

@cabe1214 commented on GitHub (Jul 11, 2021): Hi all, before creating a new issue I'd like to ask here if this is the use case i'm looking for. ATM I've organized albums in folders (Folder1, Folder2, Folder3) and I would like Photoprism to automatically create/update an album associated to each folder (Folder1=Album1, Folder2=Album2, ...). So far I can do this with 2 clicks, the point is that every time new files are added to "Folder1" I'd need to manually trigger the update of Album1 by clicking "Add Folder1 to Album1" (which will add to Album1 the new files). An automatic update of albums would be my use case. I can imagine that each folder can have an "associated album" and that every N Minutes a trigger executes the "Add Folder1 to Album1" action. Is this issue covering my use case? If not I would create a new issue. Thanks in advance!
Author
Owner

@graciousgrey commented on GitHub (Jul 12, 2021):

@cabe1214 I guess "smart/conditinal albums" would rather cover your use case.
We plan to add a smart abums feature where a user can define conditions (e.g. photos from this folder / location / keyword ....) and all photos matching those conditions are automatically added to the smart album.
Not sure if we already have a ticket for it.

@graciousgrey commented on GitHub (Jul 12, 2021): @cabe1214 I guess "smart/conditinal albums" would rather cover your use case. We plan to add a smart abums feature where a user can define conditions (e.g. photos from this folder / location / keyword ....) and all photos matching those conditions are automatically added to the smart album. Not sure if we already have a ticket for it.
Author
Owner

@cabe1214 commented on GitHub (Jul 13, 2021):

@graciousgrey thank you for the info! I've searched for smart/conditional albums but found nothing so I think there isn't a ticket for it yet. Should I create a new one?

@cabe1214 commented on GitHub (Jul 13, 2021): @graciousgrey thank you for the info! I've searched for smart/conditional albums but found nothing so I think there isn't a ticket for it yet. Should I create a new one?
Author
Owner

@graciousgrey commented on GitHub (Jul 13, 2021):

@cabe1214 a ticket would be great, thank you :)

@graciousgrey commented on GitHub (Jul 13, 2021): @cabe1214 a ticket would be great, thank you :)
Author
Owner

@cabe1214 commented on GitHub (Jul 13, 2021):

@cabe1214 a ticket would be great, thank you :)

#1431

@cabe1214 commented on GitHub (Jul 13, 2021): > > > @cabe1214 a ticket would be great, thank you :) #1431
Author
Owner

@rob-miller commented on GitHub (Nov 10, 2023):

So I guess what you want on top is an option for the indexer to automatically create albums (not folders) from the paths of files including duplicates. This might be a bit tricky as one might not want to have albums for all folders. And albums must be created before one of the duplicates is deleted in the filesystem.

I think what is wanted is 'import/index as album', which would:

  • extract the album title from the folder name, matching to existing album if present
  • if photo is duplicate, add the original to the named album
  • if photo is not duplicate, import/index and add to named album

From which tools are you exporting albums?

Photosync. My use case is that my wife creates albums on her iPad from our combined photos. This selection and often cropping/editing is extremely valuable, and before I found Photosync it was completely trapped in Apple Photos. With Photosync I at least have these albums backed up, but I have never found a system that can import these albums in a useful way.

@rob-miller commented on GitHub (Nov 10, 2023): > So I guess what you want on top is an option for the indexer to automatically create albums (not folders) from the paths of files including duplicates. This might be a bit tricky as one might not want to have albums for all folders. And albums must be created before one of the duplicates is deleted in the filesystem. I think what is wanted is 'import/index as album', which would: - extract the album title from the folder name, matching to existing album if present - if photo is duplicate, add the original to the named album - if photo is not duplicate, import/index and add to named album > From which tools are you exporting albums? Photosync. My use case is that my wife creates albums on her iPad from our combined photos. This selection and often cropping/editing is extremely valuable, and before I found Photosync it was completely trapped in Apple Photos. With Photosync I at least have these albums backed up, but I have never found a system that can import these albums in a useful way.
Author
Owner

@JirkaValtr commented on GitHub (May 4, 2024):

I would like to add my voice to those wishing for this or a similar feature, which seems to keep coming up (see e.g. #1308, #2663, #3450, #4006). To me, having a reliable view of all subfolders of my Originals folder and all images they contain would be really a great feature. As far as I see it, the Library -> Originals view pretty much already does what I want, except it does not show images, which are duplicates of those already indexed in some other folder. Wouldn't it be a simple thing to add to just show the images in each of the folders where they occur? From what I gather, it seems the information is already available in the database.

@JirkaValtr commented on GitHub (May 4, 2024): I would like to add my voice to those wishing for this or a similar feature, which seems to keep coming up (see e.g. #1308, #2663, #3450, #4006). To me, having a reliable view of all subfolders of my Originals folder and all images they contain would be really a great feature. As far as I see it, the Library -> Originals view pretty much already does what I want, except it does not show images, which are duplicates of those already indexed in some other folder. Wouldn't it be a simple thing to add to just show the images in each of the folders where they occur? From what I gather, it seems the information is already available in the database.
Author
Owner

@thomaseizinger commented on GitHub (May 25, 2024):

So I guess what you want on top is an option for the indexer to automatically create albums (not folders) from the paths of files including duplicates. This might be a bit tricky as one might not want to have albums for all folders. And albums must be created before one of the duplicates is deleted in the filesystem.

I think what is wanted is 'import/index as album', which would:

* extract the album title from the folder name, matching to existing album if present

* if photo is duplicate, add the original to the named album

* if photo is not duplicate, import/index and add to named album

I'd like something similar. In my case, I am uploading pictures from my phone to photoprism via WebDAV. I already create the "albums" on my phone by moving them into directories. It would be nice if photoprism could detect that on import and create an album from it.

@thomaseizinger commented on GitHub (May 25, 2024): > > So I guess what you want on top is an option for the indexer to automatically create albums (not folders) from the paths of files including duplicates. This might be a bit tricky as one might not want to have albums for all folders. And albums must be created before one of the duplicates is deleted in the filesystem. > > I think what is wanted is 'import/index as album', which would: > > * extract the album title from the folder name, matching to existing album if present > > * if photo is duplicate, add the original to the named album > > * if photo is not duplicate, import/index and add to named album I'd like something similar. In my case, I am uploading pictures from my phone to photoprism via WebDAV. I already create the "albums" on my phone by moving them into directories. It would be nice if photoprism could detect that on import and create an album from it.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/photoprism#950
No description provided.