Labels: Allow a model to be trained for better results #371

Author

Owner

@lastzero commented on GitHub (Aug 7, 2020):

See Developer Guide. It's not possible to train TensorFlow models using Go and I guess a few examples would also not lead to good results. Distributed learning might be a good approach, but then you need to deal with different user languages and privacy concerns. Also the hardware of most users won't be powerful enough for training models.

@lastzero commented on GitHub (Aug 7, 2020): See Developer Guide. It's not possible to train TensorFlow models using Go and I guess a few examples would also not lead to good results. Distributed learning might be a good approach, but then you need to deal with different user languages and privacy concerns. Also the hardware of most users won't be powerful enough for training models.

deekerman commented

Author

Owner

@derSoerrn95 commented on GitHub (Aug 7, 2020):

Oh sorry, I read the documents page, but - I don't know why - didn't notice the metadata section. The problem is my library has over 50,000 photos and I'm not in the mood to tag all of them.

I am with you that there are many users who do not have powerful hardware. But what about a separate Docker-Container running TensorFlow Python that does the training and shares the result with the Go container?
In this way, the user can freely decide whether he/she would like to train.

@derSoerrn95 commented on GitHub (Aug 7, 2020): Oh sorry, I read the documents page, but - I don't know why - didn't notice the metadata section. The problem is my library has over 50,000 photos and I'm not in the mood to tag all of them. I am with you that there are many users who do not have powerful hardware. But what about a separate Docker-Container running TensorFlow Python that does the training and shares the result with the Go container? In this way, the user can freely decide whether he/she would like to train.

deekerman commented

Author

Owner

@RAYs3T commented on GitHub (Aug 7, 2020):

@derSoerrn95 Wouldn't that just require an API endpoint which provides you the current assosiated lables for a specific picture (+the confedence level)?
The training container could then just grab the current labels from PP and re-evaluate the pictures with the newly (manual) added labels.

Only problem I see here is how you would merge you personal model with the public one?

Also regarding hardware, while yes a lot of people may run this on their Raspberry PI, there are also a few that actually have access to somewhat more powerful hardware. And if they have the option I'm sure there will be some that use it and are happy with it :)

I'm fairly new to the Tensorflow stuff, so I have no idea if "model merging" is a thing.

@RAYs3T commented on GitHub (Aug 7, 2020): @derSoerrn95 Wouldn't that just require an API endpoint which provides you the current assosiated lables for a specific picture (+the confedence level)? The training container could then just grab the current labels from PP and re-evaluate the pictures with the newly (manual) added labels. Only problem I see here is how you would merge you personal model with the public one? Also regarding hardware, while yes a lot of people may run this on their Raspberry PI, there are also a few that actually have access to somewhat more powerful hardware. And if they have the option I'm sure there will be some that use it and are happy with it :) I'm fairly new to the Tensorflow stuff, so I have no idea if "model merging" is a thing.

deekerman commented

Author

Owner

@derSoerrn95 commented on GitHub (Aug 7, 2020):

@RAYs3T Yes, it can make sense to use an API endpoint. You would then have to load all tags for each image, decide which should be used and fill the input / output tensors with them.

It is also possible to load a model that has already been learned and then continue training. But this can lead to some side effects, because data from the original data set may contradict those that you added yourself. In this case it would be better to train a new model.

The new or updated one could then be stored on a shared volume between the containers and then loaded from the GO container.

But I'm not a TensorFlow pro either. I'm currently playing a bit with time series forecasting with RNNs and LSTMs. I've never worked with pictures.

@derSoerrn95 commented on GitHub (Aug 7, 2020): @RAYs3T Yes, it can make sense to use an API endpoint. You would then have to load all tags for each image, decide which should be used and fill the input / output tensors with them. It is also possible to load a model that has already been learned and then continue training. But this can lead to some side effects, because data from the original data set may contradict those that you added yourself. In this case it would be better to train a new model. The new or updated one could then be stored on a shared volume between the containers and then loaded from the GO container. But I'm not a TensorFlow pro either. I'm currently playing a bit with time series forecasting with RNNs and LSTMs. I've never worked with pictures.

deekerman commented

Author

Owner

@lastzero commented on GitHub (Aug 7, 2020):

Sounds like it makes most sense to implement this as a separate app / server. We already expose an API that can be used and extended if necessary.

Might be worth looking for existing software for this use case so that only some glue code needs to be developed instead of reinventing the wheel.

@lastzero commented on GitHub (Aug 7, 2020): Sounds like it makes most sense to implement this as a separate app / server. We already expose an API that can be used and extended if necessary. Might be worth looking for existing software for this use case so that only some glue code needs to be developed instead of reinventing the wheel.

deekerman commented

Author

Owner

@dekiesel commented on GitHub (Oct 19, 2020):

This library seems like a good fit, it is written in python though.

Users could manually tag a "few" pictures and then use those existing tags to train a model (using another container running wrapper code for this library) which is then used on the pictures that haven't been tagged.

The benefit of that approach is that every time a user corrects a matching the training data will improve.

The drawback is that it's another language than Go, which increases maintenance work.

@dekiesel commented on GitHub (Oct 19, 2020): [This library ](https://github.com/ageitgey/face_recognition) seems like a good fit, it is written in python though. Users could manually tag a "few" pictures and then use those existing tags to train a model (using another container running wrapper code for this library) which is then used on the pictures that haven't been tagged. The benefit of that approach is that every time a user corrects a matching the training data will improve. The drawback is that it's another language than Go, which increases maintenance work.

deekerman commented

Author

Owner

@danielo515 commented on GitHub (Nov 6, 2020):

@dekiesel that library looks very promising.
I would very happily use it ad-hoc, by running it through my image library and using the results to feed the tags (or create new ones) that exists on photorprism. Would that be possible with the existing API @lastzero ? Or I will have more luck just modifying the DB directly (hope not, hahah)

@danielo515 commented on GitHub (Nov 6, 2020): @dekiesel that library looks very promising. I would very happily use it ad-hoc, by running it through my image library and using the results to feed the tags (or create new ones) that exists on photorprism. Would that be possible with the existing API @lastzero ? Or I will have more luck just modifying the DB directly (hope not, hahah)

deekerman commented

Author

Owner

@lastzero commented on GitHub (Nov 8, 2020):

Model training should be done in a separately. It's beyond the scope of what we can maintain right now and also might require different programming languages like Python. The TensorFlow API for Go is not made for model training.

@lastzero commented on GitHub (Nov 8, 2020): Model training should be done in a separately. It's beyond the scope of what we can maintain right now and also might require different programming languages like Python. The TensorFlow API for Go is not made for model training.

deekerman commented

Author

Owner

@danielo515 commented on GitHub (Nov 9, 2020):

I am playing with a little proof of concept to, at least, add face recognition from the outside. I'm having a lot of fun.
I'll report back if I come with something usable.
It is already open source in any case if someone wants to contribute or continue it in case I reach the point where I can not do it.

@danielo515 commented on GitHub (Nov 9, 2020): I am playing with a little proof of concept to, at least, add face recognition from the outside. I'm having a lot of fun. I'll report back if I come with something usable. It is already open source in any case if someone wants to contribute or continue it in case I reach the point where I can not do it.

deekerman commented

Author

Owner

@kalon33 commented on GitHub (Sep 21, 2021):

@danielo515 Hi, any news from your work on this? Thanks :)

@kalon33 commented on GitHub (Sep 21, 2021): @danielo515 Hi, any news from your work on this? Thanks :)

deekerman commented

Author

Owner

@lastzero commented on GitHub (Sep 21, 2021):

Face detection & recognition has been added now. To easily use additional models, it would make sense to use a standardized API designed for this purpose. First step would be to do a bit of research, e.g. figure out if that already exists or somebody is working on it. No need to reinvent the wheel.

@lastzero commented on GitHub (Sep 21, 2021): Face detection & recognition has been added now. To easily use additional models, it would make sense to use a standardized API designed for this purpose. First step would be to do a bit of research, e.g. figure out if that already exists or somebody is working on it. No need to reinvent the wheel.

deekerman commented

Author

Owner

@danielo515 commented on GitHub (Sep 24, 2021):

@danielo515 Hi, any news from your work on this? Thanks :)

I have a MVP on my personal github projects. It is publicly available, I'll post a link later.
But if it has been official implemented maybe that project is not worth continuing. It depends on the limitations of the official implementation. Does it support tagging any people? If it does, then my project doesn't add anything to it

@danielo515 commented on GitHub (Sep 24, 2021): > @danielo515 Hi, any news from your work on this? Thanks :) I have a MVP on my personal github projects. It is publicly available, I'll post a link later. But if it has been official implemented maybe that project is not worth continuing. It depends on the limitations of the official implementation. Does it support tagging any people? If it does, then my project doesn't add anything to it

deekerman commented

Author

Owner

@lastzero commented on GitHub (Sep 24, 2021):

Faces are only automatically detected for now as manually selecting faces was more work for us and our users. The backend could deal with it though, just need a nice UI. May be part of a custom image viewer.

@lastzero commented on GitHub (Sep 24, 2021): Faces are only automatically detected for now as manually selecting faces was more work for us and our users. The backend could deal with it though, just need a nice UI. May be part of a custom image viewer.

deekerman commented

Author

Owner

@laurac8r commented on GitHub (Feb 8, 2022):

Any updates? I can work on retraining as I am an ML engineer by profession. Can someone link helpful docs for retraining? How is the Tensorflow model trained typically?

@laurac8r commented on GitHub (Feb 8, 2022): Any updates? I can work on retraining as I am an ML engineer by profession. Can someone link helpful docs for retraining? How is the Tensorflow model trained typically?

deekerman commented

Author

Owner

@lastzero commented on GitHub (Feb 8, 2022):

@yarocoder It's just too much more right now. Keep in mind that we also write all the documentation and provide support for 50,000+ users. I think the first step would be to study the options and provide a decision matrix for discussion.

Technical details of the implementation are documented in the Developer Guide:

https://docs.photoprism.app/developer-guide/metadata/classification/

The public roadmap shows which features we are currently working on:

https://github.com/photoprism/photoprism/projects/5

@lastzero commented on GitHub (Feb 8, 2022): @yarocoder It's just too much more right now. Keep in mind that we also write all the documentation and provide support for 50,000+ users. I think the first step would be to study the options and provide a decision matrix for discussion. Technical details of the implementation are documented in the Developer Guide: - https://docs.photoprism.app/developer-guide/metadata/classification/ The public roadmap shows which features we are currently working on: - https://github.com/photoprism/photoprism/projects/5

deekerman commented

Author

Owner

@mateuszdrab commented on GitHub (Jun 16, 2022):

I too am interested in the concept of training up the model or being able to use another model, hoping it would work better.

@mateuszdrab commented on GitHub (Jun 16, 2022): I too am interested in the concept of training up the model or being able to use another model, hoping it would work better.

deekerman commented

Author

Owner

@freman commented on GitHub (Jun 22, 2022):

One of the things that keeps me with google is it can find "license plate" or "Shrek" (my cat), I'm not opposed to training my own model, I've tagged about 300 photos... out of several thousand.

@freman commented on GitHub (Jun 22, 2022): One of the things that keeps me with google is it can find "license plate" or "Shrek" (my cat), I'm not opposed to training my own model, I've tagged about 300 photos... out of several thousand.

deekerman commented

Author

Owner

@abviv commented on GitHub (Jan 6, 2023):

@laurac8r I like this issue since it's been open for so long and my area is ML for vision, so I think I can also contribute to this effectively with my expertise. As a starter, I would investigate one major thing: the research focused on answering the questions of computation requirements (typically on a CPU), data requirements (how much data do I need?), and accuracy (which goes without saying).

@abviv commented on GitHub (Jan 6, 2023): @laurac8r I like this issue since it's been open for so long and my area is ML for vision, so I think I can also contribute to this effectively with my expertise. As a starter, I would investigate one major thing: the research focused on answering the questions of computation requirements (typically on a CPU), data requirements (how much data do I need?), and accuracy (which goes without saying).

deekerman commented

Author

Owner

@scarolan commented on GitHub (Aug 17, 2023):

+1 for allowing users to guide the model with suggestions. Not sure if this is even feasible but it would be amazing if you could crowdsource the human labor. Users could volunteer to submit their data and corrections to a central database which could be used to improve the experience for everyone.

Also please meet my pet 'Snail' Sunny. 🤣

@scarolan commented on GitHub (Aug 17, 2023): +1 for allowing users to guide the model with suggestions. Not sure if this is even feasible but it would be amazing if you could crowdsource the human labor. Users could volunteer to submit their data and corrections to a central database which could be used to improve the experience for everyone. Also please meet my pet 'Snail' Sunny. 🤣 ![snail](https://github.com/photoprism/photoprism/assets/403332/07299012-29ba-4b62-b92b-3d1a63e9517b)

deekerman commented