starred/Lidarr-Lidarr

Fork 0

mirror of https://github.com/Lidarr/Lidarr.git synced 2026-03-03 00:26:58 -05:00

Rescans Accumulate Files Due To Incorrect DB Modification Timestamp #3255

New issue

Open

opened 2026-02-20 13:07:31 -05:00 by deekerman · 2 comments

deekerman commented

2026-02-20 13:07:31 -05:00

Owner

Originally created by @thegranddesign on GitHub (Dec 5, 2023).

Is there an existing issue for this?

I have searched the existing open and closed issues

Current Behavior

Summary

After a rescan, it appears as though the Modified column in the TrackFiles table does not get properly updated. In fact it's the DateAdded column and not the Modified column that more closely reflects the file's modified timestamp.

Details

After seeing a lot of Removing track file from DB as part of manual override of existing file log lines, I dove into the code and saw that it appears to be removing the TrackFiles record and readding it. When a track is re-added to the TrackFiles table, the DateAdded is the inserted time (which is correct), but the Modified date is retained from the removed record. This would normally be fine, but since the re-adding is also (often) accompanied with a re-tagging, the file's modification timestamp and the DB's modification field get out of sync.

This results in subsequent rescans picking up the file again since it doesn't see it as unwanted.

And because the rescan reimports it (which causes another retag and removal/addition to the TrackFiles table), the track is from then on always marked for rescan.

This ultimately results in rescans taking a very, very long time on large libraries.

When I fixed the modification timestamps in the DB to reflect the file's values, my rescans went from taking over 28 hours down to about 34 seconds.

Expected Behavior

After a rescan, the TrackFiles table's records should have a Modified column value that reflects the file's modification timestamp.

Steps To Reproduce

This is what I did to prove this was the issue:

Copied the DB
Searched for a problematic track
Copied down the timestamps
Compared them to the file's timestamps
Executed a manual rescan
Recopied the DB
Researched for the problematic track
Recopied down the timestamps
Compared all the previous timestamps to the current file's timestamps

Additionally, to test that the modification dates fixed the problem, I did the following:

Turn on trace logging
Run the "Rescan Folders" task manually
Look at the trace logs and look for the lines like:

Debug|MediaFileService|Filtering 123456 files for unchanged files
Trace|MediaFileService|Got 123456 existing files
Trace|MediaFileService|Matched paths for 123456 files
Trace|MediaFileService|123456 unchanged existing files
Debug|ImportDecisionMaker|Analyzing 123456/123456 files.

Make a note of the numbers in the log there. These are the starting values
Stop Lidarr
Delete all current logs (eg rm -f /path/to/your/lidarr/install/logs/*.txt)
Run sqlite3 /path/to/your/lidarr.db 'UPDATE "TrackFiles" SET "Modified" = "DateAdded";'
Start Lidarr
Run the "Rescan Folders" task manually again
Look at the trace logs and look for the same lines I looked for before

At the end the numbers were significantly better (though not exact since the DateAdded wouldn't always be within a second of the file's modification timestamp.

Environment

- OS: FreeBSD
- Lidarr: 1.3.5.3530
- Docker Install: N/A
- Using Reverse Proxy: No
- Browser: N/A
- Database: Sqlite 3.43.1

What branch are you running?

Master

Trace Logs?

See Reproduction Steps

Trace Logs have been provided as applicable. Reports may be closed if the required logs are not provided.

I have read and followed the steps in the wiki link above and provided the required trace logs - the logs contain trace - that are relevant and show this issue.

Originally created by @thegranddesign on GitHub (Dec 5, 2023). ### Is there an existing issue for this? - [X] I have searched the existing open and closed issues ### Current Behavior Summary ------------------------------------------------------------------------ After a rescan, it appears as though the `Modified` column in the `TrackFiles` table does not get properly updated. In fact it's the `DateAdded` column and _not_ the `Modified` column that more closely reflects the file's modified timestamp. Details ------------------------------------------------------------------------ After seeing a lot of `Removing track file from DB as part of manual override of existing file` log lines, I dove into the code and saw that it appears to be removing the `TrackFiles` record and readding it. When a track is re-added to the `TrackFiles` table, the `DateAdded` is the inserted time (which is correct), but the `Modified` date is retained from the removed record. This would normally be fine, but since the re-adding is also (often) accompanied with a re-_tagging_, the file's modification timestamp and the DB's modification field get out of sync. This results in subsequent rescans picking up the file again since it doesn't see it as [unwanted](https://github.com/Lidarr/Lidarr/blob/803c2dd66b97806a28a3ec93a9924c52bbb4906c/src/NzbDrone.Core/MediaFiles/MediaFileService.cs#L129-L134). And because the rescan reimports it (which causes another retag and removal/addition to the `TrackFiles` table), the track is from then on always marked for rescan. This ultimately results in rescans taking a very, very long time on large libraries. When I fixed the modification timestamps in the DB to reflect the file's values, my rescans went from taking over 28 hours down to about 34 seconds. ### Expected Behavior After a rescan, the `TrackFiles` table's records should have a `Modified` column value that reflects the file's modification timestamp. ### Steps To Reproduce This is what _I_ did to prove this was the issue: 1. Copied the DB 2. Searched for a problematic track 3. Copied down the timestamps 4. Compared them to the file's timestamps 5. Executed a manual rescan 6. Recopied the DB 7. Researched for the problematic track 8. Recopied down the timestamps 9. Compared all the previous timestamps to the current file's timestamps Additionally, to test that the modification dates fixed the problem, I did the following: 1. Turn on `trace` logging 2. Run the "Rescan Folders" task manually 3. Look at the trace logs and look for the lines like: ``` Debug|MediaFileService|Filtering 123456 files for unchanged files Trace|MediaFileService|Got 123456 existing files Trace|MediaFileService|Matched paths for 123456 files Trace|MediaFileService|123456 unchanged existing files Debug|ImportDecisionMaker|Analyzing 123456/123456 files. ``` 4. Make a note of the numbers in the log there. These are the starting values 5. Stop Lidarr 6. Delete all current logs (eg `rm -f /path/to/your/lidarr/install/logs/*.txt`) 7. Run `sqlite3 /path/to/your/lidarr.db 'UPDATE "TrackFiles" SET "Modified" = "DateAdded";'` 8. Start Lidarr 9. Run the "Rescan Folders" task manually again 10. Look at the trace logs and look for the same lines I looked for before At the end the numbers were significantly better (though not exact since the `DateAdded` wouldn't always be within a second of the file's modification timestamp. ### Environment ```markdown - OS: FreeBSD - Lidarr: 1.3.5.3530 - Docker Install: N/A - Using Reverse Proxy: No - Browser: N/A - Database: Sqlite 3.43.1 ``` ### What branch are you running? Master ### Trace Logs? See Reproduction Steps ### Trace Logs have been provided as applicable. Reports may be closed if the required logs are not provided. - [X] I have read and followed the steps in the wiki link above and provided the required trace logs - the logs contain `trace` - that are relevant and show this issue.

deekerman added the

Type: Bug

Status: Needs Triage

labels

2026-02-20 13:07:31 -05:00

deekerman commented

2026-02-20 13:07:32 -05:00

Author

Owner

@thegranddesign commented on GitHub (Dec 5, 2023):

Once these issues are fixed I believe it will resolve #3450

@thegranddesign commented on GitHub (Dec 5, 2023): Once these issues are fixed I believe it will resolve #3450

deekerman commented

2026-02-20 13:07:32 -05:00

Author

Owner

@thegranddesign commented on GitHub (Dec 6, 2023):

Also if it will help with debugging, let me know if you want me to attach my config table and XML values.

@thegranddesign commented on GitHub (Dec 6, 2023): Also if it will help with debugging, let me know if you want me to attach my config table and XML values.