mirror of
https://github.com/Lidarr/Lidarr.git
synced 2026-03-02 22:56:57 -05:00
"String contains invalid Unicode code points" when scaning folders #4176
Labels
No labels
Area: API
Area: Database
Area: Db-migration
Area: Download Clients
Area: Extras
Area: Import Lists
Area: Indexer
Area: Metadata API
Area: Notifications
Area: Organizer
Area: Parser
Area: Scanning
Area: Tooling
Area: UI
Area: Unit Tests
Area: Update API
On Hold: MetadataAPI Blocking
Priority: High
Priority: Low
Priority: Medium
Status: Accepted
Status: Cannot Reproduce
Status: Confirmed
Status: Don't Merge
Status: Help Wanted
Status: In Progress
Status: Info Needed
Status: Investigating
Status: Logs Needed
Status: Maybe One Day
Status: Needs Triage
Status: On Hold
Status: Ready for Review
Status: Unlikely
Status: Waiting for OP
Status: Won't Fix
Type: Bug
Type: Documentation
Type: Duplicate
Type: Enhancement
Type: Enhancement
Type: External Bug
Type: Feature Request
Type: Regression
Type: Support
Type: Support.
conflict
conflict
no-conflict
not-pulled
radarr-pull
readarr-pull
sonarr upstream
sonarr-pull
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/Lidarr#4176
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @CVB-999999999 on GitHub (Mar 19, 2025).
Is there an existing issue for this?
Current Behavior
When I try to do a folder scan it fails with the following error
I first noticed this behaviour in late December when I needed to do a manual import. At the time I ignored it as regular imports still work
After a bit of searching I think it might be related with https://github.com/Lidarr/Lidarr/issues/5236. I probably have some file with invalid chars that breaks this validation, but the log does not say what is the invalid String
Expected Behavior
Should do a search for new and updated files
Steps To Reproduce
Do one of the following:
Environment
What branch are you running?
Master
Trace Logs?
2025-03-19 20:06:41.1|Debug|DiskScanService|Scanning '/music/' for music files
2025-03-19 20:06:42.4|Trace|DiskScanService|9937 files were found in /music/
2025-03-19 20:06:42.4|Debug|DiskScanService|9235 audio files were found in /music/
2025-03-19 20:06:42.5|Debug|DiskScanService|Cleaning up media files in DB [/music/]
2025-03-19 20:06:42.6|Trace|CommandQueueManager|Updating command status
2025-03-19 20:06:42.6|Trace|EventAggregator|Publishing CommandUpdatedEvent
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandUpdatedEvent -> CommandController
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandUpdatedEvent <- CommandController
2025-03-19 20:06:42.6|Trace|EventAggregator|Publishing CommandExecutedEvent
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandExecutedEvent -> TaskManager
2025-03-19 20:06:42.6|Trace|TaskManager|Updating last run time for: NzbDrone.Core.MediaFiles.Commands.RescanFoldersCommand
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandExecutedEvent <- TaskManager
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandExecutedEvent -> TaskController
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandExecutedEvent <- TaskController
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandExecutedEvent -> QualityDefinitionController
2025-03-19 20:06:42.6|Trace|EventAggregator|CommandExecutedEvent <- QualityDefinitionController
2025-03-19 20:06:42.6|Trace|CommandExecutor|RescanFoldersCommand <- DiskScanService [00:00:01.4548392]
2025-03-19 20:06:42.6|Debug|Lidarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2025-03-19 20:06:42.6|Debug|Lidarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2025-03-19 20:06:42.6|Trace|CommandQueueManager|Getting all commands
2025-03-19 20:06:42.6|Trace|CommandQueueManager|Getting all commands
2025-03-19 20:06:42.6|Trace|Http|Res: 717 [GET] /api/v1/command: 200.OK (2 ms)
2025-03-19 20:06:42.6|Trace|Http|Res: 718 [GET] /api/v1/command: 200.OK (2 ms)
2025-03-19 20:06:42.6|Debug|Api|[GET] /api/v1/command: 200.OK (2 ms)
2025-03-19 20:06:42.6|Debug|Api|[GET] /api/v1/command: 200.OK (2 ms)
2025-03-19 20:06:42.6|Error|CommandExecutor|Error occurred while executing task RescanFolders
[v2.9.6.4552] System.ArgumentException: String contains invalid Unicode code points. (Parameter 'strInput')
at System.Globalization.Normalization.ValidateArguments(String strInput, NormalizationForm normalizationForm)
at System.Globalization.Normalization.IcuNormalize(String strInput, NormalizationForm normalizationForm)
at System.String.Normalize()
at NzbDrone.Common.PathEqualityComparer.GetHashCode(String obj) in ./Lidarr.Common/PathEqualityComparer.cs:line 27
at System.Collections.Generic.HashSet
1.AddIfNotPresent(T value, Int32& location) at System.Collections.Generic.HashSet1.UnionWith(IEnumerable1 other) at System.Collections.Generic.HashSet1..ctor(IEnumerable1 collection, IEqualityComparer1 comparer)at NzbDrone.Common.Extensions.EnumerableExtensions.ExceptBy[TFirst,TSecond,TKey](IEnumerable
1 first, Func2 firstKeySelector, IEnumerable1 second, Func2 secondKeySelector, IEqualityComparer1 keyComparer)+MoveNext() in ./Lidarr.Common/Extensions/IEnumerableExtensions.cs:line 35 at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)at NzbDrone.Core.MediaFiles.MediaFileTableCleanupService.Clean(String folder, List
1 filesOnDisk) in ./Lidarr.Core/MediaFiles/MediaFileTableCleanupService.cs:line 35 at NzbDrone.Core.MediaFiles.DiskScanService.Scan(List1 folders, FilterFilesType filter, Boolean addNewArtists, List`1 artistIds)at NzbDrone.Core.MediaFiles.DiskScanService.Execute(RescanFoldersCommand message) in ./Lidarr.Core/MediaFiles/DiskScanService.cs:line 282
at NzbDrone.Core.Messaging.Commands.CommandExecutor.ExecuteCommand[TCommand](TCommand command, CommandModel commandModel) in ./Lidarr.Core/Messaging/Commands/CommandExecutor.cs:line 115
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid3[T0,T1,T2](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at NzbDrone.Core.Messaging.Commands.CommandExecutor.ExecuteCommands() in ./Lidarr.Core/Messaging/Commands/CommandExecutor.cs:line 42
Trace Logs have been provided as applicable. Reports may be closed if the required logs are not provided.
trace- that are relevant and show this issue.@CVB-999999999 commented on GitHub (Mar 20, 2025):
Found the issue, I had a corrupted line in the
TrackFilestable. Deleting the contents ofTrackFilesandTracksfixed the issue@mynameisbogdan commented on GitHub (Mar 21, 2025):
Basically you nuked your whole library?
The issue might reappear as some file has some weird characters that triggers this issue.
@CVB-999999999 commented on GitHub (Mar 21, 2025):
I forgot to say this in my previous post, but I ran this command and found a single file that had a bad char
The problem is that after some testing I discovered that the problem was a bad record in the
TrackFilestable, but at that point I already had closed the command line and didn't remember the file name and this version of SQLite does not have native regex capabilities.On the bright side, Lidarr managed to match 95% of my library and the missing stuff was due to me marking some artists/albums as not monitored or my unmatched files. So I finally matched my unmatched files and went from about 400 files to 4.
@mynameisbogdan commented on GitHub (Mar 21, 2025):
Good to know. 👍
Next time it happens feel free to reopen this issue with the new details as I'm curious what's breaking
Normalizesince I've seen it happen on other arrs as well.@clowgg commented on GitHub (Mar 25, 2025):
@mynameisbogdan
I've had 4 recent downloaded albums give this same error.
I've been unable to find the problem character in the file or folder names.
What can I give you to help troubleshoot the issue?
@mynameisbogdan commented on GitHub (Mar 25, 2025):
Trace logs containing the issue plus names for files or folders.
@clowgg commented on GitHub (Mar 25, 2025):
Here are the 3 logs that updated when I tried a Manual Import of one folder:
lidarr.trace.txt
Here is a dir listing of the folder:
dir.txt
@mynameisbogdan commented on GitHub (Mar 25, 2025):
Do you use
Rename Tracks? The issue seems to come from the track filenames stored in the database.@clowgg commented on GitHub (Mar 25, 2025):
Yes - I do.
@mynameisbogdan commented on GitHub (Mar 26, 2025):
@clowgg can you test this docker image
ghcr.io/linuxserver-labs/prarr:lidarr-media-file-track-path-normalizewhich is basically the build for PR #5420? It should log all the paths it's failing to compare.@clowgg commented on GitHub (Mar 26, 2025):
I use Lidarr on Windows. Is there a dev build of that I could try?
@mynameisbogdan commented on GitHub (Mar 26, 2025):
Then give this a try: https://lidarr.servarr.com/v1/update/media-file-track-path-normalize/updatefile?os=windows&runtime=netcore&arch=x64&installer=true
Since I just released the stable, you can revert to the stable version at any time.
@clowgg commented on GitHub (Mar 26, 2025):
Can I install that over the top of my current stable? Will it just 'upgrade' the current install in-place?
@mynameisbogdan commented on GitHub (Mar 26, 2025):
Yes, you can. It's basically nightly + the changes in the PR. Just reinstall as you normally upgrade or downgrade.
@clowgg commented on GitHub (Mar 26, 2025):
Got that new version installed fine. (Had to reset the service logon acct)
I changed the logs to Trace, and then started a Manual Import Interactive.
The Lidarr service seems to hang or go into a loop. Browsers can connect to the web site any longer.
Task Manager is showing a constant 25% CPU utilisation. The browser where I started the import can swap Lidarr pages - but no content ever displays on those pages.
The Lidarr service wouldn't stop once it was in this state, so I have to reboot the host.
Also had to modify the Inbound f/wall rule to include an additional '\bin' in the exe path:
C:\programdata\lidarr\bin\bin\lidarr.console.exeHere are the logs:
lidarr.debug.txt
lidarr.txt
lidarr.trace.txt
Interestingly, if I try the same import with Logs set to Info I get the same String error as above and the Lidaar service doesn't crash.
@mynameisbogdan commented on GitHub (Mar 27, 2025):
Did you used manual import?
Weird, I see no
Get files with pathsin the trace logs.@clowgg commented on GitHub (Mar 27, 2025):
Yes I did do a Manual Import.
I'll do another one now.
@clowgg commented on GitHub (Mar 27, 2025):
I set logging to Trace, cleared the logs, and then tried a Manual Import of the same TinaT album.
The Lidarr service is now sitting at 25% CPU and is in a loop or hung as the web site is unresponsive.
lidarr.trace.txt
lidarr.debug.txt
@mynameisbogdan commented on GitHub (Mar 27, 2025):
Can't open the logs right now, but if there's no exception then there's no error reported. I changed the error in the build, I'll merge it most likely to be easier to debug in the future.
@clowgg commented on GitHub (Apr 1, 2025):
@mynameisbogdan Do you need any further info to help troubleshoot this issue?
@mynameisbogdan commented on GitHub (Apr 2, 2025):
Well, sadly I still don't know the source of the issue.
I just added something to nightly builds to log the faulty paths, so I'm waiting for someone to report them.
@clowgg commented on GitHub (Apr 2, 2025):
I’m happy to install the next build and re test, if you give me the link to download. Windows.
@mynameisbogdan commented on GitHub (Apr 2, 2025):
You can just switch to the latest develop build, no more custom builds. 👍
https://github.com/Lidarr/Lidarr/releases/tag/v2.11.0.4610
@clowgg commented on GitHub (Apr 3, 2025):
Updated to 4610.
Tried Manual Import of previously failing album.
So that is interesting :)
@clowgg commented on GitHub (Apr 3, 2025):
Logs of the Manual Import:
lidarr.debug.txt
lidarr.txt
lidarr.trace.txt
@mynameisbogdan commented on GitHub (Apr 3, 2025):
That seems to come from a NFS volume of your already imported music, correct? I wonder if this is related to the issue. I'm confused on why it's trying to compare the two of them, when the artists don't even match.
Try using
Replace existing files.@clowgg commented on GitHub (Apr 6, 2025):
I get the same result:
@mynameisbogdan commented on GitHub (Apr 7, 2025):
But do you have that specific path imported into your lidarr? Can't you see that artist/album to see what's up with it?
@clowgg commented on GitHub (Apr 7, 2025):
Lidarr knows this album:
You can see non-standard characters in track 10.
File Explorer view:
Directory export to txt file:
dir.txt
@clowgg commented on GitHub (Apr 8, 2025):
Removing that one indicated Nena track from the Lidarr database solved the issue.
Hopefully you can determine why that track was being referenced during a Manual Import of a completely different Artist.
@mynameisbogdan commented on GitHub (Apr 8, 2025):
Yes, that's indeed something that we should look into.
I don't think you answered me, but that seems like a SMB share with broken UTF-8 support, correct?
Latest Lidarr has better support for removing diacritics for clean naming tokens, eg. ``{{Track CleanTitle}}`.
But the presence of
cover.jpgwhich we don't support at the moment makes me think you didn't actually used the builtin import feature for this album? 🤔 Or are you using something 3rd-party for covers?@clowgg commented on GitHub (Apr 8, 2025):
All my music is stored on an SMB share. It’s on a Windows 2022 server. Lidarr runs on a separate Win10 workstation. Not sure what you mean by ‘broken UTF-8 support’?
I may have manually copied in a cover.jpg file after the album was imported.
@mynameisbogdan commented on GitHub (Apr 8, 2025):
I mean it's a quite common issue with SMB. See https://serverfault.com/questions/124611/special-characters-in-samba-filenames as mere example.
But still, maybe I'm focusing on the wrong issue. It still could be an issue coming from the sqlite database returning bad characters or is just what Lidarr could read from a file listing. I have yet to reproduce this locally.
@clowgg commented on GitHub (Apr 8, 2025):
A Windows server obviously doesn't user Samba, and the client is Windows too, so I wouldn't have thought there would be a file name translation issue between them.
Remember that this issue only came up on some album imports. Other albums would import successfully.
Please let me know if you need anything more from my system to help with your troubleshooting.
@valemaio2 commented on GitHub (Apr 28, 2025):
Hi, I'm having the very same issue. Running Lidarr in Docker on Debian 12. I'm attaching the relevant Lidarr log, please do let me know what else you need to investigate this.
lidarr.txt
@mynameisbogdan commented on GitHub (Apr 28, 2025):
Info logs aren't useful to debug anything on the matter.
Wait a few hours for a new stable release, and then update in order to get the path with the issue in the error.
@mynameisbogdan commented on GitHub (Apr 28, 2025):
v2.11.2.4629 was released.
Update and share trace logs with the issue at hand.
@valemaio2 commented on GitHub (Apr 28, 2025):
lidarr.debug.txt
Thanks for the new update, please find the debug log. There seems to be a song with an emoji in the title:
[v2.11.2.4629] System.ArgumentException: Invalid path: /music/2021 - Infinity Station Sessions/02 ❤️ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3
Would renaming the song be enough or something else needs doing as well?
EDIT: I'm answering my own question here. I've replaced the heart emoji with the equivalent Unicode character ♡ and Lidarr is finally able to scan the folders once again.
@mynameisbogdan commented on GitHub (Apr 28, 2025):
Good. Also debug logs are not trace logs.
So the path was something like /music/2021 - Infinity Station Sessions/02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?
@valemaio2 commented on GitHub (Apr 28, 2025):
Sorry, I missed the trace part of the logs. That was the patch indeed, and can confirm Lidarr is still working as expected after replacing that non-Unicode character.
@HirschBerge commented on GitHub (Jan 27, 2026):
Just wanted to add onto this issue, hopefully it'll help someone else and prevent them from banging their head into the desk like I did for a few hours.
This seems to also affect metadata. I had to go in and wipe existing metadata off a few Ado albums before I could import them.
The trace/debug output is atrocious and sorely needs to be made useful.
@industrial64 commented on GitHub (Feb 4, 2026):
Still an issue - only noticed when I went from v2.14.5.4836 to v3.1.0.4875. So either a regressed bug that was fixed previously, or a still-unsolved bug with files that have a bad character. I also had this exact issue with this exact file too: "02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?" I do not have 'rename tracks' enabled.
Tried removing the artist, re-naming the file, re-adding the artist, clearing events = still not repaired, and nothing can be imported, as any other items in the queue strangely presents this same error, with this same file.
I guess we are expected to delete or modify the database to get things working again?
@industrial64 commented on GitHub (Feb 4, 2026):
Was able to fix in SQL browser by searching the partial name and deleting an entry that was erroneously filled with 'NULL'.