&#34;String contains invalid Unicode code points&#34; when scaning folders #4057

Author

Owner

@CVB-999999999 commented on GitHub (Mar 20, 2025):

Found the issue, I had a corrupted line in the TrackFiles table. Deleting the contents of TrackFiles and Tracks fixed the issue

@CVB-999999999 commented on GitHub (Mar 20, 2025): Found the issue, I had a corrupted line in the `TrackFiles` table. Deleting the contents of `TrackFiles` and `Tracks` fixed the issue

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 21, 2025):

Basically you nuked your whole library?

The issue might reappear as some file has some weird characters that triggers this issue.

@mynameisbogdan commented on GitHub (Mar 21, 2025): Basically you nuked your whole library? The issue might reappear as some file has some weird characters that triggers this issue.

deekerman commented

Author

Owner

@CVB-999999999 commented on GitHub (Mar 21, 2025):

I forgot to say this in my previous post, but I ran this command and found a single file that had a bad char

find . -type f -exec bash -c 'echo "{}" | iconv -f utf-8 -t utf-8//IGNORE > /dev/null 2>&1 || echo "{}"' \;

The problem is that after some testing I discovered that the problem was a bad record in the TrackFiles table, but at that point I already had closed the command line and didn't remember the file name and this version of SQLite does not have native regex capabilities.

On the bright side, Lidarr managed to match 95% of my library and the missing stuff was due to me marking some artists/albums as not monitored or my unmatched files. So I finally matched my unmatched files and went from about 400 files to 4.

@CVB-999999999 commented on GitHub (Mar 21, 2025): I forgot to say this in my previous post, but I ran this command and found a single file that had a bad char ```bash find . -type f -exec bash -c 'echo "{}" | iconv -f utf-8 -t utf-8//IGNORE > /dev/null 2>&1 || echo "{}"' \; ``` The problem is that after some testing I discovered that the problem was a bad record in the `TrackFiles` table, but at that point I already had closed the command line and didn't remember the file name and this version of SQLite does not have native regex capabilities. On the bright side, Lidarr managed to match 95% of my library and the missing stuff was due to me marking some artists/albums as not monitored or my unmatched files. So I finally matched my unmatched files and went from about 400 files to 4.

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 21, 2025):

Good to know. 👍

Next time it happens feel free to reopen this issue with the new details as I'm curious what's breaking Normalize since I've seen it happen on other arrs as well.

@mynameisbogdan commented on GitHub (Mar 21, 2025): Good to know. 👍 Next time it happens feel free to reopen this issue with the new details as I'm curious what's breaking `Normalize` since I've seen it happen on other arrs as well.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Mar 25, 2025):

@mynameisbogdan
I've had 4 recent downloaded albums give this same error.
I've been unable to find the problem character in the file or folder names.

What can I give you to help troubleshoot the issue?

@clowgg commented on GitHub (Mar 25, 2025): @mynameisbogdan I've had 4 recent downloaded albums give this same error. I've been unable to find the problem character in the file or folder names. What can I give you to help troubleshoot the issue?

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 25, 2025):

Trace logs containing the issue plus names for files or folders.

@mynameisbogdan commented on GitHub (Mar 25, 2025): Trace logs containing the issue plus names for files or folders.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Mar 25, 2025):

Here are the 3 logs that updated when I tried a Manual Import of one folder:

lidarr.trace.txt

Here is a dir listing of the folder:
dir.txt

@clowgg commented on GitHub (Mar 25, 2025): Here are the 3 logs that updated when I tried a Manual Import of one folder: [lidarr.trace.txt](https://github.com/user-attachments/files/19446123/lidarr.trace.txt) Here is a dir listing of the folder: [dir.txt](https://github.com/user-attachments/files/19446142/dir.txt)

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 25, 2025):

Do you use Rename Tracks? The issue seems to come from the track filenames stored in the database.

@mynameisbogdan commented on GitHub (Mar 25, 2025): Do you use `Rename Tracks`? The issue seems to come from the track filenames stored in the database.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Mar 25, 2025):

Yes - I do.

@clowgg commented on GitHub (Mar 25, 2025): Yes - I do. ![Image](https://github.com/user-attachments/assets/69186dd2-4fb1-47f1-b315-9fc4e899db1a)

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 26, 2025):

@clowgg can you test this docker image ghcr.io/linuxserver-labs/prarr:lidarr-media-file-track-path-normalize which is basically the build for PR #5420? It should log all the paths it's failing to compare.

@mynameisbogdan commented on GitHub (Mar 26, 2025): @clowgg can you test this docker image `ghcr.io/linuxserver-labs/prarr:lidarr-media-file-track-path-normalize` which is basically the build for PR #5420? It should log all the paths it's failing to compare.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Mar 26, 2025):

I use Lidarr on Windows. Is there a dev build of that I could try?

@clowgg commented on GitHub (Mar 26, 2025): I use Lidarr on Windows. Is there a dev build of that I could try?

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 26, 2025):

Then give this a try: https://lidarr.servarr.com/v1/update/media-file-track-path-normalize/updatefile?os=windows&runtime=netcore&arch=x64&installer=true

Since I just released the stable, you can revert to the stable version at any time.

@mynameisbogdan commented on GitHub (Mar 26, 2025): Then give this a try: https://lidarr.servarr.com/v1/update/media-file-track-path-normalize/updatefile?os=windows&runtime=netcore&arch=x64&installer=true Since I just released the stable, you can revert to the stable version at any time.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Mar 26, 2025):

Can I install that over the top of my current stable? Will it just 'upgrade' the current install in-place?

@clowgg commented on GitHub (Mar 26, 2025): Can I install that over the top of my current stable? Will it just 'upgrade' the current install in-place?

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 26, 2025):

Yes, you can. It's basically nightly + the changes in the PR. Just reinstall as you normally upgrade or downgrade.

@mynameisbogdan commented on GitHub (Mar 26, 2025): Yes, you can. It's basically nightly + the changes in the PR. Just reinstall as you normally upgrade or downgrade.

deekerman commented

lidarr.debug.txt
lidarr.txt
lidarr.trace.txt

Author

Owner

@clowgg commented on GitHub (Mar 26, 2025):

Got that new version installed fine. (Had to reset the service logon acct)

I changed the logs to Trace, and then started a Manual Import Interactive.
The Lidarr service seems to hang or go into a loop. Browsers can connect to the web site any longer.
Task Manager is showing a constant 25% CPU utilisation. The browser where I started the import can swap Lidarr pages - but no content ever displays on those pages.

The Lidarr service wouldn't stop once it was in this state, so I have to reboot the host.
Also had to modify the Inbound f/wall rule to include an additional '\bin' in the exe path: C:\programdata\lidarr\bin\bin\lidarr.console.exe

Here are the logs:

Interestingly, if I try the same import with Logs set to Info I get the same String error as above and the Lidaar service doesn't crash.

@clowgg commented on GitHub (Mar 26, 2025): Got that new version installed fine. (Had to reset the service logon acct) I changed the logs to Trace, and then started a Manual Import Interactive. The Lidarr service seems to hang or go into a loop. Browsers can connect to the web site any longer. Task Manager is showing a constant 25% CPU utilisation. The browser where I started the import can swap Lidarr pages - but no content ever displays on those pages. The Lidarr service wouldn't stop once it was in this state, so I have to reboot the host. Also had to modify the Inbound f/wall rule to include an additional '\bin' in the exe path: `C:\programdata\lidarr\bin\bin\lidarr.console.exe` Here are the logs: [lidarr.debug.txt](https://github.com/user-attachments/files/19476354/lidarr.debug.txt) [lidarr.txt](https://github.com/user-attachments/files/19476356/lidarr.txt) [lidarr.trace.txt](https://github.com/user-attachments/files/19476355/lidarr.trace.txt) Interestingly, if I try the same import with Logs set to Info I get the same String error as above and the Lidaar service doesn't crash.

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 27, 2025):

Did you used manual import?

Weird, I see no Get files with paths in the trace logs.

@mynameisbogdan commented on GitHub (Mar 27, 2025): Did you used manual import? Weird, I see no `Get files with paths` in the trace logs.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Mar 27, 2025):

Yes I did do a Manual Import.
I'll do another one now.

@clowgg commented on GitHub (Mar 27, 2025): Yes I did do a Manual Import. I'll do another one now.

deekerman commented

lidarr.trace.txt
lidarr.debug.txt

Author

Owner

@clowgg commented on GitHub (Mar 27, 2025):

I set logging to Trace, cleared the logs, and then tried a Manual Import of the same TinaT album.

The Lidarr service is now sitting at 25% CPU and is in a loop or hung as the web site is unresponsive.

@clowgg commented on GitHub (Mar 27, 2025): I set logging to Trace, cleared the logs, and then tried a Manual Import of the same TinaT album. The Lidarr service is now sitting at 25% CPU and is in a loop or hung as the web site is unresponsive. [lidarr.trace.txt](https://github.com/user-attachments/files/19493431/lidarr.trace.txt) [lidarr.debug.txt](https://github.com/user-attachments/files/19493432/lidarr.debug.txt)

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Mar 27, 2025):

Can't open the logs right now, but if there's no exception then there's no error reported. I changed the error in the build, I'll merge it most likely to be easier to debug in the future.

@mynameisbogdan commented on GitHub (Mar 27, 2025): Can't open the logs right now, but if there's no exception then there's no error reported. I changed the error in the build, I'll merge it most likely to be easier to debug in the future.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 1, 2025):

@mynameisbogdan Do you need any further info to help troubleshoot this issue?

@clowgg commented on GitHub (Apr 1, 2025): @mynameisbogdan Do you need any further info to help troubleshoot this issue?

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 2, 2025):

Well, sadly I still don't know the source of the issue.

I just added something to nightly builds to log the faulty paths, so I'm waiting for someone to report them.

@mynameisbogdan commented on GitHub (Apr 2, 2025): Well, sadly I still don't know the source of the issue. I just added something to nightly builds to log the faulty paths, so I'm waiting for someone to report them.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 2, 2025):

I’m happy to install the next build and re test, if you give me the link to download. Windows.

@clowgg commented on GitHub (Apr 2, 2025): I’m happy to install the next build and re test, if you give me the link to download. Windows.

deekerman commented

https://github.com/Lidarr/Lidarr/releases/tag/v2.11.0.4610

Author

Owner

@mynameisbogdan commented on GitHub (Apr 2, 2025):

You can just switch to the latest develop build, no more custom builds. 👍

@mynameisbogdan commented on GitHub (Apr 2, 2025): You can just switch to the latest develop build, no more custom builds. 👍 https://github.com/Lidarr/Lidarr/releases/tag/v2.11.0.4610

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 3, 2025):

Updated to 4610.
Tried Manual Import of previously failing album.

So that is interesting :)

@clowgg commented on GitHub (Apr 3, 2025): Updated to 4610. Tried Manual Import of previously failing album. ![Image](https://github.com/user-attachments/assets/f9dbc2f0-446e-435d-ac14-32ff476a163d) So that is interesting :)

deekerman commented

lidarr.debug.txt
lidarr.txt
lidarr.trace.txt

Author

Owner

@clowgg commented on GitHub (Apr 3, 2025):

Logs of the Manual Import:

@clowgg commented on GitHub (Apr 3, 2025): Logs of the Manual Import: [lidarr.debug.txt](https://github.com/user-attachments/files/19580120/lidarr.debug.txt) [lidarr.txt](https://github.com/user-attachments/files/19580121/lidarr.txt) [lidarr.trace.txt](https://github.com/user-attachments/files/19580119/lidarr.trace.txt)

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 3, 2025):

That seems to come from a NFS volume of your already imported music, correct? I wonder if this is related to the issue. I'm confused on why it's trying to compare the two of them, when the artists don't even match.

Try using Replace existing files.

@mynameisbogdan commented on GitHub (Apr 3, 2025): That seems to come from a NFS volume of your already imported music, correct? I wonder if this is related to the issue. I'm confused on why it's trying to compare the two of them, when the artists don't even match. Try using `Replace existing files`.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 6, 2025):

I get the same result:

@clowgg commented on GitHub (Apr 6, 2025): I get the same result: ![Image](https://github.com/user-attachments/assets/cae25860-467e-4abc-b8c4-6815788e893f)

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 7, 2025):

But do you have that specific path imported into your lidarr? Can't you see that artist/album to see what's up with it?

@mynameisbogdan commented on GitHub (Apr 7, 2025): But do you have that specific path imported into your lidarr? Can't you see that artist/album to see what's up with it?

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 7, 2025):

Lidarr knows this album:

You can see non-standard characters in track 10.

File Explorer view:

Directory export to txt file:

dir.txt

@clowgg commented on GitHub (Apr 7, 2025): Lidarr knows this album: ![Image](https://github.com/user-attachments/assets/0be1ef8b-e029-4d99-bd19-68b20ab77343) You can see non-standard characters in track 10. File Explorer view: ![Image](https://github.com/user-attachments/assets/af3c0fe0-197e-4e0a-a3b6-2ae5e6ebd525) Directory export to txt file: [dir.txt](https://github.com/user-attachments/files/19640311/dir.txt)

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 8, 2025):

Removing that one indicated Nena track from the Lidarr database solved the issue.
Hopefully you can determine why that track was being referenced during a Manual Import of a completely different Artist.

@clowgg commented on GitHub (Apr 8, 2025): Removing that one indicated Nena track from the Lidarr database solved the issue. Hopefully you can determine why that track was being referenced during a Manual Import of a completely different Artist.

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 8, 2025):

Hopefully you can determine why that track was being referenced during a Manual Import of a completely different Artist.

Yes, that's indeed something that we should look into.

I don't think you answered me, but that seems like a SMB share with broken UTF-8 support, correct?

Latest Lidarr has better support for removing diacritics for clean naming tokens, eg. ``{{Track CleanTitle}}`.

But the presence of cover.jpg which we don't support at the moment makes me think you didn't actually used the builtin import feature for this album? 🤔 Or are you using something 3rd-party for covers?

@mynameisbogdan commented on GitHub (Apr 8, 2025): > Hopefully you can determine why that track was being referenced during a Manual Import of a completely different Artist. Yes, that's indeed something that we should look into. I don't think you answered me, but that seems like a SMB share with broken UTF-8 support, correct? Latest Lidarr has better support for removing diacritics for clean naming tokens, eg. ``{{Track CleanTitle}}`. But the presence of `cover.jpg` which we don't support at the moment makes me think you didn't actually used the builtin import feature for this album? 🤔 Or are you using something 3rd-party for covers?

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 8, 2025):

All my music is stored on an SMB share. It’s on a Windows 2022 server. Lidarr runs on a separate Win10 workstation. Not sure what you mean by ‘broken UTF-8 support’?

I may have manually copied in a cover.jpg file after the album was imported.

@clowgg commented on GitHub (Apr 8, 2025): All my music is stored on an SMB share. It’s on a Windows 2022 server. Lidarr runs on a separate Win10 workstation. Not sure what you mean by ‘broken UTF-8 support’? I may have manually copied in a cover.jpg file after the album was imported.

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 8, 2025):

I mean it's a quite common issue with SMB. See https://serverfault.com/questions/124611/special-characters-in-samba-filenames as mere example.

But still, maybe I'm focusing on the wrong issue. It still could be an issue coming from the sqlite database returning bad characters or is just what Lidarr could read from a file listing. I have yet to reproduce this locally.

@mynameisbogdan commented on GitHub (Apr 8, 2025): I mean it's a quite common issue with SMB. See https://serverfault.com/questions/124611/special-characters-in-samba-filenames as mere example. But still, maybe I'm focusing on the wrong issue. It still could be an issue coming from the sqlite database returning bad characters or is just what Lidarr could read from a file listing. I have yet to reproduce this locally.

deekerman commented

Author

Owner

@clowgg commented on GitHub (Apr 8, 2025):

A Windows server obviously doesn't user Samba, and the client is Windows too, so I wouldn't have thought there would be a file name translation issue between them.

Remember that this issue only came up on some album imports. Other albums would import successfully.
Please let me know if you need anything more from my system to help with your troubleshooting.

@clowgg commented on GitHub (Apr 8, 2025): A Windows server obviously doesn't user Samba, and the client is Windows too, so I wouldn't have thought there would be a file name translation issue between them. Remember that this issue only came up on _some_ album imports. Other albums would import successfully. Please let me know if you need anything more from my system to help with your troubleshooting.

deekerman commented

Author

Owner

@valemaio2 commented on GitHub (Apr 28, 2025):

Hi, I'm having the very same issue. Running Lidarr in Docker on Debian 12. I'm attaching the relevant Lidarr log, please do let me know what else you need to investigate this.

lidarr.txt

@valemaio2 commented on GitHub (Apr 28, 2025): Hi, I'm having the very same issue. Running Lidarr in Docker on Debian 12. I'm attaching the relevant Lidarr log, please do let me know what else you need to investigate this. [lidarr.txt](https://github.com/user-attachments/files/19936401/lidarr.txt)

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 28, 2025):

Info logs aren't useful to debug anything on the matter.

Wait a few hours for a new stable release, and then update in order to get the path with the issue in the error.

@mynameisbogdan commented on GitHub (Apr 28, 2025): Info logs aren't useful to debug anything on the matter. Wait a few hours for a new stable release, and then update in order to get the path with the issue in the error.

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 28, 2025):

v2.11.2.4629 was released.

Update and share trace logs with the issue at hand.

@mynameisbogdan commented on GitHub (Apr 28, 2025): v2.11.2.4629 was released. Update and share trace logs with the issue at hand.

deekerman commented

Author

Owner

@valemaio2 commented on GitHub (Apr 28, 2025):

lidarr.debug.txt

v2.11.2.4629 was released.

Update and share trace logs with the issue at hand.

Thanks for the new update, please find the debug log. There seems to be a song with an emoji in the title:

[v2.11.2.4629] System.ArgumentException: Invalid path: /music/2021 - Infinity Station Sessions/02 ❤️ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3

Would renaming the song be enough or something else needs doing as well?

EDIT: I'm answering my own question here. I've replaced the heart emoji with the equivalent Unicode character ♡ and Lidarr is finally able to scan the folders once again.

@valemaio2 commented on GitHub (Apr 28, 2025): [lidarr.debug.txt](https://github.com/user-attachments/files/19940304/lidarr.debug.txt) > v2.11.2.4629 was released. > > Update and share trace logs with the issue at hand. Thanks for the new update, please find the debug log. There seems to be a song with an emoji in the title: [v2.11.2.4629] System.ArgumentException: Invalid path: /music/2021 - Infinity Station Sessions/02 ❤️ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3 Would renaming the song be enough or something else needs doing as well? EDIT: I'm answering my own question here. I've replaced the heart emoji with the equivalent Unicode character ♡ and Lidarr is finally able to scan the folders once again.

deekerman commented

Author

Owner

@mynameisbogdan commented on GitHub (Apr 28, 2025):

Good. Also debug logs are not trace logs.

So the path was something like /music/2021 - Infinity Station Sessions/02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?

@mynameisbogdan commented on GitHub (Apr 28, 2025): Good. Also debug logs are not trace logs. So the path was something like /music/2021 - Infinity Station Sessions/02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?

deekerman commented

Author

Owner

@valemaio2 commented on GitHub (Apr 28, 2025):

Good. Also debug logs are not trace logs.

So the path was something like /music/2021 - Infinity Station Sessions/02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?

Sorry, I missed the trace part of the logs. That was the patch indeed, and can confirm Lidarr is still working as expected after replacing that non-Unicode character.

@valemaio2 commented on GitHub (Apr 28, 2025): > Good. Also debug logs are not trace logs. > > So the path was something like /music/2021 - Infinity Station Sessions/02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3? Sorry, I missed the trace part of the logs. That was the patch indeed, and can confirm Lidarr is still working as expected after replacing that non-Unicode character.

deekerman commented

Author

Owner

@HirschBerge commented on GitHub (Jan 27, 2026):

Just wanted to add onto this issue, hopefully it'll help someone else and prevent them from banging their head into the desk like I did for a few hours.

This seems to also affect metadata. I had to go in and wipe existing metadata off a few Ado albums before I could import them.

The trace/debug output is atrocious and sorely needs to be made useful.

@HirschBerge commented on GitHub (Jan 27, 2026): Just wanted to add onto this issue, hopefully it'll help someone else and prevent them from banging their head into the desk like I did for a few hours. This seems to also affect metadata. I had to go in and wipe existing metadata off a few Ado albums before I could import them. The trace/debug output is atrocious and sorely needs to be made useful.

deekerman commented

Author

Owner

@industrial64 commented on GitHub (Feb 4, 2026):

Still an issue - only noticed when I went from v2.14.5.4836 to v3.1.0.4875. So either a regressed bug that was fixed previously, or a still-unsolved bug with files that have a bad character. I also had this exact issue with this exact file too: "02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?" I do not have 'rename tracks' enabled.
Tried removing the artist, re-naming the file, re-adding the artist, clearing events = still not repaired, and nothing can be imported, as any other items in the queue strangely presents this same error, with this same file.
I guess we are expected to delete or modify the database to get things working again?

@industrial64 commented on GitHub (Feb 4, 2026): Still an issue - only noticed when I went from v2.14.5.4836 to v3.1.0.4875. So either a regressed bug that was fixed previously, or a still-unsolved bug with files that have a bad character. I also had this exact issue with this exact file too: "02 ❤ (Feat. We Are King And Jacob Collier) (Infinity Station Sessions).mp3?" I do not have 'rename tracks' enabled. Tried removing the artist, re-naming the file, re-adding the artist, clearing events = still not repaired, and nothing can be imported, as any other items in the queue strangely presents this same error, with this same file. I guess we are expected to delete or modify the database to get things working again?

deekerman commented