After way too much time investigating this, the encoding situation is
not great right now.
- Stable sets the "default code page" to be used for encoding filenames
to Shift-JIS (932):
c29ebd7fc5/osu!/GameBase.cs#L3099
- Lazer does nothing (therefore using UTF-8).
When importing to lazer, stable files are assumed to be UTF-8. This
means that the linked beatmaps don't work correctly. Forcing lazer to
decompress *and* compress using Shift-JIS will fix this.
Here's a rough idea of how things look for japanese character filenames
in current `master`:
| | stable | lazer |
|--------|--------|--------|
| export encoding | shift-jis | utf8 |
| utf8 [bit flag](https://superuser.com/a/1507988) set | ❌ | ❌ |
| import stable export osz | ✅ | ❌ |
| import lazer export osz | ❌ | ✅ |
| windows unzip | ❌ | ❌ |
| macos unzip | ✅ | ✅ |
and after this change
| | stable | lazer |
|--------|--------|--------|
| export encoding | shift-jis | shift-jis |
| utf8 [bit flag](https://superuser.com/a/1507988) set | ❌ | ❌ |
| import stable export osz | ✅ | ✅ |
| import lazer export osz | ✅ | ✅ |
| windows unzip | ❌ | ❌ |
| macos unzip | ✅ | ✅ |
A future endeavour to improve compatibility would be to look at setting
the utf8 flag in lazer, switching the default to utf8, and ensuring the
stable supports this flag (I don't believe it does right now).
Closes https://github.com/ppy/osu/issues/27540.
As it turns out, some ZIP archivers (like 7zip) will decide to add fake
entries for directories, and some (like windows zipfolders) won't.
The "directory" entries aren't really properly supported using any
actual data or attributes, they're detected by sharpcompress basically
by heuristics:
ab5535eba3/src/SharpCompress/Common/Zip/Headers/ZipFileEntry.cs (L19-L31)
When importing into realm we have thus far presumed that these directory
entries will not be a thing. Having them be a thing breaks multiple
things, like:
- When importing from ZIPs with separate directory entries, a separate
`RealmFile` is created for a directory entry even though it doesn't
represent a real file
- As a result, when re-exporting a model with files imported from such
an archive, a zero-byte file would be created because to the database
it looks like it was originally a zero-byte file.
If you want to have fun, google "zip empty directories". You'll see
a whole gamut of languages, libraries, and developers stepping on this
rake. Yet another episode of underspecced mistakes from decades ago
that were somebody's "good idea" but continue to wreak havoc forevermore
because now there are two competing conventions you can't just pick one.
No longer throw if copying of single files fails during data migration.
Aiming to fix https://github.com/ppy/osu/runs/7601653833?check_suite_focus=true, which could also affect end users. I've left a limit before an exception is still thrown, to handle cases like the user running out of disk space (where we probably *do* want to bail, so they can continue to use their still-intact original storage location).
If this isn't seen as a good direction, an alternative will be to make the migration code aware of the structure of the temporary files created by `Storage` (but doesn't guarantee this will cover all cases of such temporary files – there could for isntance be metadata files created by the host operating system).
Another option would be to mark those temporary files as hidden and skip any hidden files during iteration.
Sometimes I am not sure where my osu! is reading files from. This should
help somewhat.
```csharp
/Users/dean/Projects/osu/osu.Desktop/bin/Debug/net6.0/osu!
[runtime] 2022-07-13 07:22:03 [verbose]: Starting legacy IPC provider...
[runtime] 2022-07-13 07:22:03 [verbose]: Attempting to use custom storage location /Users/dean/Games/osu-lazer-2
[runtime] 2022-07-13 07:22:03 [verbose]: Storage successfully changed to /Users/dean/Games/osu-lazer-2.
[runtime] 2022-07-13 07:22:05 [verbose]: GL Initialized
```
Salman Ahmed
Dean Herbert
Bartłomiej Dach
Berkan Diler
Dan Balasescu
Krzysztof Gutkowski
BlauFx
Berkan Diler