RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 345

This commit is contained in:
Hydrus Network Developer 2019-03-27 17:01:02 -05:00
parent f7763b38ee
commit 42f2c3617f
34 changed files with 720 additions and 814 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
db/help my db is broke.txt
View File

@ -1,6 +1,6 @@
If you are getting weird but serious-sounding errors like 'database image is malformed', your database may be corrupt!
Nine times out of ten, corrupted databases happen because of hardware failure, usually the hard drive writing bad sectors. This is a serious problem.
Ninety-nine times out of a hundred, corrupted databases happen because of hardware failure, usually the hard drive writing bad sectors due to a power cut or old age. This is a serious problem.
first, shut everything down
@ -16,7 +16,7 @@ Then check your hard drive's integrity.
On Windows, go Start->Run (Win+R) and type 'cmd'. Type chkdsk into the new window and wait for it to scan your drive.
If it finds many problems, then your drive has been compromised in some way, and you should view it as unreliable. If it is an old drive, you should think about buying a replacement. The exception to 'buy a new drive' is if the existing one is new, otherwise works well, and you can trace the error to a specific event, such as an unprotected power surge during a storm that violently reset your computer. The other exception is if you cannot afford it. :/
If it finds many problems, then your drive has been compromised in some way and you should view it as unreliable. If it is an old drive, you should think about buying a replacement. The exception to 'buy a new drive' is if the existing one is new, otherwise works well, and you can trace the error to a specific event, such as an unprotected power surge during a storm that violently reset your computer. The other exception is if you cannot afford it. :/
On Windows, tell chkdsk to fix the problems it found by running it again with the /F modifier, like 'chkdsk /F'.
@ -67,14 +67,27 @@ client.mappings.db is usually the largest and busiest file in most people's data
If the errors do not look too bad, then try cloning a fresh copy of the broken db file like so:
.open client.db
.clone client_new.db
And wait a bit. It'll report its progress as it tries to copy your db's info to a new, cleaner db.
And wait a bit. It'll report its progress as it tries to copy your db's info to a new, cleaner db. Remember to go '.exit' once it is done to close the shell neatly.
Remember to go '.exit' to close the shell neatly.
If the clone operation is successful, rename client.db to client_old.db and client_new.db to client.db. If the damage is in a different file, do a similar operation:
If the clone operation is successful, rename client.db to client_old.db and client_new.db to client.db. (or similar names if you are dealing with client.master.db or whatever) Then, try running the client!
.open client.caches.db
.clone client.caches_new.db
.exit
.open client.master.db
.clone client.master_new.db
.exit
.open client.mappings.db
.clone client.mappings_new.db
.exit
Do not delete your original files for now. Just rename them to 'old' and move them somewhere safe. Make sure your new cloned files are all named right and then try running the client!
If the clone is unsuccessful or it still does not run correctly, contact me (Hydrus Dev) and I'll help you with the next steps. In the worst case, we will manually extract what you can to a new db.
If you recover ok but still get problems, please contact me. Check help/contact.html for ways to do this.
If you recover ok but still get problems, please contact me. Check install_dir/help/contact.html for ways to do this.

27
help/changelog.html
View File

@ -8,6 +8,33 @@
<div class="content">
<h3>changelog</h3>
<ul>
<ul>
<li><h3>version 345</h3></li>
<li>or search:</li>
<li>set out a plan to achieve some simple conjunctive normal form (e.g. (blue eyes OR green eyes) AND (blonde hair OR red hair)) OR search support</li>
<li>started work on the object extension and search code to support this search in a very basic (and likely inefficient-for-some-scenarios) way--we'll work on this as we discover the most common inefficiencies</li>
<li>.</li>
<li>thumbnails:</li>
<li>the client no longer uses both 'master' and 'resized' thumbnails--it uses a single, smarter thumbnail</li>
<li>only the 'txx' thumbnail directories (formerly referred to as full-size) are now used, and the thumbnails inside will regenerate and scale themselves as needed on demand (and will be careful to not save changes to disk when when their source file is non-local)</li>
<li>the old 'rxx' 'resized' thumbnail directories are no longer referred to anywhere in the code or ui</li>
<li>the old 'rxx' thumbnails directories will be permanently (i.e. no recycle bin) deleted on update. this is a big job, and you will be prompted on update before it happens</li>
<li>if you have migrated your db to put 'resized' thumbs on an SSD but not the formerly 'full-size', you will want to recheck the 'migrate database' dialog once you have booted and set a new thumbnail override to move the txx directories over</li>
<li>due to the smarter thumbnail, 200x200 is no longer the hard limit for hydrus thumbnails! you can now set up to 2048x2048</li>
<li>all file storage location information is now stored directly in the client db (rather than the options object), which should make for more easily export/importable options in future and improve manual fixing as needed</li>
<li>added more thumbnail-resizing related popup spam to file report mode</li>
<li>fixed a windows-only issue that was making the migrate db dialog close after a file move event concluded</li>
<li>updated database migration help for new concepts and ui</li>
<li>cleaned up some misc storage code</li>
<li>.</li>
<li>the rest:</li>
<li>fixed a problem in the client api with fetching file identifiers from file_ids</li>
<li>fleshed out 'help my db is broke.txt' with more specific clone recovery examples</li>
<li>fixed import support for a variety of single-frame music webms</li>
<li>fixed an edge-case preview viewer initialisation bug that was trying to draw the canvas before any media was set</li>
<li>network report mode now states url classes of urls about to be parsed</li>
<li>misc small fixes and cleanup</li>
</ul>
<li><h3>version 344</h3></li>
<ul>
<li>final v1.0 client api polish:</li>

8
help/client_api.html
View File

@ -48,10 +48,10 @@
</ul>
<h4>Searching and Fetching Files</h4>
<ul>
<li><a href="#get_files_search_files">POST /get_files/search_files</a></li>
<li><a href="#get_files_file_metadata">POST /get_files/file_metadata</a></li>
<li><a href="#get_files_file">POST /get_files/file</a></li>
<li><a href="#get_files_thumbnail">POST /get_files/thumbnail</a></li>
<li><a href="#get_files_search_files">GET /get_files/search_files</a></li>
<li><a href="#get_files_file_metadata">GET /get_files/file_metadata</a></li>
<li><a href="#get_files_file">GET /get_files/file</a></li>
<li><a href="#get_files_thumbnail">GET /get_files/thumbnail</a></li>
</ul>
</ul>
<h3>Access Management</h3>

20
help/database_migration.html
View File

@ -30,7 +30,7 @@
<p>Backing such an arrangement up is obviously more complicated, and the internal client backup is not sophisticated enough to capture everything, so I recommend you figure out a broader solution with a third-party backup program like FreeFileSync.</p>
<h3>pulling your media apart</h3>
<p><b class="warning">As always, I recommend creating a backup before you try any of this, just in case it goes wrong.</b></p>
<p>If you would like to spread your files and thumbnails across multiple locations, please do not move their folders around yourself--the database has an internal 'knowledge' of where it thinks its file and thumbnail folders are, and if you move them while it is closed, it will throw 'missing path' errors as soon as it boots. The internal hydrus logic of relative and absolute paths is not always obvious, so it is easy to make mistakes, even if you think you know what you are doing. Instead, please do it through the gui:</p>
<p>If you would like to move your files and thumbnails to new locations, I generally recommend you not move their folders around yourself--the database has an internal knowledge of where it thinks its file and thumbnail folders are, and if you move them while it is closed, it will become confused and you will have to manually relocate what is missing on the next boot via a repair dialog. This is not impossible to figure out, but if the program's 'client files' folder confuses you at all, I'd recommend you stay away. Instead, you can simply do it through the gui:</p>
<p>Go <i>database->migrate database</i>, giving you this dialog:</p>
<p><img src="db_migration.png" /></p>
<p>This is an image from my old laptop's client. At that time, I had moved the main database and its files out of the install directory but otherwise kept everything together. Your situation may be simpler or more complicated.</p>
@ -56,23 +56,23 @@
<h3>finally</h3>
<p>If your database now lives in one or more new locations, make sure to update your backup routine to follow them!</p>
<h3>moving to an SSD</h3>
<p>As an example, let's say you started using the hydrus client on your HDD, and now you have an SSD available and would like to move your resized thumbnails and main install to that SSD to speed up the client. Your database will be valid and functional at every stage of this, and it can all be undone. The basic steps are:</p>
<ul>
<p>As an example, let's say you started using the hydrus client on your HDD, and now you have an SSD available and would like to move your thumbnails and main install to that SSD to speed up the client. Your database will be valid and functional at every stage of this, and it can all be undone. The basic steps are:</p>
<ol>
<li>Move your 'fast' files to the fast location.</li>
<li>Move your 'slow' files out of the main install directory.</li>
<li>Move the install and db itself to the fast location and update shortcuts.</li>
</ul>
</ol>
<p>Specifically:</p>
<ul>
<li>Update your backup if you maintain one.</li>
<li>Create an empty folder on your HDD that is outside of your current install folder. Call it 'hydrus_files_slow' or similar.</li>
<li>Create two empty folders on your SSD with names like 'hydrus_db' and 'hydrus_files_fast'.</li>
<li>Create an empty folder on your HDD that is outside of your current install folder. Call it 'hydrus_files' or similar.</li>
<li>Create two empty folders on your SSD with names like 'hydrus_db' and 'hydrus_thumbnails'.</li>
<li>.</li>
<li>Set the 'resized thumbnail location' to 'hydrus_files_fast'. You should get that new location in the list, currently empty but prepared to take all your resized thumbs.</li>
<li>Hit 'move files now' to actually move the thumbnails. Since this involves moving a lot of individual files from a high-latency source, it will take a long time to finish.</li>
<li>Set the 'thumbnail location override' to 'hydrus_thumbnails'. You should get that new location in the list, currently empty but prepared to take all your thumbs.</li>
<li>Hit 'move files now' to actually move the thumbnails. Since this involves moving a lot of individual files from a high-latency source, it will take a long time to finish. The hydrus client may hang periodically as it works, but you can just leave it to work on its own--it will get there in the end. You can also watch it do its disk work under Task Manager.</li>
<li>.</li>
<li>Now hit 'add location' and set 'hydrus_files_slow'. 'hydrus_files_slow' should be added and willing to take some files and full-size thumbnails.</li>
<li>Select the old location (probably 'install_dir/db/client_files') and hit 'empty/remove location'. 'hydrus_files_slow' should now be willing to take all the files from the old location.</li>
<li>Now hit 'add location' and select your new 'hydrus_files'. 'hydrus_files' should be added and willing to take 50% of the files.</li>
<li>Select the old location (probably 'install_dir/db/client_files') and hit 'decrease weight' until it has weight 0 and you are prompted to remove it completely. 'hydrus_files' should now be willing to take all the files from the old location.</li>
<li>Hit 'move files now' again to make this happen. This should be fast since it is just moving a bunch of folders across the same partition.</li>
<li>.</li>
<li>With everything now 'non-portable' and hence decoupled from the db, you can now easily migrate the install and db to 'hydrus_db' simply by shutting the client down and moving the install folder in a file explorer.</li>

BIN
help/db_migration.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 24 KiB

BIN
help/db_migration_example.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 24 KiB

2
help/duplicates.html
View File

@ -54,7 +54,7 @@
</ul>
<p>The idea is to compare the two files by scrolling with your mouse wheel and then clicking to assign a status, at which point the next pair will be loaded. If you prefer different shortcuts, you can set them under <i>file->shortcuts</i> or the keyboard icon on the duplicate filter's top hover window. You can also access more 'duplicate decisions' through the labelled buttons and change what happens to the files and their tags and ratings on each different decision through the cog icon on the same top hover window.</p>
<p><a href="dupe_icons.png"><img src="dupe_icons.png" /></a></p>
<p><i>Move your move to the top of the media viewer to bring up the top hover window. Hit the cog or keyboard icons to edit how it works, and click the buttons if you do not have a shortcut mapped. 'Custom action' lets you one of the other four actions but with one-off content merge options--say if you want to set that files are alternate but still with to merge some tags.</i></p>
<p><i>Move your mouse to the top of the media viewer to bring up the top hover window. Hit the cog or keyboard icons to edit how it works, and click the buttons if you do not have a shortcut mapped. 'Custom action' lets you one of the other four actions but with one-off content merge options--say if you want to set that files are alternate but still with to merge some tags.</i></p>
<p>Because of technical limitations, you may be asked to checkpoint (save your progress to the database and then continue filtering) every now and then.</p>
<h3>different duplicate statuses</h3>
<p>There are currently five possible statuses. The client uses different logic to apply them at the database level, so please treat them as described and not a different scheme.</p>

516
include/ClientCaches.py
View File

@ -281,7 +281,7 @@ class ClientFilesManager( object ):
return path
def _GenerateExpectedFullSizeThumbnailPath( self, hash ):
def _GenerateExpectedThumbnailPath( self, hash ):
self._WaitOnWakeup()
@ -296,22 +296,7 @@ class ClientFilesManager( object ):
return path
def _GenerateExpectedResizedThumbnailPath( self, hash ):
self._WaitOnWakeup()
hash_encoded = hash.hex()
prefix = 'r' + hash_encoded[:2]
location = self._prefixes_to_locations[ prefix ]
path = os.path.join( location, prefix, hash_encoded ) + '.thumbnail.resized'
return path
def _GenerateFullSizeThumbnail( self, hash, mime ):
def _GenerateThumbnail( self, hash, mime ):
file_path = self._GenerateExpectedFilePath( hash, mime )
@ -322,84 +307,18 @@ class ClientFilesManager( object ):
try:
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
percentage_in = self._controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
thumbnail = HydrusFileHandling.GenerateThumbnail( file_path, mime, percentage_in = percentage_in )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, bounding_dimensions, mime, percentage_in = percentage_in )
except Exception as e:
raise HydrusExceptions.FileMissingException( 'The thumbnail for file ' + hash.hex() + ' was missing. It could not be regenerated from the original file for the above reason. This event could indicate hard drive corruption. Please check everything is ok.' )
full_size_path = self._GenerateExpectedFullSizeThumbnailPath( hash )
try:
HydrusPaths.MakeFileWritable( full_size_path )
with open( full_size_path, 'wb' ) as f:
f.write( thumbnail )
except Exception as e:
raise HydrusExceptions.FileMissingException( 'The thumbnail for file ' + hash.hex() + ' was missing. It was regenerated from the original file, but hydrus could not write it to the location ' + full_size_path + ' for the above reason. This event could indicate hard drive corruption, and it also suggests that hydrus does not have permission to write to its thumbnail folder. Please check everything is ok.' )
def _GenerateResizedThumbnail( self, hash, mime ):
full_size_path = self._GenerateExpectedFullSizeThumbnailPath( hash )
thumbnail_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
if mime in ( HC.IMAGE_GIF, HC.IMAGE_PNG ):
fullsize_thumbnail_mime = HC.IMAGE_PNG
else:
fullsize_thumbnail_mime = HC.IMAGE_JPEG
try:
thumbnail_resized = HydrusFileHandling.GenerateThumbnailFileBytesFromStaticImagePath( full_size_path, thumbnail_dimensions, fullsize_thumbnail_mime )
except:
try:
ClientPaths.DeletePath( full_size_path, always_delete_fully = True )
except:
raise HydrusExceptions.FileMissingException( 'The thumbnail for file ' + hash.hex() + ' was found, but it would not render. An attempt to delete it was made, but that failed as well. This event could indicate hard drive corruption, and it also suggests that hydrus does not have permission to write to its thumbnail folder. Please check everything is ok.' )
self._GenerateFullSizeThumbnail( hash, mime )
thumbnail_resized = HydrusFileHandling.GenerateThumbnailFileBytesFromStaticImagePath( full_size_path, thumbnail_dimensions, fullsize_thumbnail_mime )
resized_path = self._GenerateExpectedResizedThumbnailPath( hash )
try:
HydrusPaths.MakeFileWritable( resized_path )
with open( resized_path, 'wb' ) as f:
f.write( thumbnail_resized )
except Exception as e:
HydrusData.ShowException( e )
raise HydrusExceptions.FileMissingException( 'The thumbnail for file ' + hash.hex() + ' was found, but the resized version would not save to disk. This event suggests that hydrus does not have permission to write to its thumbnail folder. Please check everything is ok.' )
self._SaveThumbnail( hash, thumbnail_bytes )
def _GetRecoverTuple( self ):
@ -428,7 +347,7 @@ class ClientFilesManager( object ):
def _GetRebalanceTuple( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._controller.new_options.GetClientFilesLocationsToIdealWeights()
( locations_to_ideal_weights, thumbnail_override ) = self._controller.Read( 'ideal_client_files_locations' )
total_weight = sum( locations_to_ideal_weights.values() )
@ -500,65 +419,26 @@ class ClientFilesManager( object ):
else:
if full_size_thumbnail_override is None:
for hex_prefix in HydrusData.IterateHexPrefixes():
for hex_prefix in HydrusData.IterateHexPrefixes():
thumbnail_prefix = 't' + hex_prefix
if thumbnail_override is None:
full_size_prefix = 't' + hex_prefix
file_prefix = 'f' + hex_prefix
full_size_location = self._prefixes_to_locations[ full_size_prefix ]
file_location = self._prefixes_to_locations[ file_prefix ]
correct_location = self._prefixes_to_locations[ file_prefix ]
if full_size_location != file_location:
return ( full_size_prefix, full_size_location, file_location )
else:
correct_location = thumbnail_override
else:
current_thumbnails_location = self._prefixes_to_locations[ thumbnail_prefix ]
for hex_prefix in HydrusData.IterateHexPrefixes():
if current_thumbnails_location != correct_location:
full_size_prefix = 't' + hex_prefix
full_size_location = self._prefixes_to_locations[ full_size_prefix ]
if full_size_location != full_size_thumbnail_override:
return ( full_size_prefix, full_size_location, full_size_thumbnail_override )
if resized_thumbnail_override is None:
for hex_prefix in HydrusData.IterateHexPrefixes():
resized_prefix = 'r' + hex_prefix
file_prefix = 'f' + hex_prefix
resized_location = self._prefixes_to_locations[ resized_prefix ]
file_location = self._prefixes_to_locations[ file_prefix ]
if resized_location != file_location:
return ( resized_prefix, resized_location, file_location )
else:
for hex_prefix in HydrusData.IterateHexPrefixes():
resized_prefix = 'r' + hex_prefix
resized_location = self._prefixes_to_locations[ resized_prefix ]
if resized_location != resized_thumbnail_override:
return ( resized_prefix, resized_location, resized_thumbnail_override )
return ( thumbnail_prefix, current_thumbnails_location, correct_location )
@ -588,7 +468,7 @@ class ClientFilesManager( object ):
for ( prefix, location ) in list(self._prefixes_to_locations.items()):
if prefix.startswith( 't' ) or prefix.startswith( 'r' ):
if prefix.startswith( 't' ):
dir = os.path.join( location, prefix )
@ -721,6 +601,25 @@ class ClientFilesManager( object ):
def _SaveThumbnail( self, hash, thumbnail_bytes ):
thumbnail_path = self._GenerateExpectedThumbnailPath( hash )
try:
HydrusPaths.MakeFileWritable( thumbnail_path )
with open( thumbnail_path, 'wb' ) as f:
f.write( thumbnail_bytes )
except Exception as e:
raise HydrusExceptions.FileMissingException( 'The thumbnail for file "{}" failed to write to path "{}". This event suggests that hydrus does not have permission to write to its thumbnail folder. Please check everything is ok.'.format( hash.hex(), thumbnail_path ) )
def _WaitOnWakeup( self ):
if HG.client_controller.new_options.GetBoolean( 'file_system_waits_on_wakeup' ):
@ -785,35 +684,28 @@ class ClientFilesManager( object ):
def AddFullSizeThumbnailFromBytes( self, hash, thumbnail ):
def AddThumbnailFromBytes( self, hash, thumbnail_bytes ):
with self._lock:
self.LocklessAddFullSizeThumbnailFromBytes( hash, thumbnail )
self.LocklessAddThumbnailFromBytes( hash, thumbnail_bytes )
def LocklessAddFullSizeThumbnailFromBytes( self, hash, thumbnail ):
def LocklessAddThumbnailFromBytes( self, hash, thumbnail_bytes ):
dest_path = self._GenerateExpectedFullSizeThumbnailPath( hash )
dest_path = self._GenerateExpectedThumbnailPath( hash )
if HG.file_report_mode:
HydrusData.ShowText( 'Adding full-size thumbnail: ' + str( ( len( thumbnail ), dest_path ) ) )
HydrusData.ShowText( 'Adding thumbnail: ' + str( ( len( thumbnail_bytes ), dest_path ) ) )
HydrusPaths.MakeFileWritable( dest_path )
with open( dest_path, 'wb' ) as f:
f.write( thumbnail )
resized_path = self._GenerateExpectedResizedThumbnailPath( hash )
if os.path.exists( resized_path ):
ClientPaths.DeletePath( resized_path, always_delete_fully = True )
f.write( thumbnail_bytes )
self._controller.pub( 'clear_thumbnails', { hash } )
@ -1059,11 +951,9 @@ class ClientFilesManager( object ):
for hash in hashes_chunk:
path = self._GenerateExpectedFullSizeThumbnailPath( hash )
resized_path = self._GenerateExpectedResizedThumbnailPath( hash )
path = self._GenerateExpectedThumbnailPath( hash )
ClientPaths.DeletePath( path, always_delete_fully = True )
ClientPaths.DeletePath( resized_path, always_delete_fully = True )
@ -1109,7 +999,7 @@ class ClientFilesManager( object ):
if thumbnail is not None:
self.LocklessAddFullSizeThumbnailFromBytes( hash, thumbnail )
self.LocklessAddThumbnailFromBytes( hash, thumbnail )
( import_status, note ) = self._controller.WriteSynchronous( 'import_file', file_import_job )
@ -1194,28 +1084,23 @@ class ClientFilesManager( object ):
if HG.file_report_mode:
HydrusData.ShowText( 'File path request success: ' + path )
return path
def GetFullSizeThumbnailPath( self, hash, mime = None ):
def GetThumbnailPath( self, hash, mime = None ):
if HG.file_report_mode:
HydrusData.ShowText( 'Full-size thumbnail path request: ' + str( ( hash, mime ) ) )
HydrusData.ShowText( 'Thumbnail path request: ' + str( ( hash, mime ) ) )
with self._lock:
path = self._GenerateExpectedFullSizeThumbnailPath( hash )
path = self._GenerateExpectedThumbnailPath( hash )
if not os.path.exists( path ):
self._GenerateFullSizeThumbnail( hash, mime )
self._GenerateThumbnail( hash, mime )
if not self._bad_error_occurred:
@ -1225,42 +1110,17 @@ class ClientFilesManager( object ):
if HG.file_report_mode:
HydrusData.ShowText( 'Full-size thumbnail path request success: ' + path )
return path
def GetResizedThumbnailPath( self, hash, mime ):
def LocklessHasThumbnail( self, hash ):
with self._lock:
path = self._GenerateExpectedResizedThumbnailPath( hash )
if HG.file_report_mode:
HydrusData.ShowText( 'Resized thumbnail path request: ' + path )
if not os.path.exists( path ):
self._GenerateResizedThumbnail( hash, mime )
return path
def LocklessHasFullSizeThumbnail( self, hash ):
path = self._GenerateExpectedFullSizeThumbnailPath( hash )
path = self._GenerateExpectedThumbnailPath( hash )
if HG.file_report_mode:
HydrusData.ShowText( 'Full-size thumbnail path test: ' + path )
HydrusData.ShowText( 'Thumbnail path test: ' + path )
return os.path.exists( path )
@ -1348,7 +1208,7 @@ class ClientFilesManager( object ):
def RegenerateFullSizeThumbnail( self, hash, mime ):
def RegenerateThumbnail( self, hash, mime ):
with self._lock:
@ -1357,20 +1217,7 @@ class ClientFilesManager( object ):
HydrusData.ShowText( 'Thumbnail regen request: ' + str( ( hash, mime ) ) )
self._GenerateFullSizeThumbnail( hash, mime )
def RegenerateResizedThumbnail( self, hash, mime ):
with self._lock:
if HG.file_report_mode:
HydrusData.ShowText( 'Thumbnail regen request: ' + str( ( hash, mime ) ) )
self._GenerateResizedThumbnail( hash, mime )
self._GenerateThumbnail( hash, mime )
@ -1420,9 +1267,9 @@ class ClientFilesManager( object ):
hash = bytes.fromhex( hash_encoded )
full_size_path = self._GenerateExpectedFullSizeThumbnailPath( hash )
thumbnail_path = self._GenerateExpectedThumbnailPath( hash )
if only_do_missing and os.path.exists( full_size_path ):
if only_do_missing and os.path.exists( thumbnail_path ):
continue
@ -1431,14 +1278,7 @@ class ClientFilesManager( object ):
if mime in HC.MIMES_WITH_THUMBNAILS:
self._GenerateFullSizeThumbnail( hash, mime )
thumbnail_resized_path = self._GenerateExpectedResizedThumbnailPath( hash )
if os.path.exists( thumbnail_resized_path ):
ClientPaths.DeletePath( thumbnail_resized_path, always_delete_fully = True )
self._GenerateThumbnail( hash, mime )
except:
@ -1465,6 +1305,14 @@ class ClientFilesManager( object ):
def SaveThumbnail( self, hash, thumbnail_bytes ):
with self._lock:
self._SaveThumbnail( hash, thumbnail_bytes )
class DataCache( object ):
def __init__( self, controller, cache_size, timeout = 1200 ):
@ -3138,36 +2986,26 @@ class ThumbnailCache( object ):
self._controller.sub( self, 'ClearThumbnails', 'clear_thumbnails' )
def _GetResizedHydrusBitmap( self, display_media ):
def _GetThumbnailHydrusBitmap( self, display_media ):
if not HG.thumbnail_experiment_mode:
return self._GetResizedHydrusBitmapFromHardDrive( display_media )
else:
return self._GetResizedHydrusBitmapFromFullSize( display_media )
def _GetResizedHydrusBitmapFromFullSize( self, display_media ):
thumbnail_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
bounding_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
hash = display_media.GetHash()
mime = display_media.GetMime()
( media_width, media_height ) = display_media.GetResolution()
locations_manager = display_media.GetLocationsManager()
try:
path = self._controller.client_files_manager.GetFullSizeThumbnailPath( hash, mime )
path = self._controller.client_files_manager.GetThumbnailPath( hash, mime )
except HydrusExceptions.FileMissingException as e:
if locations_manager.IsLocal():
summary = 'Unable to get full-size thumbnail for file {}.'.format( hash.hex() )
summary = 'Unable to get thumbnail for file {}.'.format( hash.hex() )
self._HandleThumbnailException( e, summary )
@ -3184,7 +3022,7 @@ class ThumbnailCache( object ):
try:
# file is malformed, let's force a regen
self._controller.client_files_manager.RegenerateResizedThumbnail( hash, mime )
self._controller.client_files_manager.RegenerateThumbnail( hash, mime )
except Exception as e:
@ -3209,113 +3047,131 @@ class ThumbnailCache( object ):
( fullsize_x, fullsize_y ) = ClientImageHandling.GetNumPyImageResolution( numpy_image )
( current_width, current_height ) = ClientImageHandling.GetNumPyImageResolution( numpy_image )
( resized_x, resized_y ) = HydrusImageHandling.GetThumbnailResolution( ( fullsize_x, fullsize_y ), thumbnail_dimensions )
( expected_width, expected_height ) = HydrusImageHandling.GetThumbnailResolution( ( media_width, media_height ), bounding_dimensions )
already_correct = fullsize_x == resized_x and fullsize_y == resized_y
correct_size = current_width == expected_width and current_height == expected_height
if not already_correct:
if not correct_size:
numpy_image = ClientImageHandling.EfficientlyThumbnailNumpyImage( numpy_image, ( resized_x, resized_y ) )
it_is_definitely_too_big = current_width >= expected_width and current_height >= expected_height
hydrus_bitmap = ClientRendering.GenerateHydrusBitmapFromNumPyImage( numpy_image )
return hydrus_bitmap
def _GetResizedHydrusBitmapFromHardDrive( self, display_media ):
thumbnail_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
if tuple( thumbnail_dimensions ) == HC.UNSCALED_THUMBNAIL_DIMENSIONS:
full_size = True
else:
full_size = False
hash = display_media.GetHash()
mime = display_media.GetMime()
locations_manager = display_media.GetLocationsManager()
try:
if full_size:
if it_is_definitely_too_big:
path = self._controller.client_files_manager.GetFullSizeThumbnailPath( hash, mime )
if HG.file_report_mode:
HydrusData.ShowText( 'Thumbnail {} too big.'.format( hash.hex() ) )
# the thumb we have is larger than desired. we can use it to generate what we actually want without losing significant data
# this is _resize_, not _thumbnail_, because we already know the dimensions we want
# and in some edge cases, doing getthumbresolution on existing thumb dimensions results in float/int conversion imprecision and you get 90px/91px regen cycles that never get fixed
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, ( expected_width, expected_height ) )
if locations_manager.IsLocal():
# we have the master file, so it is safe to save our resized thumb back to disk since we can regen from source if needed
if HG.file_report_mode:
HydrusData.ShowText( 'Thumbnail {} too big, saving back to disk.'.format( hash.hex() ) )
try:
try:
thumbnail_bytes = ClientImageHandling.GenerateBytesFromCV( numpy_image, mime )
except HydrusExceptions.CantRenderWithCVException:
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
except:
summary = 'The thumbnail for file {} was too large, but an attempt to shrink it failed.'.format( hash.hex() )
self._HandleThumbnailException( e, summary )
return self._special_thumbs[ 'hydrus' ]
try:
self._controller.client_files_manager.SaveThumbnail( hash, thumbnail_bytes )
except:
summary = 'The thumbnail for file {} was too large, but an attempt to save back the shrunk file failed.'.format( hash.hex() )
self._HandleThumbnailException( e, summary )
return self._special_thumbs[ 'hydrus' ]
else:
path = self._controller.client_files_manager.GetResizedThumbnailPath( hash, mime )
# the thumb we have is either too small or completely messed up due to a previous ratio misparse
except HydrusExceptions.FileMissingException as e:
if locations_manager.IsLocal():
media_is_same_size_as_thumb = current_width == media_width and current_height == media_height
summary = 'Unable to get full-size thumbnail for file {}.'.format( hash.hex() )
if media_is_same_size_as_thumb:
# the thumb is smaller than desired, but this is a 32x32 pixilart image or whatever, so no need to scale
if HG.file_report_mode:
HydrusData.ShowText( 'Thumbnail {} too small due to small source file.'.format( hash.hex() ) )
pass
else:
self._HandleThumbnailException( e, summary )
return self._special_thumbs[ 'hydrus' ]
try:
hydrus_bitmap = ClientRendering.GenerateHydrusBitmap( path, mime )
except Exception as e:
try:
# file is malformed, let's force a regen
self._controller.client_files_manager.RegenerateFullSizeThumbnail( hash, mime )
except Exception as e:
summary = 'The thumbnail for file ' + hash.hex() + ' was not loadable. An attempt to regenerate it failed.'
self._HandleThumbnailException( e, summary )
return self._special_thumbs[ 'hydrus' ]
try:
hydrus_bitmap = ClientRendering.GenerateHydrusBitmap( path, mime )
except Exception as e:
summary = 'The thumbnail for file ' + hash.hex() + ' was not loadable. It was regenerated, but that file would not render either. Your image libraries or hard drive connection are unreliable. Please inform the hydrus developer what has happened.'
self._HandleThumbnailException( e, summary )
return self._special_thumbs[ 'hydrus' ]
if locations_manager.IsLocal():
# we have the master file, so we should regen the thumb from source
if HG.file_report_mode:
HydrusData.ShowText( 'Thumbnail {} too small, regenerating from source.'.format( hash.hex() ) )
try:
self._controller.client_files_manager.RegenerateThumbnail( hash, mime )
numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
except:
summary = 'The thumbnail for file {} was too small, but the attempt to regenerate it or load the new file back failed.'.format( hash.hex() )
self._HandleThumbnailException( e, summary )
return self._special_thumbs[ 'hydrus' ]
else:
# we do not have the master file, so we have to scale up from what we have
if HG.file_report_mode:
HydrusData.ShowText( 'Thumbnail {} too small, scaling up due to no local source.'.format( hash.hex() ) )
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, ( expected_width, expected_height ) )
( media_x, media_y ) = display_media.GetResolution()
( actual_x, actual_y ) = hydrus_bitmap.GetSize()
( desired_x, desired_y ) = self._controller.options[ 'thumbnail_dimensions' ]
too_large = actual_x > desired_x or actual_y > desired_y
small_original_image = actual_x == media_x and actual_y == media_y
too_small = actual_x < desired_x and actual_y < desired_y
if too_large or ( too_small and not small_original_image ):
self._controller.client_files_manager.RegenerateResizedThumbnail( hash, mime )
hydrus_bitmap = ClientRendering.GenerateHydrusBitmap( path, mime )
hydrus_bitmap = ClientRendering.GenerateHydrusBitmapFromNumPyImage( numpy_image )
return hydrus_bitmap
@ -3384,9 +3240,9 @@ class ThumbnailCache( object ):
path = os.path.join( HC.STATIC_DIR, name + '.png' )
thumbnail_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
bounding_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailFileBytesFromStaticImagePath( path, thumbnail_dimensions, HC.IMAGE_PNG )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePath( path, bounding_dimensions, HC.IMAGE_PNG )
with open( temp_path, 'wb' ) as f:
@ -3454,7 +3310,7 @@ class ThumbnailCache( object ):
try:
hydrus_bitmap = self._GetResizedHydrusBitmap( display_media )
hydrus_bitmap = self._GetThumbnailHydrusBitmap( display_media )
except:

2
include/ClientController.py
View File

@ -71,7 +71,7 @@ class Controller( HydrusController.HydrusController ):
# just to set up some defaults, in case some db update expects something for an odd yaml-loading reason
self.options = ClientDefaults.GetClientDefaultOptions()
self.new_options = ClientOptions.ClientOptions( self.db_dir )
self.new_options = ClientOptions.ClientOptions()
HC.options = self.options

213
include/ClientDB.py
View File

@ -2860,6 +2860,9 @@ class DB( HydrusDB.HydrusDB ):
self._c.execute( 'CREATE TABLE client_files_locations ( prefix TEXT, location TEXT );' )
self._c.execute( 'CREATE TABLE IF NOT EXISTS ideal_client_files_locations ( location TEXT, weight INTEGER );' )
self._c.execute( 'CREATE TABLE IF NOT EXISTS ideal_thumbnail_override_location ( location TEXT );' )
self._c.execute( 'CREATE TABLE current_files ( service_id INTEGER REFERENCES services ON DELETE CASCADE, hash_id INTEGER, timestamp INTEGER, PRIMARY KEY ( service_id, hash_id ) );' )
self._CreateIndex( 'current_files', [ 'timestamp' ] )
@ -2974,9 +2977,10 @@ class DB( HydrusDB.HydrusDB ):
self._c.execute( 'INSERT INTO client_files_locations ( prefix, location ) VALUES ( ?, ? );', ( 'f' + prefix, location ) )
self._c.execute( 'INSERT INTO client_files_locations ( prefix, location ) VALUES ( ?, ? );', ( 't' + prefix, location ) )
self._c.execute( 'INSERT INTO client_files_locations ( prefix, location ) VALUES ( ?, ? );', ( 'r' + prefix, location ) )
self._c.execute( 'INSERT INTO ideal_client_files_locations ( location, weight ) VALUES ( ?, ? );', ( location, 1 ) )
init_service_info = []
init_service_info.append( ( CC.COMBINED_LOCAL_FILE_SERVICE_KEY, HC.COMBINED_LOCAL_FILE, 'all local files' ) )
@ -2999,7 +3003,7 @@ class DB( HydrusDB.HydrusDB ):
self._c.executemany( 'INSERT INTO yaml_dumps VALUES ( ?, ?, ? );', ( ( YAML_DUMP_ID_IMAGEBOARD, name, imageboards ) for ( name, imageboards ) in ClientDefaults.GetDefaultImageboards() ) )
new_options = ClientOptions.ClientOptions( self._db_dir )
new_options = ClientOptions.ClientOptions()
new_options.SetSimpleDownloaderFormulae( ClientDefaults.GetDefaultSimpleDownloaderFormulae() )
@ -4625,6 +4629,8 @@ class DB( HydrusDB.HydrusDB ):
include_current_tags = search_context.IncludeCurrentTags()
include_pending_tags = search_context.IncludePendingTags()
or_predicates = search_context.GetORPredicates()
#
files_info_predicates = []
@ -4688,6 +4694,40 @@ class DB( HydrusDB.HydrusDB ):
query_hash_ids = None
#
# let's do OR preds here for now
# updating all this code to do OR and AND naturally will be a big job getting right, so let's get a functional inefficient solution and then optimise later as needed
# -tag stuff and various other exclude situations remain a pain to do quickly assuming OR
# the future extension of this will be creating an OR_search_context with all the OR_pred's subpreds and have that naturally query_hash_ids.update throughout this func based on search_context search_type
for or_predicate in or_predicates:
# blue eyes OR green eyes
or_query_hash_ids = set()
for or_subpredicate in or_predicate.GetValue():
# blue eyes
or_search_context = search_context.Duplicate()
or_search_context.SetPredicates( [ or_subpredicate ] )
or_query_hash_ids.update( self._GetHashIdsFromQuery( or_search_context, job_key ) )
if job_key.IsCancelled():
return set()
update_qhi( query_hash_ids, or_query_hash_ids )
#
if system_predicates.HasSimilarTo():
( similar_to_hash, max_hamming ) = system_predicates.GetSimilarTo()
@ -4946,13 +4986,13 @@ class DB( HydrusDB.HydrusDB ):
if file_service_key == CC.COMBINED_FILE_SERVICE_KEY:
query_hash_ids.intersection_update( self._STI( self._SelectFromList( 'SELECT hash_id FROM files_info WHERE ' + ' AND '.join( files_info_predicates ) + ';', query_hash_ids ) ) )
update_qhi( query_hash_ids, self._STI( self._SelectFromList( 'SELECT hash_id FROM files_info WHERE ' + ' AND '.join( files_info_predicates ) + ';', query_hash_ids ) ) )
else:
files_info_predicates.insert( 0, 'service_id = ' + str( file_service_id ) )
query_hash_ids.intersection_update( self._STI( self._SelectFromList( 'SELECT hash_id FROM current_files NATURAL JOIN files_info WHERE ' + ' AND '.join( files_info_predicates ) + ';', query_hash_ids ) ) )
update_qhi( query_hash_ids, self._STI( self._SelectFromList( 'SELECT hash_id FROM current_files NATURAL JOIN files_info WHERE ' + ' AND '.join( files_info_predicates ) + ';', query_hash_ids ) ) )
@ -5031,14 +5071,14 @@ class DB( HydrusDB.HydrusDB ):
service_id = self._GetServiceId( service_key )
query_hash_ids.intersection_update( self._STI( self._c.execute( 'SELECT hash_id FROM current_files WHERE service_id = ?;', ( service_id, ) ) ) )
update_qhi( query_hash_ids, self._STI( self._c.execute( 'SELECT hash_id FROM current_files WHERE service_id = ?;', ( service_id, ) ) ) )
for service_key in file_services_to_include_pending:
service_id = self._GetServiceId( service_key )
query_hash_ids.intersection_update( self._STI( self._c.execute( 'SELECT hash_id FROM file_transfers WHERE service_id = ?;', ( service_id, ) ) ) )
update_qhi( query_hash_ids, self._STI( self._c.execute( 'SELECT hash_id FROM file_transfers WHERE service_id = ?;', ( service_id, ) ) ) )
for service_key in file_services_to_exclude_current:
@ -5088,7 +5128,7 @@ class DB( HydrusDB.HydrusDB ):
hash_ids = zero_hash_ids.union( accurate_except_zero_hash_ids )
query_hash_ids.intersection_update( hash_ids )
update_qhi( query_hash_ids, hash_ids )
@ -5113,7 +5153,7 @@ class DB( HydrusDB.HydrusDB ):
hash_ids = zero_hash_ids.union( accurate_except_zero_hash_ids )
query_hash_ids.intersection_update( hash_ids )
update_qhi( query_hash_ids, hash_ids )
@ -5142,7 +5182,7 @@ class DB( HydrusDB.HydrusDB ):
if must_be_local:
query_hash_ids.intersection_update( local_hash_ids )
update_qhi( query_hash_ids, local_hash_ids )
elif must_not_be_local:
@ -5152,7 +5192,7 @@ class DB( HydrusDB.HydrusDB ):
if must_be_inbox:
query_hash_ids.intersection_update( self._inbox_hash_ids )
update_qhi( query_hash_ids, self._inbox_hash_ids )
elif must_be_archive:
@ -5169,7 +5209,7 @@ class DB( HydrusDB.HydrusDB ):
if operator: # inclusive
query_hash_ids.intersection_update( url_hash_ids )
update_qhi( query_hash_ids, url_hash_ids )
else:
@ -5239,7 +5279,7 @@ class DB( HydrusDB.HydrusDB ):
elif num_tags_nonzero:
query_hash_ids.intersection_update( nonzero_tag_query_hash_ids )
update_qhi( query_hash_ids, nonzero_tag_query_hash_ids )
@ -5256,7 +5296,7 @@ class DB( HydrusDB.HydrusDB ):
good_tag_count_hash_ids.update( zero_hash_ids )
query_hash_ids.intersection_update( good_tag_count_hash_ids )
update_qhi( query_hash_ids, good_tag_count_hash_ids )
if job_key.IsCancelled():
@ -5272,7 +5312,7 @@ class DB( HydrusDB.HydrusDB ):
good_hash_ids = self._GetHashIdsThatHaveTagAsNum( file_service_key, tag_service_key, namespace, num, '>', include_current_tags, include_pending_tags )
query_hash_ids.intersection_update( good_hash_ids )
update_qhi( query_hash_ids, good_hash_ids )
if 'max_tag_as_number' in simple_preds:
@ -5281,7 +5321,7 @@ class DB( HydrusDB.HydrusDB ):
good_hash_ids = self._GetHashIdsThatHaveTagAsNum( file_service_key, tag_service_key, namespace, num, '<', include_current_tags, include_pending_tags )
query_hash_ids.intersection_update( good_hash_ids )
update_qhi( query_hash_ids, good_hash_ids )
if job_key.IsCancelled():
@ -5816,6 +5856,33 @@ class DB( HydrusDB.HydrusDB ):
def _GetIdealClientFilesLocations( self ):
locations_to_ideal_weights = {}
for ( portable_location, weight ) in self._c.execute( 'SELECT location, weight FROM ideal_client_files_locations;' ):
abs_location = HydrusPaths.ConvertPortablePathToAbsPath( portable_location )
locations_to_ideal_weights[ abs_location ] = weight
result = self._c.execute( 'SELECT location FROM ideal_thumbnail_override_location;' ).fetchone()
if result is None:
abs_ideal_thumbnail_override_location = None
else:
( portable_ideal_thumbnail_override_location, ) = result
abs_ideal_thumbnail_override_location = HydrusPaths.ConvertPortablePathToAbsPath( portable_ideal_thumbnail_override_location )
return ( locations_to_ideal_weights, abs_ideal_thumbnail_override_location )
def _GetMaintenanceDue( self, stop_time ):
jobs_to_do = []
@ -6619,7 +6686,7 @@ class DB( HydrusDB.HydrusDB ):
hash = self._GetHash( hash_id )
if client_files_manager.LocklessHasFullSizeThumbnail( hash ):
if client_files_manager.LocklessHasThumbnail( hash ):
self._c.execute( 'INSERT OR IGNORE INTO remote_thumbnails ( service_id, hash_id ) VALUES ( ?, ? );', ( service_id, hash_id ) )
@ -9264,6 +9331,7 @@ class DB( HydrusDB.HydrusDB ):
elif action == 'force_refresh_tags_managers': result = self._GetForceRefreshTagsManagers( *args, **kwargs )
elif action == 'hash_ids_to_hashes': result = self._GetHashIdsToHashes( *args, **kwargs )
elif action == 'hash_status': result = self._GetHashStatus( *args, **kwargs )
elif action == 'ideal_client_files_locations': result = self._GetIdealClientFilesLocations( *args, **kwargs )
elif action == 'imageboards': result = self._GetYAMLDump( YAML_DUMP_ID_IMAGEBOARD, *args, **kwargs )
elif action == 'in_inbox': result = self._InInbox( *args, **kwargs )
elif action == 'is_an_orphan': result = self._IsAnOrphan( *args, **kwargs )
@ -9619,11 +9687,13 @@ class DB( HydrusDB.HydrusDB ):
if mime in HC.MIMES_WITH_THUMBNAILS:
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
percentage_in = self._controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
thumbnail = HydrusFileHandling.GenerateThumbnail( path, mime, percentage_in = percentage_in )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( path, bounding_dimensions, mime, percentage_in = percentage_in )
client_files_manager.LocklessAddFullSizeThumbnailFromBytes( hash, thumbnail )
client_files_manager.LocklessAddThumbnailFromBytes( hash, thumbnail_bytes )
result = self._c.execute( 'SELECT 1 FROM local_hashes WHERE hash_id = ?;', ( hash_id, ) ).fetchone()
@ -9762,6 +9832,32 @@ class DB( HydrusDB.HydrusDB ):
self.pub_after_job( 'notify_new_options' )
def _SetIdealClientFilesLocations( self, locations_to_ideal_weights, ideal_thumbnail_override_location ):
if len( locations_to_ideal_weights ) == 0:
raise Exception( 'No locations passed in ideal locations list!' )
self._c.execute( 'DELETE FROM ideal_client_files_locations;' )
for ( abs_location, weight ) in locations_to_ideal_weights.items():
portable_location = HydrusPaths.ConvertAbsPathToPortablePath( abs_location )
self._c.execute( 'INSERT INTO ideal_client_files_locations ( location, weight ) VALUES ( ?, ? );', ( portable_location, weight ) )
self._c.execute( 'DELETE FROM ideal_thumbnail_override_location;' )
if ideal_thumbnail_override_location is not None:
portable_ideal_thumbnail_override_location = HydrusPaths.ConvertAbsPathToPortablePath( ideal_thumbnail_override_location )
self._c.execute( 'INSERT INTO ideal_thumbnail_override_location ( location ) VALUES ( ? );', ( portable_ideal_thumbnail_override_location, ) )
def _SetJSONDump( self, obj ):
if isinstance( obj, HydrusSerialisable.SerialisableBaseNamed ):
@ -11857,6 +11953,86 @@ class DB( HydrusDB.HydrusDB ):
if version == 344:
def wx_code():
message = 'The client now only uses one thumbnail per file (previously it needed two). Your \'resized\' thumbnails will now be deleted. This is a significant step that could take some time to complete. It will also significantly impact your next backup run.'
message += os.linesep * 2
message += 'In order to keep your recycle bin sane, the thumbnails will be permanently deleted. Therefore, this operation cannot be undone. If you are not ready to do this yet (for instance if you do not have a recent backup), kill the hydrus process in Task Manager now.'
message += os.linesep * 2
message += 'BTW: If you previously put your resized thumbnails on an SSD but not your \'full-size\' ones, you should check the \'migrate database\' dialog once the client boots so you can move the remaining thumbnail directories to fast storage.'
wx.MessageBox( message )
self._controller.CallBlockingToWX( None, wx_code )
new_options = self._GetJSONDump( HydrusSerialisable.SERIALISABLE_TYPE_CLIENT_OPTIONS )
self._c.execute( 'CREATE TABLE IF NOT EXISTS ideal_client_files_locations ( location TEXT, weight INTEGER );' )
self._c.execute( 'CREATE TABLE IF NOT EXISTS ideal_thumbnail_override_location ( location TEXT );' )
for ( location, weight ) in new_options._dictionary[ 'client_files_locations_ideal_weights' ]:
self._c.execute( 'INSERT INTO ideal_client_files_locations ( location, weight ) VALUES ( ?, ? );', ( location, weight ) )
thumbnail_override_location = new_options._dictionary[ 'client_files_locations_full_size_thumbnail_override' ]
if thumbnail_override_location is not None:
self._c.execute( 'INSERT INTO ideal_thumbnail_override_location ( location ) VALUES ( ? );', ( thumbnail_override_location, ) )
self._SetJSONDump( new_options )
#
error_occurred = False
for ( i, prefix ) in enumerate( HydrusData.IterateHexPrefixes() ):
self._controller.pub( 'splash_set_status_subtext', 'deleting resized thumbnails {}'.format( HydrusData.ConvertValueRangeToPrettyString( i + 1, 256 ) ) )
resized_prefix = 'r' + prefix
try:
( location, ) = self._c.execute( 'SELECT location FROM client_files_locations WHERE prefix = ?;', ( resized_prefix, ) ).fetchone()
except:
continue
full_path = os.path.join( HydrusPaths.ConvertPortablePathToAbsPath( location ), resized_prefix )
if os.path.exists( full_path ):
try:
HydrusPaths.DeletePath( full_path )
except Exception as e:
HydrusData.PrintException( e )
if not error_occurred:
error_occurred = True
message = 'There was a problem deleting one or more of your old \'rxx\' resized thumbnail directories, perhaps because of some old read-only files. There is no big harm here, since the old directories are no longer needed, but you will want to delete them yourself. Additional error information has been written to the log. Please contact hydrus dev if you need help.'
self.pub_initial_message( message )
self._c.execute( 'DELETE FROM client_files_locations WHERE prefix = ?;', ( resized_prefix, ) )
self._controller.pub( 'splash_set_title_text', 'updated db to v' + str( version + 1 ) )
self._c.execute( 'UPDATE version SET version = ?;', ( version + 1, ) )
@ -12425,6 +12601,7 @@ class DB( HydrusDB.HydrusDB ):
elif action == 'export_mappings': self._ExportToTagArchive( *args, **kwargs )
elif action == 'file_integrity': self._CheckFileIntegrity( *args, **kwargs )
elif action == 'imageboard': self._SetYAMLDump( YAML_DUMP_ID_IMAGEBOARD, *args, **kwargs )
elif action == 'ideal_client_files_locations': self._SetIdealClientFilesLocations( *args, **kwargs )
elif action == 'import_file': result = self._ImportFile( *args, **kwargs )
elif action == 'import_update': self._ImportUpdate( *args, **kwargs )
elif action == 'last_shutdown_work_time': self._SetLastShutdownWorkTime( *args, **kwargs )

5
include/ClientGUI.py
View File

@ -2228,7 +2228,6 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
gui_actions = wx.Menu()
ClientGUIMenus.AppendMenuCheckItem( self, gui_actions, 'thumbnail experiment mode', 'Try the new experiment.', HG.thumbnail_experiment_mode, self._SwitchBoolean, 'thumbnail_experiment_mode' )
ClientGUIMenus.AppendMenuItem( self, gui_actions, 'make some popups', 'Throw some varied popups at the message manager, just to check it is working.', self._DebugMakeSomePopups )
ClientGUIMenus.AppendMenuItem( self, gui_actions, 'make a popup in five seconds', 'Throw a delayed popup at the message manager, giving you time to minimise or otherwise alter the client before it arrives.', self._controller.CallLater, 5, HydrusData.ShowText, 'This is a delayed popup message.' )
ClientGUIMenus.AppendMenuItem( self, gui_actions, 'make a modal popup in five seconds', 'Throw up a delayed modal popup to test with. It will stay alive for five seconds.', self._DebugMakeDelayedModalPopup )
@ -4015,10 +4014,6 @@ The password is cleartext here but obscured in the entry dialog. Enter a blank p
HG.thumbnail_debug_mode = not HG.thumbnail_debug_mode
elif name == 'thumbnail_experiment_mode':
HG.thumbnail_experiment_mode = not HG.thumbnail_experiment_mode
elif name == 'ui_timer_profile_mode':
HG.ui_timer_profile_mode = not HG.ui_timer_profile_mode

26
include/ClientGUICanvas.py
View File

@ -204,7 +204,7 @@ def CalculateMediaContainerSize( media, zoom, action ):
if media.GetMime() in HC.MIMES_WITH_THUMBNAILS:
( thumb_width, thumb_height ) = HydrusImageHandling.GetThumbnailResolution( media.GetResolution(), HC.UNSCALED_THUMBNAIL_DIMENSIONS )
( thumb_width, thumb_height ) = HydrusImageHandling.GetThumbnailResolution( media.GetResolution(), HG.client_controller.options[ 'thumbnail_dimensions' ] )
height = height + thumb_height
@ -385,14 +385,19 @@ class Animation( wx.Window ):
def EventPaint( self, event ):
if self._video_container is None:
if self._canvas_bmp is None:
return
if self._video_container is None and self._media is not None:
self._video_container = ClientRendering.RasterContainerVideo( self._media, self.GetClientSize(), init_position = self._current_frame_index )
dc = wx.BufferedPaintDC( self, self._canvas_bmp )
if not self._a_frame_has_been_drawn:
if not self._a_frame_has_been_drawn or self._media is not None:
self._DrawWhite( dc )
@ -5511,7 +5516,7 @@ class EmbedButton( wx.Window ):
hash = self._media.GetHash()
mime = self._media.GetMime()
thumbnail_path = HG.client_controller.client_files_manager.GetFullSizeThumbnailPath( hash, mime )
thumbnail_path = HG.client_controller.client_files_manager.GetThumbnailPath( hash, mime )
self._thumbnail_bmp = ClientRendering.GenerateHydrusBitmap( thumbnail_path, mime ).GetWxBitmap()
@ -5542,15 +5547,15 @@ class OpenExternallyPanel( wx.Panel ):
hash = self._media.GetHash()
mime = self._media.GetMime()
thumbnail_path = HG.client_controller.client_files_manager.GetFullSizeThumbnailPath( hash, mime )
thumbnail_path = HG.client_controller.client_files_manager.GetThumbnailPath( hash, mime )
bmp = ClientRendering.GenerateHydrusBitmap( thumbnail_path, mime ).GetWxBitmap()
thumbnail = ClientGUICommon.BufferedWindowIcon( self, bmp )
thumbnail_window = ClientGUICommon.BufferedWindowIcon( self, bmp )
thumbnail.Bind( wx.EVT_LEFT_DOWN, self.EventButton )
thumbnail_window.Bind( wx.EVT_LEFT_DOWN, self.EventButton )
vbox.Add( thumbnail, CC.FLAGS_CENTER )
vbox.Add( thumbnail_window, CC.FLAGS_CENTER )
m_text = HC.mime_string_lookup[ media.GetMime() ]
@ -5655,6 +5660,11 @@ class StaticImage( wx.Window ):
def EventPaint( self, event ):
if self._canvas_bmp is None:
return
dc = wx.BufferedPaintDC( self, self._canvas_bmp )
if self._dirty:

5
include/ClientGUIPopupMessages.py
View File

@ -1119,6 +1119,11 @@ class PopupMessageDialogPanel( ClientGUIScrolledPanels.ReviewPanel ):
continue
if ClientGUICommon.IsWXAncestor( self, tlw, through_tlws = True ):
continue
from . import ClientGUI
if isinstance( tlw, ClientGUI.FrameGUI ):

4
include/ClientGUIScrolledPanelsManagement.py
View File

@ -4058,8 +4058,8 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
self._new_options = new_options
self._thumbnail_width = wx.SpinCtrl( self, min = 20, max = 200 )
self._thumbnail_height = wx.SpinCtrl( self, min = 20, max = 200 )
self._thumbnail_width = wx.SpinCtrl( self, min = 20, max = 2048 )
self._thumbnail_height = wx.SpinCtrl( self, min = 20, max = 2048 )
self._thumbnail_border = wx.SpinCtrl( self, min = 0, max = 20 )
self._thumbnail_margin = wx.SpinCtrl( self, min = 0, max = 20 )

293
include/ClientGUIScrolledPanelsReview.py
View File

@ -319,8 +319,7 @@ class AdvancedContentUpdatePanel( ClientGUIScrolledPanels.ReviewPanel ):
class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
RESIZED_RATIO = 0.012
FULLSIZE_RATIO = 0.016
THUMBNAIL_RATIO = 0.012
def __init__( self, parent, controller ):
@ -330,6 +329,10 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
ClientGUIScrolledPanels.ReviewPanel.__init__( self, parent )
self._prefixes_to_locations = HG.client_controller.Read( 'client_files_locations' )
( self._locations_to_ideal_weights, self._ideal_thumbnails_location_override ) = self._controller.Read( 'ideal_client_files_locations' )
service_info = HG.client_controller.Read( 'service_info', CC.COMBINED_LOCAL_FILE_SERVICE_KEY )
self._all_local_files_total_size = service_info[ HC.SERVICE_INFO_TOTAL_SIZE ]
@ -366,17 +369,12 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
current_media_locations_listctrl_panel.AddButton( 'increase file weight', self._IncreaseWeight, enabled_check_func = self._CanIncreaseWeight )
current_media_locations_listctrl_panel.AddButton( 'decrease file weight', self._DecreaseWeight, enabled_check_func = self._CanDecreaseWeight )
self._resized_thumbs_location = wx.TextCtrl( info_panel )
self._resized_thumbs_location.Disable()
self._thumbnails_location = wx.TextCtrl( info_panel )
self._thumbnails_location.Disable()
self._fullsize_thumbs_location = wx.TextCtrl( info_panel )
self._fullsize_thumbs_location.Disable()
self._thumbnails_location_set = ClientGUICommon.BetterButton( info_panel, 'set', self._SetThumbnailLocation )
self._resized_thumbs_location_set = ClientGUICommon.BetterButton( info_panel, 'set', self._SetResizedThumbnailLocation )
self._fullsize_thumbs_location_set = ClientGUICommon.BetterButton( info_panel, 'set', self._SetFullsizeThumbnailLocation )
self._resized_thumbs_location_clear = ClientGUICommon.BetterButton( info_panel, 'clear', self._ClearResizedThumbnailLocation )
self._fullsize_thumbs_location_clear = ClientGUICommon.BetterButton( info_panel, 'clear', self._ClearFullsizeThumbnailLocation )
self._thumbnails_location_clear = ClientGUICommon.BetterButton( info_panel, 'clear', self._ClearThumbnailLocation )
self._rebalance_status_st = ClientGUICommon.BetterStaticText( info_panel, style = wx.ALIGN_RIGHT | wx.ST_NO_AUTORESIZE )
@ -390,19 +388,12 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
#
r_hbox = wx.BoxSizer( wx.HORIZONTAL )
r_hbox.Add( ClientGUICommon.BetterStaticText( info_panel, 'resized thumbnail location' ), CC.FLAGS_VCENTER )
r_hbox.Add( self._resized_thumbs_location, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
r_hbox.Add( self._resized_thumbs_location_set, CC.FLAGS_VCENTER )
r_hbox.Add( self._resized_thumbs_location_clear, CC.FLAGS_VCENTER )
t_hbox = wx.BoxSizer( wx.HORIZONTAL )
t_hbox.Add( ClientGUICommon.BetterStaticText( info_panel, 'full-size thumbnail location' ), CC.FLAGS_VCENTER )
t_hbox.Add( self._fullsize_thumbs_location, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
t_hbox.Add( self._fullsize_thumbs_location_set, CC.FLAGS_VCENTER )
t_hbox.Add( self._fullsize_thumbs_location_clear, CC.FLAGS_VCENTER )
t_hbox.Add( ClientGUICommon.BetterStaticText( info_panel, 'thumbnail location override' ), CC.FLAGS_VCENTER )
t_hbox.Add( self._thumbnails_location, CC.FLAGS_VCENTER_EXPAND_DEPTH_ONLY )
t_hbox.Add( self._thumbnails_location_set, CC.FLAGS_VCENTER )
t_hbox.Add( self._thumbnails_location_clear, CC.FLAGS_VCENTER )
rebalance_hbox = wx.BoxSizer( wx.HORIZONTAL )
@ -413,7 +404,6 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
info_panel.Add( self._current_db_path_st, CC.FLAGS_EXPAND_PERPENDICULAR )
info_panel.Add( self._current_media_paths_st, CC.FLAGS_EXPAND_PERPENDICULAR )
info_panel.Add( current_media_locations_listctrl_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
info_panel.Add( r_hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
info_panel.Add( t_hbox, CC.FLAGS_EXPAND_PERPENDICULAR )
info_panel.Add( rebalance_hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
@ -440,31 +430,29 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _AddPath( self, path, starting_weight = 1 ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
if path in locations_to_ideal_weights:
if path in self._locations_to_ideal_weights:
wx.MessageBox( 'You already have that location entered!' )
return
if path == resized_thumbnail_override or path == full_size_thumbnail_override:
if path == self._ideal_thumbnails_location_override:
wx.MessageBox( 'That path is already used as a special thumbnail location--please choose another.' )
wx.MessageBox( 'That path is already used as the special thumbnail location--please choose another.' )
return
self._new_options.SetClientFilesLocation( path, 1 )
self._locations_to_ideal_weights[ path ] = 1
self._controller.Write( 'ideal_client_files_locations', self._locations_to_ideal_weights, self._ideal_thumbnails_location_override )
self._Update()
def _AdjustWeight( self, amount ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
adjustees = set()
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
@ -473,15 +461,17 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
location = locations[0]
if location in locations_to_ideal_weights:
if location in self._locations_to_ideal_weights:
current_weight = locations_to_ideal_weights[ location ]
current_weight = self._locations_to_ideal_weights[ location ]
new_amount = current_weight + amount
if new_amount > 0:
self._new_options.SetClientFilesLocation( location, new_amount )
self._locations_to_ideal_weights[ location ] = new_amount
self._controller.Write( 'ideal_client_files_locations', self._locations_to_ideal_weights, self._ideal_thumbnails_location_override )
elif new_amount <= 0:
@ -492,7 +482,7 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if amount > 0:
if location not in ( resized_thumbnail_override, full_size_thumbnail_override ):
if location != self._ideal_thumbnails_location_override:
self._AddPath( location, starting_weight = amount )
@ -506,22 +496,20 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _CanDecreaseWeight( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
if len( locations ) > 0:
location = locations[0]
if location in locations_to_ideal_weights:
if location in self._locations_to_ideal_weights:
selection_includes_ideal_locations = True
ideal_weight = locations_to_ideal_weights[ location ]
ideal_weight = self._locations_to_ideal_weights[ location ]
is_big = ideal_weight > 1
others_can_take_slack = len( locations_to_ideal_weights ) > 1
others_can_take_slack = len( self._locations_to_ideal_weights ) > 1
if is_big or others_can_take_slack:
@ -535,9 +523,7 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _CanIncreaseWeight( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
( locations_to_file_weights, locations_to_thumb_weights ) = self._GetLocationsToCurrentWeights()
locations = self._current_media_locations_listctrl.GetData( only_selected = True )
@ -545,9 +531,9 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
location = locations[0]
if location in locations_to_ideal_weights:
if location in self._locations_to_ideal_weights:
if len( locations_to_ideal_weights ) > 1:
if len( self._locations_to_ideal_weights ) > 1:
return True
@ -561,16 +547,11 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
return False
def _ClearFullsizeThumbnailLocation( self ):
def _ClearThumbnailLocation( self ):
self._new_options.SetFullsizeThumbnailOverride( None )
self._ideal_thumbnails_location_override = None
self._Update()
def _ClearResizedThumbnailLocation( self ):
self._new_options.SetResizedThumbnailOverride( None )
self._controller.Write( 'ideal_client_files_locations', self._locations_to_ideal_weights, self._ideal_thumbnails_location_override )
self._Update()
@ -578,16 +559,11 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _ConvertLocationToListCtrlTuples( self, location ):
f_space = self._all_local_files_total_size
r_space = self._all_local_files_total_size * self.RESIZED_RATIO
t_space = self._all_local_files_total_size * self.FULLSIZE_RATIO
# ideal
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
t_space = self._all_local_files_total_size * self.THUMBNAIL_RATIO
# current
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
( locations_to_file_weights, locations_to_thumb_weights ) = self._GetLocationsToCurrentWeights()
#
@ -624,14 +600,13 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
fp = locations_to_file_weights[ location ] / 256.0
tp = locations_to_fs_thumb_weights[ location ] / 256.0
rp = locations_to_r_thumb_weights[ location ] / 256.0
tp = locations_to_thumb_weights[ location ] / 256.0
p = HydrusData.ConvertFloatToPercentage
current_bytes = fp * f_space + tp * t_space + rp * r_space
current_bytes = fp * f_space + tp * t_space
current_usage = ( fp, tp, rp )
current_usage = ( fp, tp )
usages = []
@ -642,17 +617,12 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if tp > 0:
usages.append( p( tp ) + ' full-size thumbnails' )
if rp > 0:
usages.append( p( rp ) + ' resized thumbnails' )
usages.append( p( tp ) + ' thumbnails' )
if len( usages ) > 0:
if fp == tp and tp == rp:
if fp == tp:
usages = [ p( fp ) + ' everything' ]
@ -666,9 +636,9 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
#
if location in locations_to_ideal_weights:
if location in self._locations_to_ideal_weights:
ideal_weight = locations_to_ideal_weights[ location ]
ideal_weight = self._locations_to_ideal_weights[ location ]
pretty_ideal_weight = str( int( ideal_weight ) )
@ -686,24 +656,24 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if location in locations_to_ideal_weights:
if location in self._locations_to_ideal_weights:
total_ideal_weight = sum( locations_to_ideal_weights.values() )
total_ideal_weight = sum( self._locations_to_ideal_weights.values() )
ideal_fp = locations_to_ideal_weights[ location ] / total_ideal_weight
ideal_fp = self._locations_to_ideal_weights[ location ] / total_ideal_weight
else:
ideal_fp = 0.0
if full_size_thumbnail_override is None:
if self._ideal_thumbnails_location_override is None:
ideal_tp = ideal_fp
else:
if location == full_size_thumbnail_override:
if location == self._ideal_thumbnails_location_override:
ideal_tp = 1.0
@ -713,25 +683,9 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if resized_thumbnail_override is None:
ideal_rp = ideal_fp
else:
if location == resized_thumbnail_override:
ideal_rp = 1.0
else:
ideal_rp = 0.0
ideal_bytes = ideal_fp * f_space + ideal_tp * t_space
ideal_bytes = ideal_fp * f_space + ideal_tp * t_space + ideal_rp * r_space
ideal_usage = ( ideal_fp, ideal_tp, ideal_rp )
ideal_usage = ( ideal_fp, ideal_tp )
usages = []
@ -742,17 +696,12 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if ideal_tp > 0:
usages.append( p( ideal_tp ) + ' full-size thumbnails' )
if ideal_rp > 0:
usages.append( p( ideal_rp ) + ' resized thumbnails' )
usages.append( p( ideal_tp ) + ' thumbnails' )
if len( usages ) > 0:
if ideal_fp == ideal_tp and ideal_tp == ideal_rp:
if ideal_fp == ideal_tp:
usages = [ p( ideal_fp ) + ' everything' ]
@ -777,13 +726,10 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _GetLocationsToCurrentWeights( self ):
prefixes_to_locations = HG.client_controller.Read( 'client_files_locations' )
locations_to_file_weights = collections.Counter()
locations_to_fs_thumb_weights = collections.Counter()
locations_to_r_thumb_weights = collections.Counter()
locations_to_thumb_weights = collections.Counter()
for ( prefix, location ) in list(prefixes_to_locations.items()):
for ( prefix, location ) in list(self._prefixes_to_locations.items()):
if prefix.startswith( 'f' ):
@ -792,45 +738,32 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if prefix.startswith( 't' ):
locations_to_fs_thumb_weights[ location ] += 1
if prefix.startswith( 'r' ):
locations_to_r_thumb_weights[ location ] += 1
locations_to_thumb_weights[ location ] += 1
return ( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights )
return ( locations_to_file_weights, locations_to_thumb_weights )
def _GetListCtrlLocations( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
# current
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
( locations_to_file_weights, locations_to_thumb_weights ) = self._GetLocationsToCurrentWeights()
#
all_locations = set()
all_locations.update( list(locations_to_ideal_weights.keys()) )
all_locations.update( list(self._locations_to_ideal_weights.keys()) )
if resized_thumbnail_override is not None:
if self._ideal_thumbnails_location_override is not None:
all_locations.add( resized_thumbnail_override )
if full_size_thumbnail_override is not None:
all_locations.add( full_size_thumbnail_override )
all_locations.add( self._ideal_thumbnails_location_override )
all_locations.update( list(locations_to_file_weights.keys()) )
all_locations.update( list(locations_to_fs_thumb_weights.keys()) )
all_locations.update( list(locations_to_r_thumb_weights.keys()) )
all_locations.update( list(locations_to_thumb_weights.keys()) )
all_locations = list( all_locations )
@ -901,11 +834,9 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
wx.CallAfter( self.GetParent().Close )
prefixes_to_locations = self._controller.Read( 'client_files_locations' )
portable_locations = []
for location in set( prefixes_to_locations.values() ):
for location in set( self._prefixes_to_locations.values() ):
if not os.path.exists( location ):
@ -952,20 +883,18 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _RemovePath( self, location ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
( locations_to_file_weights, locations_to_fs_thumb_weights, locations_to_r_thumb_weights ) = self._GetLocationsToCurrentWeights()
( locations_to_file_weights, locations_to_thumb_weights ) = self._GetLocationsToCurrentWeights()
removees = set()
if location not in locations_to_ideal_weights:
if location not in self._locations_to_ideal_weights:
wx.MessageBox( 'Please select a location with weight.' )
return
if len( locations_to_ideal_weights ) == 1:
if len( self._locations_to_ideal_weights ) == 1:
wx.MessageBox( 'You cannot empty every single current file location--please add a new place for the files to be moved to and then try again.' )
@ -983,12 +912,13 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
if dlg.ShowModal() == wx.ID_YES:
self._new_options.RemoveClientFilesLocation( location )
del self._locations_to_ideal_weights[ location ]
self._controller.Write( 'ideal_client_files_locations', self._locations_to_ideal_weights, self._ideal_thumbnails_location_override )
self._Update()
def _SelectPathToAdd( self ):
@ -1004,57 +934,28 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _SetFullsizeThumbnailLocation( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
def _SetThumbnailLocation( self ):
with wx.DirDialog( self ) as dlg:
if full_size_thumbnail_override is not None:
if self._ideal_thumbnails_location_override is not None:
dlg.SetPath( full_size_thumbnail_override )
dlg.SetPath( self._ideal_thumbnails_location_override )
if dlg.ShowModal() == wx.ID_OK:
path = dlg.GetPath()
if path in locations_to_ideal_weights:
if path in self._locations_to_ideal_weights:
wx.MessageBox( 'That path already exists as a regular file location! Please choose another.' )
else:
self._new_options.SetFullsizeThumbnailOverride( path )
self._ideal_thumbnails_location_override = path
self._Update()
def _SetResizedThumbnailLocation( self ):
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
with wx.DirDialog( self ) as dlg:
if resized_thumbnail_override is not None:
dlg.SetPath( resized_thumbnail_override )
if dlg.ShowModal() == wx.ID_OK:
path = dlg.GetPath()
if path in locations_to_ideal_weights:
wx.MessageBox( 'That path already exists as a regular file location! Please choose another.' )
else:
self._new_options.SetResizedThumbnailOverride( path )
self._controller.Write( 'ideal_client_files_locations', self._locations_to_ideal_weights, self._ideal_thumbnails_location_override )
self._Update()
@ -1064,22 +965,19 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
def _Update( self ):
self._prefixes_to_locations = HG.client_controller.Read( 'client_files_locations' )
( self._locations_to_ideal_weights, self._ideal_thumbnails_location_override ) = self._controller.Read( 'ideal_client_files_locations' )
approx_total_db_size = self._controller.db.GetApproxTotalFileSize()
self._current_db_path_st.SetLabelText( 'database (about ' + HydrusData.ToHumanBytes( approx_total_db_size ) + '): ' + self._controller.GetDBDir() )
self._current_install_path_st.SetLabelText( 'install: ' + HC.BASE_DIR )
approx_total_client_files = self._all_local_files_total_size
approx_total_resized_thumbs = self._all_local_files_total_size * self.RESIZED_RATIO
approx_total_fullsize_thumbs = self._all_local_files_total_size * self.FULLSIZE_RATIO
approx_total_thumbnails = self._all_local_files_total_size * self.THUMBNAIL_RATIO
label_components = []
label_components.append( 'media (about ' + HydrusData.ToHumanBytes( approx_total_client_files ) + ')' )
label_components.append( 'resized thumbnails (about ' + HydrusData.ToHumanBytes( approx_total_resized_thumbs ) + ')' )
label_components.append( 'full-size thumbnails (about ' + HydrusData.ToHumanBytes( approx_total_fullsize_thumbs ) + ')' )
label = ', '.join( label_components ) + ':'
label = 'media is about ' + HydrusData.ToHumanBytes( approx_total_client_files ) + ', thumbnails are about ' + HydrusData.ToHumanBytes( approx_total_thumbnails ) + ':'
self._current_media_paths_st.SetLabelText( label )
@ -1089,36 +987,19 @@ class MigrateDatabasePanel( ClientGUIScrolledPanels.ReviewPanel ):
#
( locations_to_ideal_weights, resized_thumbnail_override, full_size_thumbnail_override ) = self._new_options.GetClientFilesLocationsToIdealWeights()
if resized_thumbnail_override is None:
if self._ideal_thumbnails_location_override is None:
self._resized_thumbs_location.SetValue( 'none set' )
self._thumbnails_location.SetValue( 'none set' )
self._resized_thumbs_location_set.Enable()
self._resized_thumbs_location_clear.Disable()
self._thumbnails_location_set.Enable()
self._thumbnails_location_clear.Disable()
else:
self._resized_thumbs_location.SetValue( resized_thumbnail_override )
self._thumbnails_location.SetValue( self._ideal_thumbnails_location_override )
self._resized_thumbs_location_set.Disable()
self._resized_thumbs_location_clear.Enable()
if full_size_thumbnail_override is None:
self._fullsize_thumbs_location.SetValue( 'none set' )
self._fullsize_thumbs_location_set.Enable()
self._fullsize_thumbs_location_clear.Disable()
else:
self._fullsize_thumbs_location.SetValue( full_size_thumbnail_override )
self._fullsize_thumbs_location_set.Disable()
self._fullsize_thumbs_location_clear.Enable()
self._thumbnails_location_set.Disable()
self._thumbnails_location_clear.Enable()
#

107
include/ClientImageHandling.py
View File

@ -3,6 +3,7 @@ from . import ClientConstants as CC
import cv2
from . import HydrusConstants as HC
from . import HydrusData
from . import HydrusExceptions
from . import HydrusImageHandling
from . import HydrusGlobals as HG
from functools import reduce
@ -34,35 +35,6 @@ cv_interpolation_enum_lookup[ CC.ZOOM_AREA ] = cv2.INTER_AREA
cv_interpolation_enum_lookup[ CC.ZOOM_CUBIC ] = cv2.INTER_CUBIC
cv_interpolation_enum_lookup[ CC.ZOOM_LANCZOS4 ] = cv2.INTER_LANCZOS4
def EfficientlyResizeNumpyImage( numpy_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_y, im_x, depth ) = numpy_image.shape
if target_x >= im_x and target_y >= im_y:
return numpy_image
# this seems to slow things down a lot, at least for cv!
#if im_x > 2 * target_x and im_y > 2 * target_y: result = cv2.resize( numpy_image, ( 2 * target_x, 2 * target_y ), interpolation = cv2.INTER_NEAREST )
return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = cv2.INTER_AREA )
def EfficientlyThumbnailNumpyImage( numpy_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_y, im_x, depth ) = numpy_image.shape
if target_x >= im_x and target_y >= im_y:
return numpy_image
( target_x, target_y ) = HydrusImageHandling.GetThumbnailResolution( ( im_x, im_y ), ( target_x, target_y ) )
return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = cv2.INTER_AREA )
def GenerateNumpyImage( path, mime ):
if HG.media_load_report_mode:
@ -169,7 +141,7 @@ def GenerateShapePerceptualHashes( path, mime ):
if depth == 4:
# doing this on 10000x10000 pngs eats ram like mad
numpy_image = EfficientlyThumbnailNumpyImage( numpy_image, ( 1024, 1024 ) )
numpy_image = ThumbnailNumpyImage( numpy_image, ( 1024, 1024 ) )
( y, x, depth ) = numpy_image.shape
@ -272,23 +244,9 @@ def GenerateShapePerceptualHashes( path, mime ):
return phashes
def GenerateThumbnailFileBytesFromStaticImagePathCV( path, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS, mime = None ):
def GenerateBytesFromCV( numpy_image, mime ):
if mime is None:
mime = HydrusFileHandling.GetMime( path )
if mime == HC.IMAGE_GIF:
return HydrusFileHandling.GenerateThumbnailFileBytesFromStaticImagePathPIL( path, dimensions, mime )
numpy_image = GenerateNumpyImage( path, mime )
thumbnail_numpy_image = EfficientlyThumbnailNumpyImage( numpy_image, dimensions )
( im_y, im_x, depth ) = thumbnail_numpy_image.shape
( im_y, im_x, depth ) = numpy_image.shape
if depth == 4:
@ -299,7 +257,7 @@ def GenerateThumbnailFileBytesFromStaticImagePathCV( path, dimensions = HC.UNSCA
convert = cv2.COLOR_RGB2BGR
thumbnail_numpy_image = cv2.cvtColor( thumbnail_numpy_image, convert )
numpy_image = cv2.cvtColor( numpy_image, convert )
if mime == HC.IMAGE_JPEG:
@ -314,22 +272,44 @@ def GenerateThumbnailFileBytesFromStaticImagePathCV( path, dimensions = HC.UNSCA
params = CV_PNG_THUMBNAIL_ENCODE_PARAMS
( result_success, result_byte_array ) = cv2.imencode( ext, thumbnail_numpy_image, params )
( result_success, result_byte_array ) = cv2.imencode( ext, numpy_image, params )
if result_success:
thumbnail = result_byte_array.tostring()
thumbnail_bytes = result_byte_array.tostring()
return thumbnail
return thumbnail_bytes
else:
return HydrusFileHandling.GenerateThumbnailFileBytesFromStaticImagePathPIL( path, dimensions, mime )
raise HydrusExceptions.CantRenderWithCVException( 'Thumb failed to encode!' )
def GenerateThumbnailBytesFromStaticImagePathCV( path, bounding_dimensions, mime ):
if mime == HC.IMAGE_GIF:
return HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
numpy_image = GenerateNumpyImage( path, mime )
thumbnail_numpy_image = ThumbnailNumpyImage( numpy_image, bounding_dimensions )
try:
thumbnail_bytes = GenerateBytesFromCV( thumbnail_numpy_image, mime )
return thumbnail_bytes
except HydrusExceptions.CantRenderWithCVException:
return HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
from . import HydrusFileHandling
HydrusFileHandling.GenerateThumbnailFileBytesFromStaticImagePath = GenerateThumbnailFileBytesFromStaticImagePathCV
HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePath = GenerateThumbnailBytesFromStaticImagePathCV
def GetNumPyImageResolution( numpy_image ):
@ -337,7 +317,14 @@ def GetNumPyImageResolution( numpy_image ):
return ( image_x, image_y )
def ResizeNumpyImage( mime, numpy_image, target_resolution ):
def ResizeNumpyImage( numpy_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_y, im_x, depth ) = numpy_image.shape
return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = cv2.INTER_AREA )
def ResizeNumpyImageForMediaViewer( mime, numpy_image, target_resolution ):
( target_x, target_y ) = target_resolution
new_options = HG.client_controller.new_options
@ -364,3 +351,17 @@ def ResizeNumpyImage( mime, numpy_image, target_resolution ):
return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = interpolation )
def ThumbnailNumpyImage( numpy_image, bounding_dimensions ):
( target_x, target_y ) = bounding_dimensions
( im_y, im_x, depth ) = numpy_image.shape
if target_x >= im_x and target_y >= im_y:
return numpy_image
( target_x, target_y ) = HydrusImageHandling.GetThumbnailResolution( ( im_x, im_y ), ( target_x, target_y ) )
return cv2.resize( numpy_image, ( target_x, target_y ), interpolation = cv2.INTER_AREA )

4
include/ClientImportFileSeeds.py
View File

@ -286,9 +286,11 @@ class FileImportJob( object ):
if mime in HC.MIMES_WITH_THUMBNAILS:
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
percentage_in = HG.client_controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
self._thumbnail = HydrusFileHandling.GenerateThumbnail( self._temp_path, mime, percentage_in = percentage_in )
self._thumbnail = HydrusFileHandling.GenerateThumbnailBytes( self._temp_path, bounding_dimensions, mime, percentage_in = percentage_in )
if mime in HC.MIMES_WE_CAN_PHASH:

6
include/ClientLocalServerResources.py
View File

@ -462,7 +462,7 @@ class HydrusResourceBooruThumbnail( HydrusResourceBooru ):
client_files_manager = HG.client_controller.client_files_manager
path = client_files_manager.GetFullSizeThumbnailPath( hash )
path = client_files_manager.GetThumbnailPath( hash )
response_context_mime = HC.APPLICATION_UNKNOWN
@ -1252,7 +1252,7 @@ class HydrusResourceClientAPIRestrictedGetFilesFileMetadata( HydrusResourceClien
if only_return_identifiers:
file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', file_ids = file_ids )
file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', hash_ids = file_ids )
else:
@ -1395,7 +1395,7 @@ class HydrusResourceClientAPIRestrictedGetFilesGetThumbnail( HydrusResourceClien
hash = media_result.GetHash()
mime = media_result.GetMime()
path = HG.client_controller.client_files_manager.GetFullSizeThumbnailPath( hash, mime )
path = HG.client_controller.client_files_manager.GetThumbnailPath( hash, mime )
except HydrusExceptions.FileMissingException:

10
include/ClientNetworkingDomain.py
View File

@ -1383,7 +1383,15 @@ class NetworkDomainManager( HydrusSerialisable.SerialisableBase ):
( url_to_fetch, parser ) = result
HydrusData.ShowText( 'URL to fetch and parser request: ' + url + ' -> ' + url_to_fetch + ': ' + parser.GetName() )
url_match = self._GetURLMatch( url )
url_name = url_match.GetName()
url_to_fetch_match = self._GetURLMatch( url_to_fetch )
url_to_fetch_name = url_to_fetch_match.GetName()
HydrusData.ShowText( 'request for URL to fetch and parser: {} ({}) -> {} ({}): {}'.format( url, url_name, url_to_fetch, url_to_fetch_name, parser.GetName() ) )
return result

95
include/ClientOptions.py
View File

@ -19,20 +19,15 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_NAME = 'Client Options'
SERIALISABLE_VERSION = 3
def __init__( self, db_dir = None ):
def __init__( self ):
HydrusSerialisable.SerialisableBase.__init__( self )
if db_dir is None:
db_dir = HC.DEFAULT_DB_DIR
self._dictionary = HydrusSerialisable.SerialisableDictionary()
self._lock = threading.Lock()
self._InitialiseDefaults( db_dir )
self._InitialiseDefaults()
def _GetSerialisableInfo( self ):
@ -42,7 +37,7 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
return serialisable_info
def _InitialiseDefaults( self, db_dir ):
def _InitialiseDefaults( self ):
self._dictionary[ 'booleans' ] = {}
@ -349,14 +344,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
#
client_files_default = os.path.join( db_dir, 'client_files' )
self._dictionary[ 'client_files_locations_ideal_weights' ] = [ ( HydrusPaths.ConvertAbsPathToPortablePath( client_files_default ), 1.0 ) ]
self._dictionary[ 'client_files_locations_resized_thumbnail_override' ] = None
self._dictionary[ 'client_files_locations_full_size_thumbnail_override' ] = None
#
self._dictionary[ 'default_file_import_options' ] = HydrusSerialisable.SerialisableDictionary()
exclude_deleted = True
@ -648,38 +635,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def GetClientFilesLocationsToIdealWeights( self ):
with self._lock:
paths_to_weights = {}
for ( portable_path, weight ) in self._dictionary[ 'client_files_locations_ideal_weights' ]:
abs_path = HydrusPaths.ConvertPortablePathToAbsPath( portable_path )
paths_to_weights[ abs_path ] = weight
resized_thumbnail_override = self._dictionary[ 'client_files_locations_resized_thumbnail_override' ]
if resized_thumbnail_override is not None:
resized_thumbnail_override = HydrusPaths.ConvertPortablePathToAbsPath( resized_thumbnail_override )
full_size_thumbnail_override = self._dictionary[ 'client_files_locations_full_size_thumbnail_override' ]
if full_size_thumbnail_override is not None:
full_size_thumbnail_override = HydrusPaths.ConvertPortablePathToAbsPath( full_size_thumbnail_override )
return ( paths_to_weights, resized_thumbnail_override, full_size_thumbnail_override )
def GetColour( self, colour_type, colourset = None ):
with self._lock:
@ -937,21 +892,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def RemoveClientFilesLocation( self, location ):
with self._lock:
if len( self._dictionary[ 'client_files_locations_ideal_weights' ] ) < 2:
raise Exception( 'Cannot remove any more files locations!' )
portable_location = HydrusPaths.ConvertAbsPathToPortablePath( location )
self._dictionary[ 'client_files_locations_ideal_weights' ] = [ ( l, w ) for ( l, w ) in self._dictionary[ 'client_files_locations_ideal_weights' ] if l != portable_location ]
def SetBoolean( self, name, value ):
with self._lock:
@ -960,19 +900,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def SetClientFilesLocation( self, location, weight ):
with self._lock:
portable_location = HydrusPaths.ConvertAbsPathToPortablePath( location )
weight = float( weight )
self._dictionary[ 'client_files_locations_ideal_weights' ] = [ ( l, w ) for ( l, w ) in self._dictionary[ 'client_files_locations_ideal_weights' ] if l != portable_location ]
self._dictionary[ 'client_files_locations_ideal_weights' ].append( ( portable_location, weight ) )
def SetColour( self, colour_type, colourset, colour ):
with self._lock:
@ -1046,14 +973,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def SetFullsizeThumbnailOverride( self, full_size_thumbnail_override ):
with self._lock:
self._dictionary[ 'client_files_locations_full_size_thumbnail_override' ] = full_size_thumbnail_override
def SetInteger( self, name, value ):
with self._lock:
@ -1118,14 +1037,6 @@ class ClientOptions( HydrusSerialisable.SerialisableBase ):
def SetResizedThumbnailOverride( self, resized_thumbnail_override ):
with self._lock:
self._dictionary[ 'client_files_locations_resized_thumbnail_override' ] = resized_thumbnail_override
def SetSimpleDownloaderFormulae( self, simple_downloader_formulae ):
with self._lock:

6
include/ClientRendering.py
View File

@ -56,11 +56,11 @@ def GenerateHydrusBitmap( path, mime, compressed = True, desired_dimensions = No
if do_thumbnail_resize:
numpy_image = ClientImageHandling.EfficientlyThumbnailNumpyImage( numpy_image, desired_dimensions )
numpy_image = ClientImageHandling.ThumbnailNumpyImage( numpy_image, desired_dimensions )
else:
numpy_image = ClientImageHandling.EfficientlyResizeNumpyImage( numpy_image, desired_dimensions )
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, desired_dimensions )
@ -141,7 +141,7 @@ class ImageRenderer( object ):
else:
wx_numpy_image = ClientImageHandling.ResizeNumpyImage( self._media.GetMime(), self._numpy_image, target_resolution )
wx_numpy_image = ClientImageHandling.ResizeNumpyImageForMediaViewer( self._media.GetMime(), self._numpy_image, target_resolution )
( wx_height, wx_width, wx_depth ) = wx_numpy_image.shape

37
include/ClientSearch.py
View File

@ -164,19 +164,24 @@ def SortPredicates( predicates ):
return predicates
SEARCH_TYPE_AND = 0
SEARCH_TYPE_OR = 1
class FileSearchContext( HydrusSerialisable.SerialisableBase ):
SERIALISABLE_TYPE = HydrusSerialisable.SERIALISABLE_TYPE_FILE_SEARCH_CONTEXT
SERIALISABLE_NAME = 'File Search Context'
SERIALISABLE_VERSION = 1
SERIALISABLE_VERSION = 2
def __init__( self, file_service_key = CC.COMBINED_FILE_SERVICE_KEY, tag_service_key = CC.COMBINED_TAG_SERVICE_KEY, include_current_tags = True, include_pending_tags = True, predicates = None ):
def __init__( self, file_service_key = CC.COMBINED_FILE_SERVICE_KEY, tag_service_key = CC.COMBINED_TAG_SERVICE_KEY, search_type = SEARCH_TYPE_AND, include_current_tags = True, include_pending_tags = True, predicates = None ):
if predicates is None: predicates = []
self._file_service_key = file_service_key
self._tag_service_key = tag_service_key
self._search_type = search_type
self._include_current_tags = include_current_tags
self._include_pending_tags = include_pending_tags
@ -191,12 +196,12 @@ class FileSearchContext( HydrusSerialisable.SerialisableBase ):
serialisable_predicates = [ predicate.GetSerialisableTuple() for predicate in self._predicates ]
return ( self._file_service_key.hex(), self._tag_service_key.hex(), self._include_current_tags, self._include_pending_tags, serialisable_predicates, self._search_complete )
return ( self._file_service_key.hex(), self._tag_service_key.hex(), self._search_type, self._include_current_tags, self._include_pending_tags, serialisable_predicates, self._search_complete )
def _InitialiseFromSerialisableInfo( self, serialisable_info ):
( file_service_key, tag_service_key, self._include_current_tags, self._include_pending_tags, serialisable_predicates, self._search_complete ) = serialisable_info
( file_service_key, tag_service_key, self._include_current_tags, self._search_type, self._include_pending_tags, serialisable_predicates, self._search_complete ) = serialisable_info
self._file_service_key = bytes.fromhex( file_service_key )
self._tag_service_key = bytes.fromhex( tag_service_key )
@ -250,7 +255,7 @@ class FileSearchContext( HydrusSerialisable.SerialisableBase ):
else: self._namespaces_to_exclude.append( namespace )
wildcard_predicates = [ predicate for predicate in self._predicates if predicate.GetType() == HC.PREDICATE_TYPE_WILDCARD ]
wildcard_predicates = [ predicate for predicate in self._predicates if predicate.GetType() == HC.PREDICATE_TYPE_WILDCARD ]
self._wildcards_to_include = []
self._wildcards_to_exclude = []
@ -263,10 +268,27 @@ class FileSearchContext( HydrusSerialisable.SerialisableBase ):
else: self._wildcards_to_exclude.append( wildcard )
self._or_predicates = [ predicate for predicate in self._predicates if predicate.GetType() == HC.PREDICATE_TYPE_OR_CONTAINER ]
def _UpdateSerialisableInfo( self, version, old_serialisable_info ):
if version == 1:
( file_service_key_hex, tag_service_key_hex, include_current_tags, include_pending_tags, serialisable_predicates, search_complete ) = old_serialisable_info
search_type = SEARCH_TYPE_AND
new_serialisable_info = ( file_service_key_hex, tag_service_key_hex, search_type, include_current_tags, include_pending_tags, serialisable_predicates, search_complete )
return ( 2, new_serialisable_info )
def GetFileServiceKey( self ): return self._file_service_key
def GetNamespacesToExclude( self ): return self._namespaces_to_exclude
def GetNamespacesToInclude( self ): return self._namespaces_to_include
def GetORPredicates( self ): return self._or_predicates
def GetPredicates( self ): return self._predicates
def GetSystemPredicates( self ): return self._system_predicates
def GetTagServiceKey( self ): return self._tag_service_key
@ -1499,7 +1521,10 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
return self.GetCopy()
def GetValue( self ): return self._value
def GetValue( self ):
return self._value
def HasNonZeroCount( self ):

4
include/ClientServices.py
View File

@ -1586,7 +1586,7 @@ class ServiceRepository( ServiceRestricted ):
try:
thumbnail = self.Request( HC.GET, 'thumbnail', { 'hash' : thumbnail_hash } )
thumbnail_bytes = self.Request( HC.GET, 'thumbnail', { 'hash' : thumbnail_hash } )
except HydrusExceptions.CancelledException as e:
@ -1603,7 +1603,7 @@ class ServiceRepository( ServiceRestricted ):
return
client_files_manager.AddFullSizeThumbnailFromBytes( thumbnail_hash, thumbnail )
client_files_manager.AddThumbnailFromBytes( thumbnail_hash, thumbnail_bytes )
job_key.SetVariable( 'popup_text_1', 'finished' )

4
include/ClientVideoHandling.py
View File

@ -195,7 +195,7 @@ class GIFRenderer( object ):
numpy_image = self._GetCurrentFrame()
numpy_image = ClientImageHandling.EfficientlyResizeNumpyImage( numpy_image, self._target_resolution )
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, self._target_resolution )
numpy_image = cv2.cvtColor( numpy_image, cv2.COLOR_BGR2RGB )
@ -222,7 +222,7 @@ class GIFRenderer( object ):
numpy_image = self._GetCurrentFrame()
numpy_image = ClientImageHandling.EfficientlyResizeNumpyImage( numpy_image, self._target_resolution )
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, self._target_resolution )
self._last_frame = numpy_image

5
include/HydrusConstants.py
View File

@ -67,10 +67,10 @@ options = {}
# Misc
NETWORK_VERSION = 18
SOFTWARE_VERSION = 344
SOFTWARE_VERSION = 345
CLIENT_API_VERSION = 5
UNSCALED_THUMBNAIL_DIMENSIONS = ( 200, 200 )
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )
HYDRUS_KEY_LENGTH = 32
@ -657,6 +657,7 @@ PREDICATE_TYPE_SYSTEM_DUPLICATE_RELATIONSHIPS = 26
PREDICATE_TYPE_SYSTEM_TAG_AS_NUMBER = 27
PREDICATE_TYPE_SYSTEM_KNOWN_URLS = 28
PREDICATE_TYPE_SYSTEM_FILE_VIEWING_STATS = 29
PREDICATE_TYPE_OR_CONTAINER = 30
SYSTEM_PREDICATES = [ PREDICATE_TYPE_SYSTEM_EVERYTHING, PREDICATE_TYPE_SYSTEM_INBOX, PREDICATE_TYPE_SYSTEM_ARCHIVE, PREDICATE_TYPE_SYSTEM_UNTAGGED, PREDICATE_TYPE_SYSTEM_NUM_TAGS, PREDICATE_TYPE_SYSTEM_LIMIT, PREDICATE_TYPE_SYSTEM_SIZE, PREDICATE_TYPE_SYSTEM_AGE, PREDICATE_TYPE_SYSTEM_HASH, PREDICATE_TYPE_SYSTEM_WIDTH, PREDICATE_TYPE_SYSTEM_HEIGHT, PREDICATE_TYPE_SYSTEM_RATIO, PREDICATE_TYPE_SYSTEM_DURATION, PREDICATE_TYPE_SYSTEM_MIME, PREDICATE_TYPE_SYSTEM_RATING, PREDICATE_TYPE_SYSTEM_SIMILAR_TO, PREDICATE_TYPE_SYSTEM_LOCAL, PREDICATE_TYPE_SYSTEM_NOT_LOCAL, PREDICATE_TYPE_SYSTEM_NUM_WORDS, PREDICATE_TYPE_SYSTEM_FILE_SERVICE, PREDICATE_TYPE_SYSTEM_NUM_PIXELS, PREDICATE_TYPE_SYSTEM_DIMENSIONS, PREDICATE_TYPE_SYSTEM_TAG_AS_NUMBER, PREDICATE_TYPE_SYSTEM_DUPLICATE_RELATIONSHIPS, PREDICATE_TYPE_SYSTEM_KNOWN_URLS, PREDICATE_TYPE_SYSTEM_FILE_VIEWING_STATS ]

38
include/HydrusFileHandling.py
View File

@ -53,7 +53,7 @@ header_and_mime = [
( 0, b'\x30\x26\xB2\x75\x8E\x66\xCF\x11\xA6\xD9\x00\xAA\x00\x62\xCE\x6C', HC.UNDETERMINED_WM )
]
def SaveThumbnailToStreamPIL( pil_image, dimensions, f ):
def SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f ):
# when the palette is limited, the thumbnail antialias won't add new colours, so you get nearest-neighbour-like behaviour
@ -61,7 +61,7 @@ def SaveThumbnailToStreamPIL( pil_image, dimensions, f ):
pil_image = HydrusImageHandling.Dequantize( pil_image )
HydrusImageHandling.EfficientlyThumbnailPILImage( pil_image, dimensions )
HydrusImageHandling.ThumbnailPILImage( pil_image, bounding_dimensions )
if original_file_was_png or pil_image.mode == 'RGBA':
@ -72,11 +72,11 @@ def SaveThumbnailToStreamPIL( pil_image, dimensions, f ):
pil_image.save( f, 'JPEG', quality = 92 )
def GenerateThumbnail( path, mime, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS, percentage_in = 35 ):
def GenerateThumbnailBytes( path, bounding_dimensions, mime, percentage_in = 35 ):
if mime in ( HC.IMAGE_JPEG, HC.IMAGE_PNG, HC.IMAGE_GIF, HC.IMAGE_WEBP, HC.IMAGE_TIFF ):
thumbnail = GenerateThumbnailFileBytesFromStaticImagePath( path, dimensions, mime )
thumbnail_bytes = GenerateThumbnailBytesFromStaticImagePath( path, bounding_dimensions, mime )
else:
@ -92,7 +92,7 @@ def GenerateThumbnail( path, mime, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS
pil_image = HydrusImageHandling.GeneratePILImage( temp_path )
SaveThumbnailToStreamPIL( pil_image, dimensions, f )
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
except:
@ -100,7 +100,7 @@ def GenerateThumbnail( path, mime, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS
pil_image = HydrusImageHandling.GeneratePILImage( flash_default_path )
SaveThumbnailToStreamPIL( pil_image, dimensions, f )
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
finally:
@ -113,7 +113,7 @@ def GenerateThumbnail( path, mime, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS
( size, mime, width, height, duration, num_frames, num_words ) = GetFileInfo( path )
cropped_dimensions = HydrusImageHandling.GetThumbnailResolution( ( width, height ), dimensions )
cropped_dimensions = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions )
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, cropped_dimensions )
@ -132,7 +132,7 @@ def GenerateThumbnail( path, mime, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS
pil_image = HydrusImageHandling.GeneratePILImageFromNumpyImage( numpy_image )
SaveThumbnailToStreamPIL( pil_image, dimensions, f )
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
renderer.Stop()
@ -141,30 +141,34 @@ def GenerateThumbnail( path, mime, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS
f.seek( 0 )
thumbnail = f.read()
thumbnail_bytes = f.read()
f.close()
return thumbnail
return thumbnail_bytes
def GenerateThumbnailFileBytesFromStaticImagePathPIL( path, dimensions = HC.UNSCALED_THUMBNAIL_DIMENSIONS, mime = None ):
def GenerateThumbnailBytesFromPIL( pil_image, bounding_dimensions, mime ):
f = io.BytesIO()
pil_image = HydrusImageHandling.GeneratePILImage( path )
SaveThumbnailToStreamPIL( pil_image, dimensions, f )
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
f.seek( 0 )
thumbnail = f.read()
thumbnail_bytes = f.read()
f.close()
return thumbnail
return thumbnail_bytes
GenerateThumbnailFileBytesFromStaticImagePath = GenerateThumbnailFileBytesFromStaticImagePathPIL
def GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime ):
pil_image = HydrusImageHandling.GeneratePILImage( path )
return GenerateThumbnailBytesFromPIL( pil_image, bounding_dimensions, mime )
GenerateThumbnailBytesFromStaticImagePath = GenerateThumbnailBytesFromStaticImagePathPIL
def GetExtraHashesFromPath( path ):

2
include/HydrusGlobals.py
View File

@ -41,8 +41,6 @@ shutdown_complete = False
restart = False
emergency_exit = False
thumbnail_experiment_mode = False
twisted_is_broke = False
do_not_catch_char_hook = False

46
include/HydrusImageHandling.py
View File

@ -83,35 +83,6 @@ def Dequantize( pil_image ):
return pil_image
def EfficientlyResizePILImage( pil_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_x, im_y ) = pil_image.size
if target_x >= im_x and target_y >= im_y: return pil_image
#if pil_image.mode == 'RGB': # low quality resize screws up alpha channel!
#
# if im_x > 2 * target_x and im_y > 2 * target_y: pil_image.thumbnail( ( 2 * target_x, 2 * target_y ), PILImage.NEAREST )
#
return pil_image.resize( ( target_x, target_y ), PILImage.ANTIALIAS )
def EfficientlyThumbnailPILImage( pil_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_x, im_y ) = pil_image.size
#if pil_image.mode == 'RGB': # low quality resize screws up alpha channel!
#
# if im_x > 2 * target_x or im_y > 2 * target_y: pil_image.thumbnail( ( 2 * target_x, 2 * target_y ), PILImage.NEAREST )
#
if im_x > target_x or im_y > target_y:
pil_image.thumbnail( ( target_x, target_y ), PILImage.ANTIALIAS )
def GeneratePILImage( path ):
fp = open( path, 'rb' )
@ -386,3 +357,20 @@ def IsDecompressionBomb( path ):
return False
def ResizePILImage( pil_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_x, im_y ) = pil_image.size
return pil_image.resize( ( target_x, target_y ), PILImage.ANTIALIAS )
def ThumbnailPILImage( pil_image, bounding_dimensions ):
( target_x, target_y ) = bounding_dimensions
( im_x, im_y ) = pil_image.size
if im_x > target_x or im_y > target_y:
pil_image.thumbnail( ( target_x, target_y ), PILImage.ANTIALIAS )

6
include/HydrusServerResources.py
View File

@ -251,7 +251,9 @@ def ParseFileArguments( path, decompression_bombs_ok = False ):
try:
thumbnail = HydrusFileHandling.GenerateThumbnail( path, mime )
bounding_dimensions = HC.SERVER_THUMBNAIL_DIMENSIONS
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( path, bounding_dimensions, mime )
except Exception as e:
@ -260,7 +262,7 @@ def ParseFileArguments( path, decompression_bombs_ok = False ):
raise HydrusExceptions.BadRequestException( 'Could not generate thumbnail from that file:' + os.linesep + tb )
args[ 'thumbnail' ] = thumbnail
args[ 'thumbnail' ] = thumbnail_bytes
return args

6
include/HydrusVideoHandling.py
View File

@ -101,13 +101,15 @@ def GetFFMPEGInfoLines( path, count_frames_manually = False ):
if count_frames_manually:
# added -an here to remove audio component, which was sometimes causing convert fails on single-frame music webms
if HC.PLATFORM_WINDOWS:
cmd += [ "-f", "null", "NUL" ]
cmd += [ "-an", "-f", "null", "NUL" ]
else:
cmd += [ "-f", "null", "/dev/null" ]
cmd += [ "-an", "-f", "null", "/dev/null" ]

4
include/ServerDB.py
View File

@ -157,11 +157,11 @@ class DB( HydrusDB.HydrusDB ):
thumbnail_dest_path = ServerFiles.GetExpectedThumbnailPath( hash )
thumbnail = file_dict[ 'thumbnail' ]
thumbnail_bytes = file_dict[ 'thumbnail' ]
with open( thumbnail_dest_path, 'wb' ) as f:
f.write( thumbnail )
f.write( thumbnail_bytes )

2
include/TestClientAPI.py
View File

@ -1212,7 +1212,7 @@ class TestClientAPI( unittest.TestCase ):
path = os.path.join( HC.STATIC_DIR, 'hydrus_small.png' )
thumb_path = HG.test_controller.client_files_manager._GenerateExpectedFullSizeThumbnailPath( hash )
thumb_path = HG.test_controller.client_files_manager._GenerateExpectedThumbnailPath( hash )
shutil.copy2( path, thumb_path )

4
include/TestController.py
View File

@ -194,7 +194,7 @@ class Controller( object ):
self._pubsub = HydrusPubSub.HydrusPubSub( self )
self.new_options = ClientOptions.ClientOptions( self.db_dir )
self.new_options = ClientOptions.ClientOptions()
HC.options = ClientDefaults.GetClientDefaultOptions()
@ -234,7 +234,7 @@ class Controller( object ):
for prefix in HydrusData.IterateHexPrefixes():
for c in ( 'f', 't', 'r' ):
for c in ( 'f', 't' ):
client_files_locations[ c + prefix ] = client_files_default

2
include/TestDB.py
View File

@ -836,7 +836,7 @@ class TestClientDB( unittest.TestCase ):
for prefix in HydrusData.IterateHexPrefixes():
for c in ( 'f', 't', 'r' ):
for c in ( 'f', 't' ):
dir = os.path.join( client_files_default, c + prefix )