RepoMirrors
/
hydrus
mirror of https://github.com/hydrusnetwork/hydrus.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Version 351

This commit is contained in:
Hydrus Network Developer 2019-05-08 16:06:42 -05:00
parent 058559e806
commit 35ad43c717
42 changed files with 1979 additions and 1566 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
help/changelog.html
View File

@ -8,6 +8,30 @@
<div class="content">
<h3>changelog</h3>
<ul>
<li><h3>version 351</h3></li>
<ul>
<li>wrote a new (always on top!) hover window for the duplicate filter that sits on the middle-right. the duplicate cog button and action buttons are moved to this new window, as are the file comparison statements</li>
<li>the duplicate file comparison statements now state the relevant actual metadata along with better '>>'-style operators to highlight differences and green/red/blue colouring based on given score. it is now much easier to see and action clearly better files at-a-glance</li>
<li>improved some hover window focus display calculations to play with the new always-on-top tech</li>
<li>both the 'show some random dupes' button and finding dupe pairs for the filter should be a bit faster for very large search domains. the basic file search and indexing still has to run, but the second sampling step in both cases will bail out earlier once it has a decent result</li>
<li>core image handling functions now uniformly use OpenCV (faster, more accurate) by default, falling to PIL/Pillow on errors. image importing in the client and server should be a bit faster, and some unusual image rotations should now be read correctly</li>
<li>the server now supports OpenCV for image operations, it _should_ also still work with only PIL/Pillow, if you are running from source and cannot get it</li>
<li>unified all thumbnail generation code and insulated it from suprises due to unexpectedly-sized source files, fixing a potential client-level thumbnail generation looping bug</li>
<li>gave all image processing a refactor and general cleanup pass, deleted a bunch of old code</li>
<li>wrote a new 'local tag cache' for the db that will speed up tag definition lookups for all local files. this should speed up a variety of tag and file result fetching, particularly right after client boot. it will take a minute or two on update to generate</li>
<li>sped up how fast the tag parent structure builds itself</li>
<li>the review services panel now uses nested notebooks, rather than the old badly coded listbook control. I don't really like how it looks, but the code is now saner</li>
<li>similar-files metadata generation now discards blank frames more reliably</li>
<li>subscription popups now report x/y progress in terms of the current job, discarding historical work previously done. 1001/1003 is gone, 1/3 is in</li>
<li>made the disk cache more conservative on non-pre-processing calls</li>
<li>cleaned up some file import code, moving responsibility from the file locations manager to the file import object</li>
<li>updated the ipfs service listctrl to use the new listctrl object. also cleaned up its action code to be more async and stable</li>
<li>I believe I fixed a rare vector for the 'tryendmodal' dialog bug</li>
<li>fixed a bug in presenting the available importable downloader objects in the easy drag-and-drop downloader import when the multiple downloaders dropped included objects of the same type and name--duplicate-named objects in this case will now be discarded</li>
<li>unified url_match/url_class code differences to url class everywhere</li>
<li>updated some common db list selection code to use new python string formatting</li>
<li>plenty of misc code cleanup</li>
</ul>
<li><h3>version 350</h3></li>
<ul>
<li>the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long</li>

104
include/ClientCaches.py
View File

@ -84,7 +84,10 @@ def BuildSimpleChildrenToParents( pairs ):
continue
if LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ): continue
if parent in simple_children_to_parents and LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):
continue
simple_children_to_parents[ child ].add( parent )
@ -185,13 +188,16 @@ def LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):
potential_loop_paths = { parent }
while len( potential_loop_paths.intersection( list(simple_children_to_parents.keys()) ) ) > 0:
while True:
new_potential_loop_paths = set()
for potential_loop_path in potential_loop_paths.intersection( list(simple_children_to_parents.keys()) ):
for potential_loop_path in potential_loop_paths:
new_potential_loop_paths.update( simple_children_to_parents[ potential_loop_path ] )
if potential_loop_path in simple_children_to_parents:
new_potential_loop_paths.update( simple_children_to_parents[ potential_loop_path ] )
potential_loop_paths = new_potential_loop_paths
@ -200,10 +206,12 @@ def LoopInSimpleChildrenToParents( simple_children_to_parents, child, parent ):
return True
elif len( potential_loop_paths ) == 0:
return False
return False
class BitmapManager( object ):
MAX_MEMORY_ALLOWANCE = 512 * 1024 * 1024
@ -604,11 +612,13 @@ class ClientFilesManager( object ):
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions )
percentage_in = self._controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
try:
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, bounding_dimensions, mime, width, height, duration, num_frames, percentage_in = percentage_in )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( file_path, target_resolution, mime, duration, num_frames, percentage_in = percentage_in )
except Exception as e:
@ -956,6 +966,19 @@ class ClientFilesManager( object ):
def AddFile( self, hash, mime, source_path, thumbnail_bytes = None ):
with self._rwlock.write:
self._AddFile( hash, mime, source_path )
if thumbnail_bytes is not None:
self._AddThumbnailFromBytes( hash, thumbnail_bytes )
def AddThumbnailFromBytes( self, hash, thumbnail_bytes ):
with self._rwlock.write:
@ -1226,47 +1249,6 @@ class ClientFilesManager( object ):
return self._missing_locations
def ImportFile( self, file_import_job ):
if HG.file_report_mode:
HydrusData.ShowText( 'New file import job!' )
( pre_import_status, hash, note ) = file_import_job.GenerateHashAndStatus()
if file_import_job.IsNewToDB():
file_import_job.GenerateInfo()
file_import_job.CheckIsGoodToImport()
( temp_path, thumbnail ) = file_import_job.GetTempPathAndThumbnail()
mime = file_import_job.GetMime()
with self._rwlock.write:
self._AddFile( hash, mime, temp_path )
if thumbnail is not None:
self._AddThumbnailFromBytes( hash, thumbnail )
( import_status, note ) = self._controller.WriteSynchronous( 'import_file', file_import_job )
else:
import_status = pre_import_status
file_import_job.PubsubContentUpdates()
return ( import_status, hash, note )
def LocklessChangeFileExt( self, hash, old_mime, mime ):
old_path = self._GenerateExpectedFilePath( hash, old_mime )
@ -1511,9 +1493,9 @@ class ClientFilesManager( object ):
path = self._GenerateExpectedThumbnailPath( hash )
numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )
( current_width, current_height ) = ClientImageHandling.GetNumPyImageResolution( numpy_image )
( current_width, current_height ) = HydrusImageHandling.GetResolutionNumPy( numpy_image )
bounding_dimensions = self._controller.options[ 'thumbnail_dimensions' ]
@ -3262,7 +3244,7 @@ class ThumbnailCache( object ):
try:
numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )
except Exception as e:
@ -3282,7 +3264,7 @@ class ThumbnailCache( object ):
try:
numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )
except Exception as e:
@ -3294,7 +3276,7 @@ class ThumbnailCache( object ):
( current_width, current_height ) = ClientImageHandling.GetNumPyImageResolution( numpy_image )
( current_width, current_height ) = HydrusImageHandling.GetResolutionNumPy( numpy_image )
( expected_width, expected_height ) = HydrusImageHandling.GetThumbnailResolution( ( media_width, media_height ), bounding_dimensions )
@ -3319,7 +3301,7 @@ class ThumbnailCache( object ):
# this is _resize_, not _thumbnail_, because we already know the dimensions we want
# and in some edge cases, doing getthumbresolution on existing thumb dimensions results in float/int conversion imprecision and you get 90px/91px regen cycles that never get fixed
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, ( expected_width, expected_height ) )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, ( expected_width, expected_height ) )
if locations_manager.IsLocal():
@ -3334,11 +3316,11 @@ class ThumbnailCache( object ):
try:
thumbnail_bytes = ClientImageHandling.GenerateBytesFromCV( numpy_image, mime )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesNumPy( numpy_image, mime )
except HydrusExceptions.CantRenderWithCVException:
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesFromStaticImagePath( path, ( expected_width, expected_height ), mime )
except:
@ -3383,7 +3365,7 @@ class ThumbnailCache( object ):
else:
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, ( expected_width, expected_height ) )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, ( expected_width, expected_height ) )
if locations_manager.IsLocal():
@ -3555,9 +3537,13 @@ class ThumbnailCache( object ):
path = os.path.join( HC.STATIC_DIR, name + '.png' )
numpy_image = ClientImageHandling.GenerateNumpyImage( path, HC.IMAGE_PNG )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, HC.IMAGE_PNG )
numpy_image = ClientImageHandling.ThumbnailNumpyImage( numpy_image, bounding_dimensions )
numpy_image_resolution = HydrusImageHandling.GetResolutionNumPy( numpy_image )
target_resolution = HydrusImageHandling.GetThumbnailResolution( numpy_image_resolution, bounding_dimensions )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution )
hydrus_bitmap = ClientRendering.GenerateHydrusBitmapFromNumPyImage( numpy_image )

403
include/ClientDB.py
View File

File diff suppressed because it is too large Load Diff

2
include/ClientDaemons.py
View File

@ -154,7 +154,7 @@ def DAEMONDownloadFiles( controller ):
file_import_job = ClientImportFileSeeds.FileImportJob( temp_path, file_import_options )
client_files_manager.ImportFile( file_import_job )
file_import_job.DoWork()
successful_hashes.add( hash )

8
include/ClientDefaults.py
View File

@ -441,13 +441,13 @@ def GetDefaultSimpleDownloaderFormulae():
return GetDefaultObjectsFromPNGs( dir_path, ( ClientParsing.SimpleDownloaderParsingFormula, ) )
def GetDefaultURLMatches():
def GetDefaultURLClasses():
dir_path = os.path.join( HC.STATIC_DIR, 'default', 'url_classes' )
from . import ClientNetworkingDomain
return GetDefaultObjectsFromPNGs( dir_path, ( ClientNetworkingDomain.URLMatch, ) )
return GetDefaultObjectsFromPNGs( dir_path, ( ClientNetworkingDomain.URLClass, ) )
def GetDefaultObjectsFromPNGs( dir_path, allowed_object_types ):
@ -619,7 +619,7 @@ def SetDefaultDomainManagerData( domain_manager ):
#
domain_manager.SetURLMatches( GetDefaultURLMatches() )
domain_manager.SetURLClasses( GetDefaultURLClasses() )
#
@ -627,7 +627,7 @@ def SetDefaultDomainManagerData( domain_manager ):
#
domain_manager.TryToLinkURLMatchesAndParsers()
domain_manager.TryToLinkURLClassesAndParsers()
def SetDefaultLoginManagerScripts( login_manager ):

56
include/ClientGUI.py
View File

@ -2014,14 +2014,14 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
submenu = wx.Menu()
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage gallery url generators', 'Manage the client\'s GUGs, which convert search terms into URLs.', self._ManageGUGs )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage url classes', 'Configure which URLs the client can recognise.', self._ManageURLMatches )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage url classes', 'Configure which URLs the client can recognise.', self._ManageURLClasses )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage parsers', 'Manage the client\'s parsers, which convert URL content into hydrus metadata.', self._ManageParsers )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage login scripts', 'Manage the client\'s login scripts, which define how to log in to different sites.', self._ManageLoginScripts )
ClientGUIMenus.AppendSeparator( submenu )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage url class links', 'Configure how URLs present across the client.', self._ManageURLMatchLinks )
ClientGUIMenus.AppendMenuItem( self, submenu, 'manage url class links', 'Configure how URLs present across the client.', self._ManageURLClassLinks )
ClientGUIMenus.AppendMenuItem( self, submenu, 'export downloaders', 'Export downloader components to easy-import pngs.', self._ExportDownloader )
ClientGUIMenus.AppendSeparator( submenu )
@ -2485,7 +2485,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
message += os.linesep * 2
message += 'Since this URL cannot be parsed, a downloader cannot be created for it! Please check your url class links under the \'networking\' menu.'
raise HydrusExceptions.URLMatchException( message )
raise HydrusExceptions.URLClassException( message )
if url_type in ( HC.URL_TYPE_UNKNOWN, HC.URL_TYPE_FILE, HC.URL_TYPE_POST, HC.URL_TYPE_GALLERY ):
@ -2616,22 +2616,22 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
domain_manager = self._controller.network_engine.domain_manager
( file_post_default_tag_import_options, watchable_default_tag_import_options, url_match_keys_to_tag_import_options ) = domain_manager.GetDefaultTagImportOptions()
( file_post_default_tag_import_options, watchable_default_tag_import_options, url_class_keys_to_tag_import_options ) = domain_manager.GetDefaultTagImportOptions()
url_matches = domain_manager.GetURLMatches()
url_classes = domain_manager.GetURLClasses()
parsers = domain_manager.GetParsers()
url_match_keys_to_parser_keys = domain_manager.GetURLMatchKeysToParserKeys()
url_class_keys_to_parser_keys = domain_manager.GetURLClassKeysToParserKeys()
panel = ClientGUIScrolledPanelsEdit.EditDefaultTagImportOptionsPanel( dlg, url_matches, parsers, url_match_keys_to_parser_keys, file_post_default_tag_import_options, watchable_default_tag_import_options, url_match_keys_to_tag_import_options )
panel = ClientGUIScrolledPanelsEdit.EditDefaultTagImportOptionsPanel( dlg, url_classes, parsers, url_class_keys_to_parser_keys, file_post_default_tag_import_options, watchable_default_tag_import_options, url_class_keys_to_tag_import_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
( file_post_default_tag_import_options, watchable_default_tag_import_options, url_match_keys_to_tag_import_options ) = panel.GetValue()
( file_post_default_tag_import_options, watchable_default_tag_import_options, url_class_keys_to_tag_import_options ) = panel.GetValue()
domain_manager.SetDefaultTagImportOptions( file_post_default_tag_import_options, watchable_default_tag_import_options, url_match_keys_to_tag_import_options )
domain_manager.SetDefaultTagImportOptions( file_post_default_tag_import_options, watchable_default_tag_import_options, url_class_keys_to_tag_import_options )
@ -2648,20 +2648,20 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
gug_keys_to_display = domain_manager.GetGUGKeysToDisplay()
url_matches = domain_manager.GetURLMatches()
url_classes = domain_manager.GetURLClasses()
url_match_keys_to_display = domain_manager.GetURLMatchKeysToDisplay()
url_class_keys_to_display = domain_manager.GetURLClassKeysToDisplay()
panel = ClientGUIScrolledPanelsEdit.EditDownloaderDisplayPanel( dlg, self._controller.network_engine, gugs, gug_keys_to_display, url_matches, url_match_keys_to_display )
panel = ClientGUIScrolledPanelsEdit.EditDownloaderDisplayPanel( dlg, self._controller.network_engine, gugs, gug_keys_to_display, url_classes, url_class_keys_to_display )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
( gug_keys_to_display, url_match_keys_to_display ) = panel.GetValue()
( gug_keys_to_display, url_class_keys_to_display ) = panel.GetValue()
domain_manager.SetGUGKeysToDisplay( gug_keys_to_display )
domain_manager.SetURLMatchKeysToDisplay( url_match_keys_to_display )
domain_manager.SetURLClassKeysToDisplay( url_class_keys_to_display )
@ -3213,7 +3213,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
def _ManageURLMatches( self ):
def _ManageURLClasses( self ):
title = 'manage url classes'
@ -3221,22 +3221,22 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
domain_manager = self._controller.network_engine.domain_manager
url_matches = domain_manager.GetURLMatches()
url_classes = domain_manager.GetURLClasses()
panel = ClientGUIScrolledPanelsEdit.EditURLMatchesPanel( dlg, url_matches )
panel = ClientGUIScrolledPanelsEdit.EditURLClassesPanel( dlg, url_classes )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
url_matches = panel.GetValue()
url_classes = panel.GetValue()
domain_manager.SetURLMatches( url_matches )
domain_manager.SetURLClasses( url_classes )
def _ManageURLMatchLinks( self ):
def _ManageURLClassLinks( self ):
title = 'manage url class links'
@ -3244,20 +3244,20 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
domain_manager = self._controller.network_engine.domain_manager
url_matches = domain_manager.GetURLMatches()
url_classes = domain_manager.GetURLClasses()
parsers = domain_manager.GetParsers()
url_match_keys_to_parser_keys = domain_manager.GetURLMatchKeysToParserKeys()
url_class_keys_to_parser_keys = domain_manager.GetURLClassKeysToParserKeys()
panel = ClientGUIScrolledPanelsEdit.EditURLMatchLinksPanel( dlg, self._controller.network_engine, url_matches, parsers, url_match_keys_to_parser_keys )
panel = ClientGUIScrolledPanelsEdit.EditURLClassLinksPanel( dlg, self._controller.network_engine, url_classes, parsers, url_class_keys_to_parser_keys )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
url_match_keys_to_parser_keys = panel.GetValue()
url_class_keys_to_parser_keys = panel.GetValue()
domain_manager.SetURLMatchKeysToParserKeys( url_match_keys_to_parser_keys )
domain_manager.SetURLClassKeysToParserKeys( url_class_keys_to_parser_keys )
@ -3681,7 +3681,7 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
for i in range( 16 ):
t += 0.5
t += 1.0
for j in range( i + 1 ):
@ -3692,12 +3692,12 @@ class FrameGUI( ClientGUITopLevelWindows.FrameThatResizes ):
HG.client_controller.CallLaterWXSafe( self, t, uias.Char, wx.WXK_RETURN )
t += 0.5
t += 1.0
HG.client_controller.CallLaterWXSafe( self, t, uias.Char, wx.WXK_RETURN )
t += 0.5
t += 1.0
HG.client_controller.CallLaterWXSafe( self, t, uias.Char, wx.WXK_DOWN )

31
include/ClientGUICanvas.py
View File

@ -3091,6 +3091,8 @@ class CanvasFilterDuplicates( CanvasWithHovers ):
CanvasWithHovers.__init__( self, parent )
self._hover_duplicates = ClientGUIHoverFrames.FullscreenHoverFrameRightDuplicates( self, self, self._canvas_key )
self._file_search_context = file_search_context
self._both_files_match = both_files_match
@ -3295,33 +3297,6 @@ class CanvasFilterDuplicates( CanvasWithHovers ):
def _DrawAdditionalTopMiddleInfo( self, dc, current_y ):
if self._current_media is not None:
shown_media = self._current_media
comparison_media = self._media_list.GetNext( shown_media )
if shown_media != comparison_media:
( statements, score ) = ClientMedia.GetDuplicateComparisonStatements( shown_media, comparison_media )
( client_width, client_height ) = self.GetClientSize()
for statement in statements:
( width, height ) = dc.GetTextExtent( statement )
dc.DrawText( statement, ( client_width - width ) // 2, current_y )
current_y += height + 3
return current_y
def _DrawBackgroundDetails( self, dc ):
if self._currently_fetching_pairs:
@ -3599,7 +3574,7 @@ class CanvasFilterDuplicates( CanvasWithHovers ):
first_media = ClientMedia.MediaSingleton( first_media_result )
second_media = ClientMedia.MediaSingleton( second_media_result )
( statements, score ) = ClientMedia.GetDuplicateComparisonStatements( first_media, second_media )
score = ClientMedia.GetDuplicateComparisonScore( first_media, second_media )
if score > 0:

28
include/ClientGUICommon.py
View File

@ -750,9 +750,18 @@ class BetterChoice( wx.Choice ):
selection = self.GetSelection()
if selection != wx.NOT_FOUND: return self.GetClientData( selection )
elif self.GetCount() > 0: return self.GetClientData( 0 )
else: return None
if selection != wx.NOT_FOUND:
return self.GetClientData( selection )
elif self.GetCount() > 0:
return self.GetClientData( 0 )
else:
return None
def SelectClientData( self, client_data ):
@ -864,6 +873,19 @@ class BetterNotebook( wx.Notebook ):
self._ShiftSelection( -1 )
def SelectPage( self, page ):
for i in range( self.GetPageCount() ):
if self.GetPage( i ) == page:
self.SetSelection( i )
return
def SelectRight( self ):
self._ShiftSelection( 1 )

6
include/ClientGUIDialogs.py
View File

@ -1339,7 +1339,7 @@ class DialogInputTags( Dialog ):
self._tag_box = ClientGUIACDropdown.AutoCompleteDropdownTagsWrite( self, self.EnterTags, expand_parents, CC.LOCAL_FILE_SERVICE_KEY, service_key, null_entry_callable = self.OK )
self._ok = wx.Button( self, id= wx.ID_OK, label = 'OK' )
self._ok = wx.Button( self, id = wx.ID_OK, label = 'OK' )
self._ok.SetForegroundColour( ( 0, 128, 0 ) )
self._cancel = wx.Button( self, id = wx.ID_CANCEL, label = 'Cancel' )
@ -2002,6 +2002,8 @@ class DialogYesNo( Dialog ):
self._no.SetForegroundColour( ( 128, 0, 0 ) )
self._no.SetLabelText( no_label )
self.SetEscapeId( wx.ID_NO )
#
hbox = wx.BoxSizer( wx.HORIZONTAL )
@ -2056,6 +2058,8 @@ class DialogYesYesNo( Dialog ):
self._no.SetForegroundColour( ( 128, 0, 0 ) )
self._no.SetLabelText( no_label )
self.SetEscapeId( wx.ID_NO )
#
hbox = wx.BoxSizer( wx.HORIZONTAL )

387
include/ClientGUIHoverFrames.py
View File

@ -36,6 +36,8 @@ class FullscreenHoverFrame( wx.Frame ):
self._canvas_key = canvas_key
self._current_media = None
self._always_on_top = False
self._last_ideal_position = None
self.SetBackgroundColour( wx.SystemSettings.GetColour( wx.SYS_COLOUR_FRAMEBK ) )
@ -66,6 +68,11 @@ class FullscreenHoverFrame( wx.Frame ):
self.SetSize( my_ideal_size )
if HC.PLATFORM_OSX and self.GetPosition() != my_ideal_position and self._always_on_top:
self.Raise()
self.SetPosition( my_ideal_position )
@ -82,7 +89,7 @@ class FullscreenHoverFrame( wx.Frame ):
new_options = HG.client_controller.new_options
if new_options.GetBoolean( 'always_show_hover_windows' ):
if self._always_on_top or new_options.GetBoolean( 'always_show_hover_windows' ):
self._SizeAndPosition()
@ -103,7 +110,7 @@ class FullscreenHoverFrame( wx.Frame ):
if self._current_media is None or not self.GetParent().IsShown():
if self._current_media is None or not self._my_canvas.IsShown():
if self.IsShown():
@ -164,13 +171,15 @@ class FullscreenHoverFrame( wx.Frame ):
mime = self._current_media.GetMime()
mouse_is_over_interactable_media = mime == HC.APPLICATION_FLASH and self.GetParent().MouseIsOverMedia()
mouse_is_over_interactable_media = mime == HC.APPLICATION_FLASH and self._my_canvas.MouseIsOverMedia()
mouse_is_near_animation_bar = self.GetParent().MouseIsNearAnimationBar()
mouse_is_near_animation_bar = self._my_canvas.MouseIsNearAnimationBar()
mouse_is_over_something_important = mouse_is_over_interactable_media or mouse_is_near_animation_bar
focus_is_good = ClientGUICommon.TLPHasFocus( self ) or ClientGUICommon.TLPHasFocus( self.GetParent() )
current_focus = ClientGUICommon.GetFocusTLP()
focus_is_good = ClientGUICommon.IsWXAncestor( current_focus, self._my_canvas.GetTopLevelParent(), through_tlws = True )
ready_to_show = in_position and not mouse_is_over_something_important and focus_is_good and not dialog_open and not menu_open
ready_to_hide = not menu_open and ( not in_position or dialog_open or not focus_is_good )
@ -221,6 +230,263 @@ class FullscreenHoverFrame( wx.Frame ):
class FullscreenHoverFrameRightDuplicates( FullscreenHoverFrame ):
def __init__( self, parent, my_canvas, canvas_key ):
FullscreenHoverFrame.__init__( self, parent, my_canvas, canvas_key )
self._always_on_top = True
self._current_index_string = ''
self._comparison_media = None
menu_items = []
menu_items.append( ( 'normal', 'edit duplicate action options for \'this is better\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_BETTER ) ) )
menu_items.append( ( 'normal', 'edit duplicate action options for \'same quality\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_SAME_QUALITY ) ) )
menu_items.append( ( 'normal', 'edit duplicate action options for \'alternates\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_ALTERNATE ) ) )
menu_items.append( ( 'normal', 'edit duplicate action options for \'not duplicates\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_NOT_DUPLICATE ) ) )
menu_items.append( ( 'separator', None, None, None ) )
menu_items.append( ( 'normal', 'edit background lighten/darken switch intensity', 'edit how much the background will brighten or darken as you switch between the pair', self._EditBackgroundSwitchIntensity ) )
self._back_a_pair = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.first, HG.client_controller.pub, 'canvas_application_command', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_back' ), self._canvas_key )
self._back_a_pair.SetToolTip( 'go back a pair' )
self._previous_button = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.previous, HG.client_controller.pub, 'canvas_application_command', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'view_previous' ), self._canvas_key )
self._previous_button.SetToolTip( 'previous' )
self._index_text = ClientGUICommon.BetterStaticText( self, 'index' )
self._next_button = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.next_bmp, HG.client_controller.pub, 'canvas_application_command', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'view_next' ), self._canvas_key )
self._next_button.SetToolTip( 'next' )
self._skip_a_pair = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.last, HG.client_controller.pub, 'canvas_application_command', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_skip' ), self._canvas_key )
self._skip_a_pair.SetToolTip( 'show a different pair' )
self._cog_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.cog, menu_items )
dupe_commands = []
dupe_commands.append( ( 'this is better', 'Set that the current file you are looking at is better than the other in the pair.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_this_is_better' ) ) )
dupe_commands.append( ( 'same quality', 'Set that the two files are duplicates of very similar quality.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_exactly_the_same' ) ) )
dupe_commands.append( ( 'alternates', 'Set that the files are not duplicates, but that one is derived from the other or that they are both descendants of a common ancestor.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_alternates' ) ) )
dupe_commands.append( ( 'not duplicates', 'Set that the files are not duplicates or otherwise related--that this pair is a false-positive match.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_not_dupes' ) ) )
dupe_commands.append( ( 'custom action', 'Choose one of the other actions but customise the merge and delete options for this specific decision.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_custom_action' ) ) )
command_button_vbox = wx.BoxSizer( wx.VERTICAL )
for ( label, tooltip, command ) in dupe_commands:
command_button = ClientGUICommon.BetterButton( self, label, HG.client_controller.pub, 'canvas_application_command', command, self._canvas_key )
command_button.SetToolTip( tooltip )
command_button_vbox.Add( command_button, CC.FLAGS_EXPAND_PERPENDICULAR )
self._comparison_statements_vbox = wx.BoxSizer( wx.VERTICAL )
self._comparison_statement_names = [ 'filesize', 'resolution', 'mime', 'num_tags', 'time_imported' ]
self._comparison_statements_sts = {}
for name in self._comparison_statement_names:
panel = wx.Panel( self )
st = ClientGUICommon.BetterStaticText( panel, 'init' )
self._comparison_statements_sts[ name ] = ( panel, st )
hbox = wx.BoxSizer( wx.HORIZONTAL )
hbox.Add( ( 20, 20 ), CC.FLAGS_EXPAND_BOTH_WAYS )
hbox.Add( st, CC.FLAGS_VCENTER )
hbox.Add( ( 20, 20 ), CC.FLAGS_EXPAND_BOTH_WAYS )
panel.SetSizer( hbox )
panel.Hide()
self._comparison_statements_vbox.Add( panel, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
#
navigation_button_hbox = wx.BoxSizer( wx.HORIZONTAL )
navigation_button_hbox.Add( self._back_a_pair, CC.FLAGS_VCENTER )
navigation_button_hbox.Add( self._previous_button, CC.FLAGS_VCENTER )
navigation_button_hbox.Add( ( 20, 20 ), CC.FLAGS_EXPAND_BOTH_WAYS )
navigation_button_hbox.Add( self._index_text, CC.FLAGS_VCENTER )
navigation_button_hbox.Add( ( 20, 20 ), CC.FLAGS_EXPAND_BOTH_WAYS )
navigation_button_hbox.Add( self._next_button, CC.FLAGS_VCENTER )
navigation_button_hbox.Add( self._skip_a_pair, CC.FLAGS_VCENTER )
navigation_button_hbox.Add( self._cog_button, CC.FLAGS_VCENTER )
vbox = wx.BoxSizer( wx.VERTICAL )
vbox.Add( navigation_button_hbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
vbox.Add( command_button_vbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
vbox.Add( self._comparison_statements_vbox, CC.FLAGS_EXPAND_SIZER_PERPENDICULAR )
self.SetSizer( vbox )
HG.client_controller.sub( self, 'SetDuplicatePair', 'canvas_new_duplicate_pair' )
HG.client_controller.sub( self, 'SetIndexString', 'canvas_new_index_string' )
self.Bind( wx.EVT_MOUSEWHEEL, self.EventMouseWheel )
def _EditBackgroundSwitchIntensity( self ):
new_options = HG.client_controller.new_options
value = new_options.GetNoneableInteger( 'duplicate_background_switch_intensity' )
with ClientGUITopLevelWindows.DialogEdit( self, 'edit lighten/darken intensity' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditNoneableIntegerPanel( dlg, value, message = 'intensity: ', none_phrase = 'do not change', min = 1, max = 9 )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
new_value = panel.GetValue()
new_options.SetNoneableInteger( 'duplicate_background_switch_intensity', new_value )
def _EditMergeOptions( self, duplicate_type ):
new_options = HG.client_controller.new_options
duplicate_action_options = new_options.GetDuplicateActionOptions( duplicate_type )
with ClientGUITopLevelWindows.DialogEdit( self, 'edit duplicate merge options' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditDuplicateActionOptionsPanel( dlg, duplicate_type, duplicate_action_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
duplicate_action_options = panel.GetValue()
new_options.SetDuplicateActionOptions( duplicate_type, duplicate_action_options )
def _GetIdealSizeAndPosition( self ):
parent = self.GetParent()
( parent_width, parent_height ) = parent.GetClientSize()
( my_width, my_height ) = self.GetSize()
my_ideal_width = int( parent_width * 0.2 )
should_resize = my_ideal_width != my_width
ideal_size = ( my_ideal_width, -1 )
ideal_position = ClientGUICommon.ClientToScreen( parent, ( int( parent_width * 0.8 ), int( parent_height * 0.3 ) ) )
return ( should_resize, ideal_size, ideal_position )
def _ResetComparisonStatements( self ):
fit_needed = False
statements_and_scores = ClientMedia.GetDuplicateComparisonStatements( self._current_media, self._comparison_media )
for name in self._comparison_statement_names:
( panel, st ) = self._comparison_statements_sts[ name ]
if name in statements_and_scores:
( statement, score ) = statements_and_scores[ name ]
if not panel.IsShown():
fit_needed = True
panel.Show()
st.SetLabelText( statement )
if score > 0:
colour = ( 0, 128, 0 )
elif score < 0:
colour = ( 128, 0, 0 )
else:
colour = ( 0, 0, 128 )
st.SetForegroundColour( colour )
else:
if panel.IsShown():
fit_needed = True
panel.Hide()
if fit_needed:
self.Fit()
else:
self.Layout()
def EventMouseWheel( self, event ):
event.ResumePropagation( 1 )
event.Skip()
def SetDuplicatePair( self, canvas_key, shown_media, comparison_media ):
if canvas_key == self._canvas_key:
self._current_media = shown_media
self._comparison_media = comparison_media
self._ResetComparisonStatements()
def SetIndexString( self, canvas_key, text ):
if canvas_key == self._canvas_key:
self._current_index_string = text
self._index_text.SetLabelText( self._current_index_string )
class FullscreenHoverFrameTop( FullscreenHoverFrame ):
def __init__( self, parent, my_canvas, canvas_key ):
@ -644,90 +910,6 @@ class FullscreenHoverFrameTopNavigable( FullscreenHoverFrameTop ):
class FullscreenHoverFrameTopDuplicatesFilter( FullscreenHoverFrameTopNavigable ):
def __init__( self, parent, my_canvas, canvas_key ):
FullscreenHoverFrameTopNavigable.__init__( self, parent, my_canvas, canvas_key )
HG.client_controller.sub( self, 'SetDuplicatePair', 'canvas_new_duplicate_pair' )
def _PopulateCenterButtons( self ):
menu_items = []
menu_items.append( ( 'normal', 'edit duplicate action options for \'this is better\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_BETTER ) ) )
menu_items.append( ( 'normal', 'edit duplicate action options for \'same quality\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_SAME_QUALITY ) ) )
menu_items.append( ( 'normal', 'edit duplicate action options for \'alternates\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_ALTERNATE ) ) )
menu_items.append( ( 'normal', 'edit duplicate action options for \'not duplicates\'', 'edit what content is merged when you filter files', HydrusData.Call( self._EditMergeOptions, HC.DUPLICATE_NOT_DUPLICATE ) ) )
menu_items.append( ( 'separator', None, None, None ) )
menu_items.append( ( 'normal', 'edit background lighten/darken switch intensity', 'edit how much the background will brighten or darken as you switch between the pair', self._EditBackgroundSwitchIntensity ) )
cog_button = ClientGUICommon.MenuBitmapButton( self, CC.GlobalBMPs.cog, menu_items )
self._top_hbox.Add( cog_button, CC.FLAGS_SIZER_VCENTER )
FullscreenHoverFrameTopNavigable._PopulateCenterButtons( self )
dupe_commands = []
dupe_commands.append( ( 'this is better', 'Set that the current file you are looking at is better than the other in the pair.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_this_is_better' ) ) )
dupe_commands.append( ( 'same quality', 'Set that the two files are duplicates of very similar quality.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_exactly_the_same' ) ) )
dupe_commands.append( ( 'alternates', 'Set that the files are not duplicates, but that one is derived from the other or that they are both descendants of a common ancestor.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_alternates' ) ) )
dupe_commands.append( ( 'not duplicates', 'Set that the files are not duplicates or otherwise related--that this pair is a false-positive match.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_not_dupes' ) ) )
dupe_commands.append( ( 'custom action', 'Choose one of the other actions but customise the merge and delete options for this specific decision.', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_custom_action' ) ) )
for ( label, tooltip, command ) in dupe_commands:
command_button = ClientGUICommon.BetterButton( self, label, HG.client_controller.pub, 'canvas_application_command', command, self._canvas_key )
command_button.SetToolTip( tooltip )
self._button_hbox.Add( command_button, CC.FLAGS_VCENTER )
def _EditBackgroundSwitchIntensity( self ):
new_options = HG.client_controller.new_options
value = new_options.GetNoneableInteger( 'duplicate_background_switch_intensity' )
with ClientGUITopLevelWindows.DialogEdit( self, 'edit lighten/darken intensity' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditNoneableIntegerPanel( dlg, value, message = 'intensity: ', none_phrase = 'do not change', min = 1, max = 9 )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
new_value = panel.GetValue()
new_options.SetNoneableInteger( 'duplicate_background_switch_intensity', new_value )
def _EditMergeOptions( self, duplicate_type ):
new_options = HG.client_controller.new_options
duplicate_action_options = new_options.GetDuplicateActionOptions( duplicate_type )
with ClientGUITopLevelWindows.DialogEdit( self, 'edit duplicate merge options' ) as dlg:
panel = ClientGUIScrolledPanelsEdit.EditDuplicateActionOptionsPanel( dlg, duplicate_type, duplicate_action_options )
dlg.SetPanel( panel )
if dlg.ShowModal() == wx.ID_OK:
duplicate_action_options = panel.GetValue()
new_options.SetDuplicateActionOptions( duplicate_type, duplicate_action_options )
def _PopulateLeftButtons( self ):
self._first_button = ClientGUICommon.BetterBitmapButton( self, CC.GlobalBMPs.first, HG.client_controller.pub, 'canvas_application_command', ClientData.ApplicationCommand( CC.APPLICATION_COMMAND_TYPE_SIMPLE, 'duplicate_filter_back' ), self._canvas_key )
@ -743,33 +925,6 @@ class FullscreenHoverFrameTopDuplicatesFilter( FullscreenHoverFrameTopNavigable
self._top_hbox.Add( self._last_button, CC.FLAGS_VCENTER )
def SetDisplayMedia( self, canvas_key, media ):
if canvas_key == self._canvas_key:
if media is None:
self._additional_info_text.SetLabelText( '' )
FullscreenHoverFrameTopNavigable.SetDisplayMedia( self, canvas_key, media )
def SetDuplicatePair( self, canvas_key, shown_media, comparison_media ):
if canvas_key == self._canvas_key:
( statements, score ) = ClientMedia.GetDuplicateComparisonStatements( shown_media, comparison_media )
self._additional_info_text.SetLabelText( os.linesep.join( statements ) )
self._ResetText()
self._ResetButtons()
class FullscreenHoverFrameTopNavigableList( FullscreenHoverFrameTopNavigable ):
def _PopulateLeftButtons( self ):

70
include/ClientGUIMedia.py
View File

@ -61,7 +61,7 @@ def CopyMediaURLs( medias ):
HG.client_controller.pub( 'clipboard', 'text', urls_string )
def CopyMediaURLMatchURLs( medias, url_match ):
def CopyMediaURLClassURLs( medias, url_class ):
urls = set()
@ -71,7 +71,7 @@ def CopyMediaURLMatchURLs( medias, url_match ):
for url in media_urls:
if url_match.Matches( url ):
if url_class.Matches( url ):
urls.add( url )
@ -97,15 +97,15 @@ def DoOpenKnownURLFromShortcut( win, media ):
for url in urls:
url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( url )
url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( url )
if url_match is None:
if url_class is None:
unmatched_urls.append( url )
else:
label = url_match.GetName() + ': ' + url
label = url_class.GetName() + ': ' + url
matched_labels_and_urls.append( ( label, url ) )
@ -226,7 +226,7 @@ def OpenMediaURLs( medias ):
OpenURLs( urls )
def OpenMediaURLMatchURLs( medias, url_match ):
def OpenMediaURLClassURLs( medias, url_class ):
urls = set()
@ -236,7 +236,7 @@ def OpenMediaURLMatchURLs( medias, url_match ):
for url in media_urls:
if url_match.Matches( url ):
if url_class.Matches( url ):
urls.add( url )
@ -300,15 +300,15 @@ def AddKnownURLsViewCopyMenu( win, menu, focus_media, selected_media = None ):
for url in focus_urls:
url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( url )
url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( url )
if url_match is None:
if url_class is None:
focus_unmatched_urls.append( url )
else:
label = url_match.GetName() + ': ' + url
label = url_class.GetName() + ': ' + url
focus_matched_labels_and_urls.append( ( label, url ) )
@ -324,8 +324,8 @@ def AddKnownURLsViewCopyMenu( win, menu, focus_media, selected_media = None ):
# figure out which urls these selected files have
selected_media_url_matches = set()
multiple_or_unmatching_selection_url_matches = False
selected_media_url_classes = set()
multiple_or_unmatching_selection_url_classes = False
if selected_media is not None and len( selected_media ) > 1:
@ -348,26 +348,26 @@ def AddKnownURLsViewCopyMenu( win, menu, focus_media, selected_media = None ):
for url in media_urls:
url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( url )
url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( url )
if url_match is None:
if url_class is None:
multiple_or_unmatching_selection_url_matches = True
multiple_or_unmatching_selection_url_classes = True
else:
selected_media_url_matches.add( url_match )
selected_media_url_classes.add( url_class )
if len( selected_media_url_matches ) > 1:
if len( selected_media_url_classes ) > 1:
multiple_or_unmatching_selection_url_matches = True
multiple_or_unmatching_selection_url_classes = True
if len( focus_labels_and_urls ) > 0 or len( selected_media_url_matches ) > 0 or multiple_or_unmatching_selection_url_matches:
if len( focus_labels_and_urls ) > 0 or len( selected_media_url_classes ) > 0 or multiple_or_unmatching_selection_url_classes:
urls_menu = wx.Menu()
@ -387,16 +387,16 @@ def AddKnownURLsViewCopyMenu( win, menu, focus_media, selected_media = None ):
# copy this file's urls
there_are_focus_url_matches_to_action = len( focus_matched_labels_and_urls ) > 1
multiple_or_unmatching_focus_url_matches = len( focus_unmatched_urls ) > 0 and len( focus_labels_and_urls ) > 1 # if there are unmatched urls and more than one thing total
there_are_focus_url_classes_to_action = len( focus_matched_labels_and_urls ) > 1
multiple_or_unmatching_focus_url_classes = len( focus_unmatched_urls ) > 0 and len( focus_labels_and_urls ) > 1 # if there are unmatched urls and more than one thing total
if there_are_focus_url_matches_to_action or multiple_or_unmatching_focus_url_matches:
if there_are_focus_url_classes_to_action or multiple_or_unmatching_focus_url_classes:
ClientGUIMenus.AppendSeparator( urls_visit_menu )
ClientGUIMenus.AppendSeparator( urls_copy_menu )
if there_are_focus_url_matches_to_action:
if there_are_focus_url_classes_to_action:
urls = [ url for ( label, url ) in focus_matched_labels_and_urls ]
@ -411,7 +411,7 @@ def AddKnownURLsViewCopyMenu( win, menu, focus_media, selected_media = None ):
ClientGUIMenus.AppendMenuItem( win, urls_copy_menu, label, 'Copy these urls to your clipboard.', HG.client_controller.pub, 'clipboard', 'text', urls_string )
if multiple_or_unmatching_focus_url_matches:
if multiple_or_unmatching_focus_url_classes:
urls = [ url for ( label, url ) in focus_labels_and_urls ]
@ -428,35 +428,35 @@ def AddKnownURLsViewCopyMenu( win, menu, focus_media, selected_media = None ):
# now by url match type
there_are_selection_url_matches_to_action = len( selected_media_url_matches ) > 0
there_are_selection_url_classes_to_action = len( selected_media_url_classes ) > 0
if there_are_selection_url_matches_to_action or multiple_or_unmatching_selection_url_matches:
if there_are_selection_url_classes_to_action or multiple_or_unmatching_selection_url_classes:
ClientGUIMenus.AppendSeparator( urls_visit_menu )
ClientGUIMenus.AppendSeparator( urls_copy_menu )
if there_are_selection_url_matches_to_action:
if there_are_selection_url_classes_to_action:
selected_media_url_matches = list( selected_media_url_matches )
selected_media_url_classes = list( selected_media_url_classes )
selected_media_url_matches.sort( key = lambda url_match: url_match.GetName() )
selected_media_url_classes.sort( key = lambda url_class: url_class.GetName() )
for url_match in selected_media_url_matches:
for url_class in selected_media_url_classes:
label = 'open files\' ' + url_match.GetName() + ' urls in your web browser'
label = 'open files\' ' + url_class.GetName() + ' urls in your web browser'
ClientGUIMenus.AppendMenuItem( win, urls_visit_menu, label, 'Open this url class in your web browser for all files.', OpenMediaURLMatchURLs, selected_media, url_match )
ClientGUIMenus.AppendMenuItem( win, urls_visit_menu, label, 'Open this url class in your web browser for all files.', OpenMediaURLClassURLs, selected_media, url_class )
label = 'copy files\' ' + url_match.GetName() + ' urls'
label = 'copy files\' ' + url_class.GetName() + ' urls'
ClientGUIMenus.AppendMenuItem( win, urls_copy_menu, label, 'Copy this url class for all files.', CopyMediaURLMatchURLs, selected_media, url_match )
ClientGUIMenus.AppendMenuItem( win, urls_copy_menu, label, 'Copy this url class for all files.', CopyMediaURLClassURLs, selected_media, url_class )
# now everything
if multiple_or_unmatching_selection_url_matches:
if multiple_or_unmatching_selection_url_classes:
label = 'open all files\' urls'

137
include/ClientGUIPanels.py
View File

@ -25,6 +25,8 @@ class ReviewServicePanel( wx.Panel ):
wx.Panel.__init__( self, parent )
self.SetBackgroundColour( wx.SystemSettings.GetColour( wx.SYS_COLOUR_FRAMEBK ) )
self._service = service
service_type = self._service.GetServiceType()
@ -1189,12 +1191,18 @@ class ReviewServicePanel( wx.Panel ):
self._check_running_button = ClientGUICommon.BetterButton( self, 'check daemon', self._CheckRunning )
self._ipfs_shares = ClientGUIListCtrl.SaneListCtrl( self, 200, [ ( 'multihash', 120 ), ( 'num files', 80 ), ( 'total size', 80 ), ( 'note', -1 ) ], delete_key_callback = self._Unpin, activation_callback = self._SetNotes )
self._ipfs_shares_panel = ClientGUIListCtrl.BetterListCtrlPanel( self )
self._copy_multihash_button = ClientGUICommon.BetterButton( self, 'copy multihashes', self._CopyMultihashes )
self._show_selected_button = ClientGUICommon.BetterButton( self, 'show selected in main gui', self._ShowSelectedInNewPages )
self._set_notes_button = ClientGUICommon.BetterButton( self, 'set notes', self._SetNotes )
self._unpin_button = ClientGUICommon.BetterButton( self, 'unpin selected', self._Unpin )
columns = [ ( 'multihash', 34 ), ( 'num files', 6 ), ( 'total size', 6 ), ( 'note', -1 ) ]
self._ipfs_shares = ClientGUIListCtrl.BetterListCtrl( self._ipfs_shares_panel, 'ipfs_shares', 12, 32, columns, self._ConvertDataToListCtrlTuple, delete_key_callback = self._Unpin, activation_callback = self._SetNotes )
self._ipfs_shares_panel.SetListCtrl( self._ipfs_shares )
self._ipfs_shares_panel.AddButton( 'copy multihashes', self._CopyMultihashes, enabled_only_on_selection = True )
self._ipfs_shares_panel.AddButton( 'show selected in main gui', self._ShowSelectedInNewPages, enabled_only_on_selection = True )
self._ipfs_shares_panel.AddButton( 'set notes', self._SetNotes, enabled_only_on_selection = True )
self._ipfs_shares_panel.AddButton( 'unpin selected', self._Unpin, enabled_only_on_selection = True )
#
@ -1202,16 +1210,8 @@ class ReviewServicePanel( wx.Panel ):
#
button_box = wx.BoxSizer( wx.HORIZONTAL )
button_box.Add( self._copy_multihash_button, CC.FLAGS_VCENTER )
button_box.Add( self._show_selected_button, CC.FLAGS_VCENTER )
button_box.Add( self._set_notes_button, CC.FLAGS_VCENTER )
button_box.Add( self._unpin_button, CC.FLAGS_VCENTER )
self.Add( self._check_running_button, CC.FLAGS_LONE_BUTTON )
self.Add( self._ipfs_shares, CC.FLAGS_EXPAND_BOTH_WAYS )
self.Add( button_box, CC.FLAGS_BUTTON_SIZER )
self.Add( self._ipfs_shares_panel, CC.FLAGS_EXPAND_BOTH_WAYS )
HG.client_controller.sub( self, 'ServiceUpdated', 'service_updated' )
@ -1253,13 +1253,28 @@ class ReviewServicePanel( wx.Panel ):
HG.client_controller.CallToThread( do_it )
def _ConvertDataToListCtrlTuple( self, data ):
( multihash, num_files, total_size, note ) = data
pretty_multihash = multihash
pretty_num_files = HydrusData.ToHumanInt( num_files )
pretty_total_size = HydrusData.ToHumanBytes( total_size )
pretty_note = note
display_tuple = ( pretty_multihash, pretty_num_files, pretty_total_size, pretty_note )
sort_tuple = ( multihash, num_files, total_size, note )
return ( display_tuple, sort_tuple )
def _CopyMultihashes( self ):
multihashes = [ multihash for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetSelectedClientData() ]
multihashes = [ multihash for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetData( only_selected = True ) ]
if len( multihashes ) == 0:
multihashes = [ multihash for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetClientData() ]
multihashes = [ multihash for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetData() ]
if len( multihashes ) > 0:
@ -1272,18 +1287,6 @@ class ReviewServicePanel( wx.Panel ):
def _GetDisplayTuple( self, sort_tuple ):
( multihash, num_files, total_size, note ) = sort_tuple
pretty_multihash = multihash
pretty_num_files = HydrusData.ToHumanInt( num_files )
pretty_total_size = HydrusData.ToHumanBytes( total_size )
pretty_note = note
return ( pretty_multihash, pretty_num_files, pretty_total_size, pretty_note )
def _Refresh( self ):
if not self:
@ -1296,74 +1299,89 @@ class ReviewServicePanel( wx.Panel ):
def _SetNotes( self ):
for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetSelectedClientData():
datas = self._ipfs_shares.GetData( only_selected = True )
if len( datas ) > 0:
with ClientGUIDialogs.DialogTextEntry( self, 'Set a note for ' + multihash + '.' ) as dlg:
with ClientGUIDialogs.DialogTextEntry( self, 'Set a note for these shares.' ) as dlg:
if dlg.ShowModal() == wx.ID_OK:
hashes = HG.client_controller.Read( 'service_directory', self._service.GetServiceKey(), multihash )
note = dlg.GetValue()
content_update_row = ( hashes, multihash, note )
content_updates = []
content_updates = [ HydrusData.ContentUpdate( HC.CONTENT_TYPE_DIRECTORIES, HC.CONTENT_UPDATE_ADD, content_update_row ) ]
for ( multihash, num_files, total_size, old_note ) in datas:
hashes = HG.client_controller.Read( 'service_directory', self._service.GetServiceKey(), multihash )
content_update_row = ( hashes, multihash, note )
content_updates.append( HydrusData.ContentUpdate( HC.CONTENT_TYPE_DIRECTORIES, HC.CONTENT_UPDATE_ADD, content_update_row ) )
HG.client_controller.Write( 'content_updates', { self._service.GetServiceKey() : content_updates } )
else:
break
self._my_updater.Update()
self._my_updater.Update()
def _ShowSelectedInNewPages( self ):
def do_it( shares ):
def wx_done():
self._ipfs_shares_panel.Enable()
def do_it( service_key, pages_of_hashes_to_show ):
try:
for ( multihash, num_files, total_size, note ) in shares:
hashes = HG.client_controller.Read( 'service_directory', self._service.GetServiceKey(), multihash )
hashes = HG.client_controller.Read( 'service_directory', service_key, multihash )
HG.client_controller.pub( 'new_page_query', CC.LOCAL_FILE_SERVICE_KEY, initial_hashes = hashes, page_name = 'ipfs directory' )
time.sleep( 0.5 )
finally:
wx.CallAfter( self._ipfs_shares.Enable )
wx.CallAfter( wx_done )
shares = self._ipfs_shares.GetSelectedClientData()
shares = self._ipfs_shares.GetData( only_selected = True )
self._ipfs_shares.Disable()
self._ipfs_shares_panel.Disable()
HG.client_controller.CallToThread( do_it, shares )
HG.client_controller.CallToThread( do_it, self._service.GetServiceKey(), shares )
def _Unpin( self ):
def do_it( multihashes ):
def wx_done():
self._ipfs_shares_panel.Enable()
self._my_updater.Update()
def do_it( service, multihashes ):
try:
for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetSelectedClientData():
for multihash in multihashes:
self._service.UnpinDirectory( multihash )
service.UnpinDirectory( multihash )
self._ipfs_shares.RemoveAllSelected()
finally:
wx.CallAfter( self._ipfs_shares.Enable )
wx.CallAfter( wx_done )
@ -1371,11 +1389,11 @@ class ReviewServicePanel( wx.Panel ):
if dlg.ShowModal() == wx.ID_YES:
multihashes = [ multihash for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetSelectedClientData() ]
multihashes = [ multihash for ( multihash, num_files, total_size, note ) in self._ipfs_shares.GetData( only_selected = True ) ]
self._ipfs_shares.Disable()
self._ipfs_shares_panel.Disable()
HG.client_controller.CallToThread( do_it, multihashes )
HG.client_controller.CallToThread( do_it, self._service, multihashes )
@ -1399,16 +1417,9 @@ class ReviewServicePanel( wx.Panel ):
return
self._ipfs_shares.DeleteAllItems()
# list of ( multihash, num_files, total_size, note )
for ( multihash, num_files, total_size, note ) in ipfs_shares:
sort_tuple = ( multihash, num_files, total_size, note )
display_tuple = self._GetDisplayTuple( sort_tuple )
self._ipfs_shares.Append( display_tuple, sort_tuple )
self._ipfs_shares.SetData( ipfs_shares )
ipfs_shares = HG.client_controller.Read( 'service_directories', service.GetServiceKey() )

66
include/ClientGUIParsing.py
View File

@ -65,7 +65,7 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
listctrl_panel.SetListCtrl( self._listctrl )
listctrl_panel.AddButton( 'add gug', self._AddGUG )
listctrl_panel.AddButton( 'add url class', self._AddURLMatch )
listctrl_panel.AddButton( 'add url class', self._AddURLClass )
listctrl_panel.AddButton( 'add parser', self._AddParser )
listctrl_panel.AddButton( 'add login script', self._AddLoginScript )
listctrl_panel.AddButton( 'add headers/bandwidth rules', self._AddDomainMetadata )
@ -141,15 +141,15 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
domain_metadatas_to_include = self._GetDomainMetadatasToInclude( domains )
url_matches_to_include = self._GetURLMatchesToInclude( gugs_to_include )
url_classes_to_include = self._GetURLClassesToInclude( gugs_to_include )
url_matches_to_include = self._FleshOutURLMatchesWithAPILinks( url_matches_to_include )
url_classes_to_include = self._FleshOutURLClassesWithAPILinks( url_classes_to_include )
parsers_to_include = self._GetParsersToInclude( url_matches_to_include )
parsers_to_include = self._GetParsersToInclude( url_classes_to_include )
self._listctrl.AddDatas( domain_metadatas_to_include )
self._listctrl.AddDatas( gugs_to_include )
self._listctrl.AddDatas( url_matches_to_include )
self._listctrl.AddDatas( url_classes_to_include )
self._listctrl.AddDatas( parsers_to_include )
@ -207,13 +207,13 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
self._listctrl.AddDatas( parsers_to_include )
def _AddURLMatch( self ):
def _AddURLClass( self ):
existing_data = self._listctrl.GetData()
choosable_url_matches = [ u for u in self._network_engine.domain_manager.GetURLMatches() if u not in existing_data ]
choosable_url_classes = [ u for u in self._network_engine.domain_manager.GetURLClasses() if u not in existing_data ]
choice_tuples = [ ( url_match.GetName(), url_match, False ) for url_match in choosable_url_matches ]
choice_tuples = [ ( url_class.GetName(), url_class, False ) for url_class in choosable_url_classes ]
with ClientGUITopLevelWindows.DialogEdit( self, 'select url classes' ) as dlg:
@ -223,7 +223,7 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
if dlg.ShowModal() == wx.ID_OK:
url_matches_to_include = panel.GetValue()
url_classes_to_include = panel.GetValue()
else:
@ -231,11 +231,11 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
url_matches_to_include = self._FleshOutURLMatchesWithAPILinks( url_matches_to_include )
url_classes_to_include = self._FleshOutURLClassesWithAPILinks( url_classes_to_include )
parsers_to_include = self._GetParsersToInclude( url_matches_to_include )
parsers_to_include = self._GetParsersToInclude( url_classes_to_include )
self._listctrl.AddDatas( url_matches_to_include )
self._listctrl.AddDatas( url_classes_to_include )
self._listctrl.AddDatas( parsers_to_include )
@ -354,31 +354,31 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
def _FleshOutURLMatchesWithAPILinks( self, url_matches ):
def _FleshOutURLClassesWithAPILinks( self, url_classes ):
url_matches_to_include = set( url_matches )
url_classes_to_include = set( url_classes )
api_links_dict = dict( ClientNetworkingDomain.ConvertURLMatchesIntoAPIPairs( self._network_engine.domain_manager.GetURLMatches() ) )
api_links_dict = dict( ClientNetworkingDomain.ConvertURLClassesIntoAPIPairs( self._network_engine.domain_manager.GetURLClasses() ) )
for url_match in url_matches:
for url_class in url_classes:
added_this_cycle = set()
while url_match in api_links_dict and url_match not in added_this_cycle:
while url_class in api_links_dict and url_class not in added_this_cycle:
added_this_cycle.add( url_match )
added_this_cycle.add( url_class )
url_match = api_links_dict[ url_match ]
url_class = api_links_dict[ url_class ]
url_matches_to_include.add( url_match )
url_classes_to_include.add( url_class )
existing_data = self._listctrl.GetData()
url_matches_to_include = [ u for u in url_matches_to_include if u not in existing_data ]
url_classes_to_include = [ u for u in url_classes_to_include if u not in existing_data ]
return url_matches_to_include
return url_classes_to_include
def _GetDomainMetadatasToInclude( self, domains ):
@ -433,13 +433,13 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
return domain_metadatas
def _GetParsersToInclude( self, url_matches ):
def _GetParsersToInclude( self, url_classes ):
parsers_to_include = set()
for url_match in url_matches:
for url_class in url_classes:
example_url = url_match.GetExampleURL()
example_url = url_class.GetExampleURL()
( url_type, match_name, can_parse ) = self._network_engine.domain_manager.GetURLParseCapability( example_url )
@ -463,9 +463,9 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
return [ p for p in parsers_to_include if p not in existing_data ]
def _GetURLMatchesToInclude( self, gugs ):
def _GetURLClassesToInclude( self, gugs ):
url_matches_to_include = set()
url_classes_to_include = set()
for gug in gugs:
@ -480,21 +480,21 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
for example_url in example_urls:
url_match = self._network_engine.domain_manager.GetURLMatch( example_url )
url_class = self._network_engine.domain_manager.GetURLClass( example_url )
if url_match is not None:
if url_class is not None:
url_matches_to_include.add( url_match )
url_classes_to_include.add( url_class )
# add post url matches from same domain
domain = ClientNetworkingDomain.ConvertURLIntoSecondLevelDomain( example_url )
for um in list( self._network_engine.domain_manager.GetURLMatches() ):
for um in list( self._network_engine.domain_manager.GetURLClasses() ):
if ClientNetworkingDomain.ConvertURLIntoSecondLevelDomain( um.GetExampleURL() ) == domain and um.GetURLType() in ( HC.URL_TYPE_POST, HC.URL_TYPE_FILE ):
url_matches_to_include.add( um )
url_classes_to_include.add( um )
@ -503,7 +503,7 @@ class DownloaderExportPanel( ClientGUIScrolledPanels.ReviewPanel ):
existing_data = self._listctrl.GetData()
return [ u for u in url_matches_to_include if u not in existing_data ]
return [ u for u in url_classes_to_include if u not in existing_data ]
class EditCompoundFormulaPanel( ClientGUIScrolledPanels.EditPanel ):

22
include/ClientGUIPredicates.py
View File

@ -53,7 +53,7 @@ class InputFileSystemPredicate( ClientGUIScrolledPanels.EditPanel ):
pred_classes.append( PanelPredicateSystemKnownURLsExactURL )
pred_classes.append( PanelPredicateSystemKnownURLsDomain )
pred_classes.append( PanelPredicateSystemKnownURLsRegex )
pred_classes.append( PanelPredicateSystemKnownURLsURLMatch )
pred_classes.append( PanelPredicateSystemKnownURLsURLClass )
elif predicate_type == HC.PREDICATE_TYPE_SYSTEM_HASH:
@ -777,7 +777,7 @@ class PanelPredicateSystemKnownURLsRegex( PanelPredicateSystem ):
return ( operator, rule_type, rule, description )
class PanelPredicateSystemKnownURLsURLMatch( PanelPredicateSystem ):
class PanelPredicateSystemKnownURLsURLClass( PanelPredicateSystem ):
PREDICATE_TYPE = HC.PREDICATE_TYPE_SYSTEM_KNOWN_URLS
@ -790,13 +790,13 @@ class PanelPredicateSystemKnownURLsURLMatch( PanelPredicateSystem ):
self._operator.Append( 'has', True )
self._operator.Append( 'does not have', False )
self._url_matches = ClientGUICommon.BetterChoice( self )
self._url_classes = ClientGUICommon.BetterChoice( self )
for url_match in HG.client_controller.network_engine.domain_manager.GetURLMatches():
for url_class in HG.client_controller.network_engine.domain_manager.GetURLClasses():
if url_match.ShouldAssociateWithFiles():
if url_class.ShouldAssociateWithFiles():
self._url_matches.Append( url_match.GetName(), url_match )
self._url_classes.Append( url_class.GetName(), url_class )
@ -805,7 +805,7 @@ class PanelPredicateSystemKnownURLsURLMatch( PanelPredicateSystem ):
hbox.Add( ClientGUICommon.BetterStaticText( self, 'system:known url' ), CC.FLAGS_VCENTER )
hbox.Add( self._operator, CC.FLAGS_VCENTER )
hbox.Add( ClientGUICommon.BetterStaticText( self, 'url matching this class:' ), CC.FLAGS_VCENTER )
hbox.Add( self._url_matches, CC.FLAGS_VCENTER )
hbox.Add( self._url_classes, CC.FLAGS_VCENTER )
self.SetSizer( hbox )
@ -823,13 +823,13 @@ class PanelPredicateSystemKnownURLsURLMatch( PanelPredicateSystem ):
operator_description = 'does not have '
rule_type = 'url_match'
rule_type = 'url_class'
url_match = self._url_matches.GetChoice()
url_class = self._url_classes.GetChoice()
rule = url_match
rule = url_class
description = operator_description + url_match.GetName() + ' url'
description = operator_description + url_class.GetName() + ' url'
return ( operator, rule_type, rule, description )

386
include/ClientGUIScrolledPanelsEdit.py
View File

File diff suppressed because it is too large Load Diff

4
include/ClientGUIScrolledPanelsManagement.py
View File

@ -3500,9 +3500,9 @@ class ManageOptionsPanel( ClientGUIScrolledPanels.ManagePanel ):
message += os.linesep * 2
message += 'To get around this, the client populates a pre-boot and ongoing disk cache. By contiguously frontloading the database into memory, the most important functions do not need to wait on your disk for most of their work.'
message += os.linesep * 2
message += 'If you tend to leave your client on in the background and have a slow drive but a lot of ram, you might like to pump these numbers up. 15s boot cache and 2048MB ongoing can really make a difference on, for instance, a slow laptop drive.'
message += 'If you tend to leave your client on in the background and have a slow drive but a lot of ram, you might like to pump these numbers up. 10s boot cache and 1024MB ongoing can really make a difference on, for instance, a slow laptop drive.'
message += os.linesep * 2
message += 'If you run the database from an SSD, you can reduce or entirely eliminate these values, as the benefit is not so stark. 2s and 256MB is fine.'
message += 'If you run the database from an SSD, you can reduce or entirely eliminate these values, as the benefit is not so stark. 2s and 256MB is plenty.'
message += os.linesep * 2
message += 'Unless you are testing, do not go crazy with this stuff. You can set 8192MB if you like, but there are diminishing (and potentially negative) returns.'

158
include/ClientGUIScrolledPanelsReview.py
View File

@ -1483,7 +1483,7 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
def _ImportPaths( self, paths ):
gugs = []
url_matches = []
url_classes = []
parsers = []
domain_metadatas = []
login_scripts = []
@ -1518,7 +1518,7 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
continue
if isinstance( obj_list, ( ClientNetworkingDomain.GalleryURLGenerator, ClientNetworkingDomain.NestedGalleryURLGenerator, ClientNetworkingDomain.URLMatch, ClientParsing.PageParser, ClientNetworkingDomain.DomainMetadataPackage, ClientNetworkingLogin.LoginScriptDomain ) ):
if isinstance( obj_list, ( ClientNetworkingDomain.GalleryURLGenerator, ClientNetworkingDomain.NestedGalleryURLGenerator, ClientNetworkingDomain.URLClass, ClientParsing.PageParser, ClientNetworkingDomain.DomainMetadataPackage, ClientNetworkingLogin.LoginScriptDomain ) ):
obj_list = HydrusSerialisable.SerialisableList( [ obj_list ] )
@ -1536,9 +1536,9 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
gugs.append( obj )
elif isinstance( obj, ClientNetworkingDomain.URLMatch ):
elif isinstance( obj, ClientNetworkingDomain.URLClass ):
url_matches.append( obj )
url_classes.append( obj )
elif isinstance( obj, ClientParsing.PageParser ):
@ -1571,30 +1571,46 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
# url matches first
dupe_url_matches = []
num_exact_dupe_url_matches = 0
new_url_matches = []
url_class_names_seen = set()
for url_match in url_matches:
dupe_url_classes = []
num_exact_dupe_url_classes = 0
new_url_classes = []
for url_class in url_classes:
if domain_manager.AlreadyHaveExactlyThisURLMatch( url_match ):
if url_class.GetName() in url_class_names_seen:
dupe_url_matches.append( url_match )
num_exact_dupe_url_matches += 1
continue
if domain_manager.AlreadyHaveExactlyThisURLClass( url_class ):
dupe_url_classes.append( url_class )
num_exact_dupe_url_classes += 1
else:
new_url_matches.append( url_match )
new_url_classes.append( url_class )
url_class_names_seen.add( url_class.GetName() )
# now gugs
gug_names_seen = set()
num_exact_dupe_gugs = 0
new_gugs = []
for gug in gugs:
if gug.GetName() in gug_names_seen:
continue
if domain_manager.AlreadyHaveExactlyThisGUG( gug ):
num_exact_dupe_gugs += 1
@ -1603,15 +1619,24 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
new_gugs.append( gug )
gug_names_seen.add( gug.GetName() )
# now parsers
parser_names_seen = set()
num_exact_dupe_parsers = 0
new_parsers = []
for parser in parsers:
if parser.GetName() in parser_names_seen:
continue
if domain_manager.AlreadyHaveExactlyThisParser( parser ):
num_exact_dupe_parsers += 1
@ -1620,15 +1645,24 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
new_parsers.append( parser )
parser_names_seen.add( parser.GetName() )
# now login scripts
login_script_names_seen = set()
num_exact_dupe_login_scripts = 0
new_login_scripts = []
for login_script in login_scripts:
if login_script.GetName() in login_script_names_seen:
continue
if login_manager.AlreadyHaveExactlyThisLoginScript( login_script ):
num_exact_dupe_login_scripts += 1
@ -1637,10 +1671,14 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
new_login_scripts.append( login_script )
login_script_names_seen.add( login_script.GetName() )
# now domain metadata
domains_seen = set()
num_exact_dupe_domain_metadatas = 0
new_domain_metadatas = []
@ -1652,6 +1690,11 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
headers_list = None
bandwidth_rules = None
if domain in domains_seen:
continue
nc = ClientNetworkingContexts.NetworkContext( CC.NETWORK_CONTEXT_DOMAIN, domain )
if domain_metadata.HasHeaders():
@ -1684,13 +1727,15 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
new_domain_metadatas.append( new_dm )
domains_seen.add( domain )
#
total_num_dupes = num_exact_dupe_gugs + num_exact_dupe_url_matches + num_exact_dupe_parsers + num_exact_dupe_domain_metadatas + num_exact_dupe_login_scripts
total_num_dupes = num_exact_dupe_gugs + num_exact_dupe_url_classes + num_exact_dupe_parsers + num_exact_dupe_domain_metadatas + num_exact_dupe_login_scripts
if len( new_gugs ) + len( new_url_matches ) + len( new_parsers ) + len( new_domain_metadatas ) + len( new_login_scripts ) == 0:
if len( new_gugs ) + len( new_url_classes ) + len( new_parsers ) + len( new_domain_metadatas ) + len( new_login_scripts ) == 0:
wx.MessageBox( 'All ' + HydrusData.ToHumanInt( total_num_dupes ) + ' downloader objects in that package appeared to already be in the client, so nothing need be added.' )
@ -1703,7 +1748,7 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
choice_tuples = []
choice_tuples.extend( [ ( 'GUG: ' + gug.GetName(), gug, True ) for gug in new_gugs ] )
choice_tuples.extend( [ ( 'URL Class: ' + url_match.GetName(), url_match, True ) for url_match in new_url_matches ] )
choice_tuples.extend( [ ( 'URL Class: ' + url_class.GetName(), url_class, True ) for url_class in new_url_classes ] )
choice_tuples.extend( [ ( 'Parser: ' + parser.GetName(), parser, True ) for parser in new_parsers ] )
choice_tuples.extend( [ ( 'Login Script: ' + login_script.GetName(), login_script, True ) for login_script in new_login_scripts ] )
choice_tuples.extend( [ ( 'Domain Metadata: ' + domain_metadata.GetDomain(), domain_metadata, True ) for domain_metadata in new_domain_metadatas ] )
@ -1719,7 +1764,7 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
new_objects = panel.GetValue()
new_gugs = [ obj for obj in new_objects if isinstance( obj, ( ClientNetworkingDomain.GalleryURLGenerator, ClientNetworkingDomain.NestedGalleryURLGenerator ) ) ]
new_url_matches = [ obj for obj in new_objects if isinstance( obj, ClientNetworkingDomain.URLMatch ) ]
new_url_classes = [ obj for obj in new_objects if isinstance( obj, ClientNetworkingDomain.URLClass ) ]
new_parsers = [ obj for obj in new_objects if isinstance( obj, ClientParsing.PageParser ) ]
new_login_scripts = [ obj for obj in new_objects if isinstance( obj, ClientNetworkingLogin.LoginScriptDomain ) ]
new_domain_metadatas = [ obj for obj in new_objects if isinstance( obj, ClientNetworkingDomain.DomainMetadataPackage ) ]
@ -1734,7 +1779,7 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
# final ask
new_gugs.sort( key = lambda o: o.GetName() )
new_url_matches.sort( key = lambda o: o.GetName() )
new_url_classes.sort( key = lambda o: o.GetName() )
new_parsers.sort( key = lambda o: o.GetName() )
new_login_scripts.sort( key = lambda o: o.GetName() )
new_domain_metadatas.sort( key = lambda o: o.GetDomain() )
@ -1760,7 +1805,7 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
all_to_add = list( new_gugs )
all_to_add.extend( new_url_matches )
all_to_add.extend( new_url_classes )
all_to_add.extend( new_parsers )
all_to_add.extend( new_login_scripts )
all_to_add.extend( new_domain_metadatas )
@ -1793,14 +1838,14 @@ class ReviewDownloaderImport( ClientGUIScrolledPanels.ReviewPanel ):
domain_manager.AddGUGs( new_gugs )
domain_manager.AutoAddURLMatchesAndParsers( new_url_matches, dupe_url_matches, new_parsers )
domain_manager.AutoAddURLClassesAndParsers( new_url_classes, dupe_url_classes, new_parsers )
bandwidth_manager.AutoAddDomainMetadatas( new_domain_metadatas )
domain_manager.AutoAddDomainMetadatas( new_domain_metadatas, approved = True )
login_manager.AutoAddLoginScripts( new_login_scripts )
num_new_gugs = len( new_gugs )
num_aux = len( new_url_matches ) + len( new_parsers ) + len( new_login_scripts ) + len( new_domain_metadatas )
num_aux = len( new_url_classes ) + len( new_parsers ) + len( new_login_scripts ) + len( new_domain_metadatas )
final_message = 'Successfully added ' + HydrusData.ToHumanInt( num_new_gugs ) + ' new downloaders and ' + HydrusData.ToHumanInt( num_aux ) + ' auxiliary objects.'
@ -2703,13 +2748,13 @@ class ReviewServicesPanel( ClientGUIScrolledPanels.ReviewPanel ):
ClientGUIScrolledPanels.ReviewPanel.__init__( self, parent )
self._notebook = wx.Notebook( self )
self._notebook = ClientGUICommon.BetterNotebook( self )
self._local_listbook = ClientGUICommon.ListBook( self._notebook )
self._remote_listbook = ClientGUICommon.ListBook( self._notebook )
self._local_notebook = ClientGUICommon.BetterNotebook( self._notebook )
self._remote_notebook = ClientGUICommon.BetterNotebook( self._notebook )
self._notebook.AddPage( self._local_listbook, 'local' )
self._notebook.AddPage( self._remote_listbook, 'remote' )
self._notebook.AddPage( self._local_notebook, 'local' )
self._notebook.AddPage( self._remote_notebook, 'remote' )
self._InitialiseServices()
@ -2739,24 +2784,22 @@ class ReviewServicesPanel( ClientGUIScrolledPanels.ReviewPanel ):
previous_service_key = page.GetServiceKey()
self._local_listbook.DeleteAllPages()
self._remote_listbook.DeleteAllPages()
self._local_notebook.DeleteAllPages()
self._remote_notebook.DeleteAllPages()
listbook_dict = {}
notebook_dict = {}
services = self._controller.services_manager.GetServices( randomised = False )
lb_to_select = None
service_type_name_to_select = None
local_remote_notebook_to_select = None
service_type_lb = None
service_name_to_select = None
for service in services:
service_type = service.GetServiceType()
if service_type in HC.LOCAL_SERVICES: parent_listbook = self._local_listbook
else: parent_listbook = self._remote_listbook
if service_type in HC.LOCAL_SERVICES: parent_notebook = self._local_notebook
else: parent_notebook = self._remote_notebook
if service_type == HC.TAG_REPOSITORY: service_type_name = 'tag repositories'
elif service_type == HC.FILE_REPOSITORY: service_type_name = 'file repositories'
@ -2771,51 +2814,34 @@ class ReviewServicesPanel( ClientGUIScrolledPanels.ReviewPanel ):
elif service_type == HC.IPFS: service_type_name = 'ipfs'
else: continue
if service_type_name not in listbook_dict:
if service_type_name not in notebook_dict:
listbook = ClientGUICommon.ListBook( parent_listbook )
services_notebook = ClientGUICommon.BetterNotebook( parent_notebook )
listbook_dict[ service_type_name ] = listbook
notebook_dict[ service_type_name ] = services_notebook
parent_listbook.AddPage( service_type_name, service_type_name, listbook )
parent_notebook.AddPage( services_notebook, service_type_name, select = False )
listbook = listbook_dict[ service_type_name ]
services_notebook = notebook_dict[ service_type_name ]
name = service.GetName()
panel_class = ClientGUIPanels.ReviewServicePanel
listbook.AddPageArgs( name, name, panel_class, ( listbook, service ), {} )
page = ClientGUIPanels.ReviewServicePanel( services_notebook, service )
if service.GetServiceKey() == previous_service_key:
lb_to_select = parent_listbook
service_type_name_to_select = service_name_to_select
service_type_lb = listbook
name_to_select = name
self._notebook.SelectPage( parent_notebook )
parent_notebook.SelectPage( services_notebook )
select = True
else:
select = False
if lb_to_select is not None:
name = service.GetName()
if self._notebook.GetCurrentPage() != lb_to_select:
selection = self._notebook.GetSelection()
if selection == 0:
self._notebook.SetSelection( 1 )
else:
self._notebook.SetSelection( 0 )
lb_to_select.Select( service_name_to_select )
service_type_lb.Select( name_to_select )
services_notebook.AddPage( page, name, select = select )

18
include/ClientGUITags.py
View File

@ -46,7 +46,7 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
#
self._notebook = wx.Notebook( self )
self._notebook = ClientGUICommon.BetterNotebook( self )
#
@ -82,22 +82,22 @@ class EditTagFilterPanel( ClientGUIScrolledPanels.EditPanel ):
if prefer_blacklist:
selection_tests.append( ( blacklist_possible, 0 ) )
selection_tests.append( ( whitelist_possible, 1 ) )
selection_tests.append( ( True, 2 ) )
selection_tests.append( ( blacklist_possible, self._blacklist_panel ) )
selection_tests.append( ( whitelist_possible, self._whitelist_panel ) )
selection_tests.append( ( True, self._advanced_panel ) )
else:
selection_tests.append( ( whitelist_possible, 0 ) )
selection_tests.append( ( blacklist_possible, 1 ) )
selection_tests.append( ( True, 2 ) )
selection_tests.append( ( whitelist_possible, self._whitelist_panel ) )
selection_tests.append( ( blacklist_possible, self._blacklist_panel ) )
selection_tests.append( ( True, self._advanced_panel ) )
for ( test, index ) in selection_tests:
for ( test, page ) in selection_tests:
if test:
self._notebook.SetSelection( index )
self._notebook.SelectPage( page )
break

5
include/ClientGUITopLevelWindows.py
View File

@ -302,8 +302,6 @@ class NewDialog( wx.Dialog ):
self.SetIcon( HG.client_controller.frame_icon )
self.Bind( wx.EVT_BUTTON, self.EventDialogButton )
self.Bind( wx.EVT_MENU_CLOSE, self.EventMenuClose )
self.Bind( wx.EVT_MENU_HIGHLIGHT_ALL, self.EventMenuHighlight )
self.Bind( wx.EVT_MENU_OPEN, self.EventMenuOpen )
@ -689,10 +687,11 @@ class DialogApplyCancel( DialogThatTakesScrollablePanel ):
self._apply = wx.Button( self, id = wx.ID_OK, label = 'apply' )
self._apply.SetForegroundColour( ( 0, 128, 0 ) )
#self._apply.Bind( wx.EVT_BUTTON, self.EventOK )
self._apply.Bind( wx.EVT_BUTTON, self.EventDialogButton )
self._cancel = wx.Button( self, id = wx.ID_CANCEL, label = 'cancel' )
self._cancel.SetForegroundColour( ( 128, 0, 0 ) )
self._cancel.Bind( wx.EVT_BUTTON, self.EventDialogButton )
if self._hide_buttons:

234
include/ClientImageHandling.py
View File

@ -1,3 +1,4 @@
import numpy
import numpy.core.multiarray # important this comes before cv!
from . import ClientConstants as CC
import cv2
@ -8,25 +9,6 @@ from . import HydrusImageHandling
from . import HydrusGlobals as HG
from functools import reduce
if cv2.__version__.startswith( '2' ):
CV_IMREAD_FLAGS_SUPPORTS_ALPHA = cv2.CV_LOAD_IMAGE_UNCHANGED
CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION = CV_IMREAD_FLAGS_SUPPORTS_ALPHA
# there's something wrong with these, but I don't have an easy test env for it atm
# CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION = cv2.CV_LOAD_IMAGE_ANYDEPTH | cv2.CV_LOAD_IMAGE_ANYCOLOR
CV_JPEG_THUMBNAIL_ENCODE_PARAMS = []
CV_PNG_THUMBNAIL_ENCODE_PARAMS = []
else:
CV_IMREAD_FLAGS_SUPPORTS_ALPHA = cv2.IMREAD_UNCHANGED
CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION = cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR # this preserves colour info but does EXIF reorientation and flipping
CV_JPEG_THUMBNAIL_ENCODE_PARAMS = [ cv2.IMWRITE_JPEG_QUALITY, 92 ]
CV_PNG_THUMBNAIL_ENCODE_PARAMS = [ cv2.IMWRITE_PNG_COMPRESSION, 9 ]
cv_interpolation_enum_lookup = {}
cv_interpolation_enum_lookup[ CC.ZOOM_NEAREST ] = cv2.INTER_NEAREST
@ -35,113 +17,24 @@ cv_interpolation_enum_lookup[ CC.ZOOM_AREA ] = cv2.INTER_AREA
cv_interpolation_enum_lookup[ CC.ZOOM_CUBIC ] = cv2.INTER_CUBIC
cv_interpolation_enum_lookup[ CC.ZOOM_LANCZOS4 ] = cv2.INTER_LANCZOS4
def GenerateNumpyImage( path, mime ):
def GenerateNumPyImage( path, mime ):
if HG.media_load_report_mode:
HydrusData.ShowText( 'Loading media: ' + path )
force_pil = HG.client_controller.new_options.GetBoolean( 'load_images_with_pil' )
if mime == HC.IMAGE_GIF or HG.client_controller.new_options.GetBoolean( 'load_images_with_pil' ):
if HG.media_load_report_mode:
HydrusData.ShowText( 'Loading with PIL' )
# a regular cv.imread call, can crash the whole process on random thumbs, hooray, so have this as backup
# it was just the read that was the problem, so this seems to work fine, even if pil is only about half as fast
pil_image = HydrusImageHandling.GeneratePILImage( path )
numpy_image = GenerateNumPyImageFromPILImage( pil_image )
else:
if HG.media_load_report_mode:
HydrusData.ShowText( 'Loading with OpenCV' )
if mime == HC.IMAGE_JPEG:
flags = CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION
else:
flags = CV_IMREAD_FLAGS_SUPPORTS_ALPHA
numpy_image = cv2.imread( path, flags = flags )
if numpy_image is None: # doesn't support static gifs and some random other stuff
if HG.media_load_report_mode:
HydrusData.ShowText( 'OpenCV Failed, loading with PIL' )
pil_image = HydrusImageHandling.GeneratePILImage( path )
numpy_image = GenerateNumPyImageFromPILImage( pil_image )
else:
if numpy_image.dtype == 'uint16':
numpy_image //= 256
numpy_image = numpy.array( numpy_image, dtype = 'uint8' )
shape = numpy_image.shape
if len( shape ) == 2:
# monochrome image
convert = cv2.COLOR_GRAY2RGB
else:
( im_y, im_x, depth ) = shape
if depth == 4:
convert = cv2.COLOR_BGRA2RGBA
else:
convert = cv2.COLOR_BGR2RGB
numpy_image = cv2.cvtColor( numpy_image, convert )
return numpy_image
def GenerateNumPyImageFromPILImage( pil_image ):
pil_image = HydrusImageHandling.Dequantize( pil_image )
( w, h ) = pil_image.size
s = pil_image.tobytes()
return numpy.fromstring( s, dtype = 'uint8' ).reshape( ( h, w, len( s ) // ( w * h ) ) )
return HydrusImageHandling.GenerateNumPyImage( path, mime, force_pil = force_pil )
def GenerateShapePerceptualHashes( path, mime ):
numpy_image = GenerateNumpyImage( path, mime )
numpy_image = GenerateNumPyImage( path, mime )
( y, x, depth ) = numpy_image.shape
if depth == 4:
# doing this on 10000x10000 pngs eats ram like mad
numpy_image = ThumbnailNumpyImage( numpy_image, ( 1024, 1024 ) )
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( x, y ), ( 1024, 1024 ) )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution )
( y, x, depth ) = numpy_image.shape
@ -238,106 +131,19 @@ def GenerateShapePerceptualHashes( path, mime ):
phashes.add( phash )
phashes.discard( CC.BLANK_PHASH )
phashes = DiscardBlankPerceptualHashes( phashes )
# we good
return phashes
def GenerateBytesFromCV( numpy_image, mime ):
def DiscardBlankPerceptualHashes( phashes ):
( im_y, im_x, depth ) = numpy_image.shape
phashes = { phash for phash in phashes if HydrusData.Get64BitHammingDistance( phash, CC.BLANK_PHASH ) > 4 }
if depth == 4:
convert = cv2.COLOR_RGBA2BGRA
else:
convert = cv2.COLOR_RGB2BGR
return phashes
numpy_image = cv2.cvtColor( numpy_image, convert )
if mime == HC.IMAGE_JPEG:
ext = '.jpg'
params = CV_JPEG_THUMBNAIL_ENCODE_PARAMS
else:
ext = '.png'
params = CV_PNG_THUMBNAIL_ENCODE_PARAMS
( result_success, result_byte_array ) = cv2.imencode( ext, numpy_image, params )
if result_success:
thumbnail_bytes = result_byte_array.tostring()
return thumbnail_bytes
else:
raise HydrusExceptions.CantRenderWithCVException( 'Thumb failed to encode!' )
def GenerateThumbnailBytesFromStaticImagePathCV( path, bounding_dimensions, mime ):
if mime == HC.IMAGE_GIF:
return HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
numpy_image = GenerateNumpyImage( path, mime )
thumbnail_numpy_image = ThumbnailNumpyImage( numpy_image, bounding_dimensions )
try:
thumbnail_bytes = GenerateBytesFromCV( thumbnail_numpy_image, mime )
return thumbnail_bytes
except HydrusExceptions.CantRenderWithCVException:
return HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime )
from . import HydrusFileHandling
HydrusFileHandling.GenerateThumbnailBytesFromStaticImagePath = GenerateThumbnailBytesFromStaticImagePathCV
def GetNumPyImageResolution( numpy_image ):
( image_height, image_width, depth ) = numpy_image.shape
return ( image_width, image_height )
def ResizeNumpyImage( numpy_image, target_resolution ):
( target_width, target_height ) = target_resolution
( image_width, image_height, depth ) = numpy_image.shape
if target_width == image_width and target_height == target_width:
return numpy_image
elif target_width > image_height or target_height > image_width:
interpolation = cv2.INTER_LANCZOS4
else:
interpolation = cv2.INTER_AREA
return cv2.resize( numpy_image, ( target_width, target_height ), interpolation = interpolation )
def ResizeNumpyImageForMediaViewer( mime, numpy_image, target_resolution ):
def ResizeNumPyImageForMediaViewer( mime, numpy_image, target_resolution ):
( target_width, target_height ) = target_resolution
new_options = HG.client_controller.new_options
@ -364,17 +170,3 @@ def ResizeNumpyImageForMediaViewer( mime, numpy_image, target_resolution ):
return cv2.resize( numpy_image, ( target_width, target_height ), interpolation = interpolation )
def ThumbnailNumpyImage( numpy_image, bounding_dimensions ):
( bounding_width, bounding_height ) = bounding_dimensions
( image_height, image_width, depth ) = numpy_image.shape
if bounding_width >= image_height and bounding_height >= image_width:
return numpy_image
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( image_width, image_height ), ( bounding_width, bounding_height ) )
return ResizeNumpyImage( numpy_image, target_resolution )

142
include/ClientImportFileSeeds.py
View File

@ -170,7 +170,7 @@ class FileImportJob( object ):
self._pre_import_status = None
self._file_info = None
self._thumbnail = None
self._thumbnail_bytes = None
self._phashes = None
self._extra_hashes = None
@ -182,6 +182,85 @@ class FileImportJob( object ):
self._file_import_options.CheckFileIsValid( size, mime, width, height )
def DoWork( self ):
if HG.file_report_mode:
HydrusData.ShowText( 'New file import job!' )
( pre_import_status, hash, note ) = self.GenerateHashAndStatus()
if self.IsNewToDB():
self.GenerateInfo()
self.CheckIsGoodToImport()
mime = self.GetMime()
HG.client_controller.client_files_manager.AddFile( hash, mime, self._temp_path, thumbnail_bytes = self._thumbnail_bytes )
( import_status, note ) = HG.client_controller.WriteSynchronous( 'import_file', self )
else:
import_status = pre_import_status
self.PubsubContentUpdates()
return ( import_status, hash, note )
def GenerateHashAndStatus( self ):
HydrusImageHandling.ConvertToPngIfBmp( self._temp_path )
self._hash = HydrusFileHandling.GetHashFromPath( self._temp_path )
( self._pre_import_status, hash, note ) = HG.client_controller.Read( 'hash_status', 'sha256', self._hash, prefix = 'file recognised' )
return ( self._pre_import_status, self._hash, note )
def GenerateInfo( self ):
mime = HydrusFileHandling.GetMime( self._temp_path )
new_options = HG.client_controller.new_options
if mime in HC.DECOMPRESSION_BOMB_IMAGES and not self._file_import_options.AllowsDecompressionBombs():
if HydrusImageHandling.IsDecompressionBomb( self._temp_path ):
raise HydrusExceptions.DecompressionBombException( 'Image seems to be a Decompression Bomb!' )
self._file_info = HydrusFileHandling.GetFileInfo( self._temp_path, mime )
( size, mime, width, height, duration, num_frames, num_words ) = self._file_info
if mime in HC.MIMES_WITH_THUMBNAILS:
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions )
percentage_in = HG.client_controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
self._thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( self._temp_path, target_resolution, mime, duration, num_frames, percentage_in = percentage_in )
if mime in HC.MIMES_WE_CAN_PHASH:
self._phashes = ClientImageHandling.GenerateShapePerceptualHashes( self._temp_path, mime )
self._extra_hashes = HydrusFileHandling.GetExtraHashesFromPath( self._temp_path )
def GetExtraHashes( self ):
return self._extra_hashes
@ -219,11 +298,6 @@ class FileImportJob( object ):
return self._phashes
def GetTempPathAndThumbnail( self ):
return ( self._temp_path, self._thumbnail )
def PubsubContentUpdates( self ):
if self._pre_import_status == CC.STATUS_SUCCESSFUL_BUT_REDUNDANT:
@ -255,52 +329,6 @@ class FileImportJob( object ):
return False
def GenerateHashAndStatus( self ):
HydrusImageHandling.ConvertToPngIfBmp( self._temp_path )
self._hash = HydrusFileHandling.GetHashFromPath( self._temp_path )
( self._pre_import_status, hash, note ) = HG.client_controller.Read( 'hash_status', 'sha256', self._hash, prefix = 'file recognised' )
return ( self._pre_import_status, self._hash, note )
def GenerateInfo( self ):
mime = HydrusFileHandling.GetMime( self._temp_path )
new_options = HG.client_controller.new_options
if mime in HC.DECOMPRESSION_BOMB_IMAGES and not self._file_import_options.AllowsDecompressionBombs():
if HydrusImageHandling.IsDecompressionBomb( self._temp_path ):
raise HydrusExceptions.DecompressionBombException( 'Image seems to be a Decompression Bomb!' )
self._file_info = HydrusFileHandling.GetFileInfo( self._temp_path, mime )
( size, mime, width, height, duration, num_frames, num_words ) = self._file_info
if mime in HC.MIMES_WITH_THUMBNAILS:
bounding_dimensions = HG.client_controller.options[ 'thumbnail_dimensions' ]
percentage_in = HG.client_controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
self._thumbnail = HydrusFileHandling.GenerateThumbnailBytes( self._temp_path, bounding_dimensions, mime, width, height, duration, num_frames, percentage_in = percentage_in )
if mime in HC.MIMES_WE_CAN_PHASH:
self._phashes = ClientImageHandling.GenerateShapePerceptualHashes( self._temp_path, mime )
self._extra_hashes = HydrusFileHandling.GetExtraHashesFromPath( self._temp_path )
FILE_SEED_TYPE_HDD = 0
FILE_SEED_TYPE_URL = 1
@ -394,7 +422,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
url = HG.client_controller.network_engine.domain_manager.NormaliseURL( url )
except HydrusExceptions.URLMatchException:
except HydrusExceptions.URLClassException:
continue # not a url--something like "file:///C:/Users/Tall%20Man/Downloads/maxresdefault.jpg" ha ha ha
@ -714,7 +742,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
if url != my_url:
my_url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( my_url )
my_url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( my_url )
( media_result, ) = HG.client_controller.Read( 'media_results', ( hash, ) )
@ -724,9 +752,9 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
if this_files_url != my_url:
this_url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( this_files_url )
this_url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( this_files_url )
if my_url_match == this_url_match:
if my_url_class == this_url_class:
# oh no, the file this source url refers to has a different known url in this same domain
# it is more likely that an edit on this site points to the original elsewhere
@ -787,7 +815,7 @@ class FileSeed( HydrusSerialisable.SerialisableBase ):
file_import_job = FileImportJob( temp_path, file_import_options )
( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
( status, hash, note ) = file_import_job.DoWork()
self.SetStatus( status, note = note )
self.SetHash( hash )

6
include/ClientImportGallerySeeds.py
View File

@ -355,13 +355,13 @@ class GallerySeed( HydrusSerialisable.SerialisableBase ):
# we have failed to parse a next page url, but we would still like one, so let's see if the url match can provide one
url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( url_to_check )
url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( url_to_check )
if url_match is not None and url_match.CanGenerateNextGalleryPage():
if url_class is not None and url_class.CanGenerateNextGalleryPage():
try:
next_page_url = url_match.GetNextGalleryPage( url_to_check )
next_page_url = url_class.GetNextGalleryPage( url_to_check )
next_page_urls = [ next_page_url ]

4
include/ClientImportSimpleURLs.py
View File

@ -955,9 +955,9 @@ class URLsImport( HydrusSerialisable.SerialisableBase ):
for url in urls:
url_match = HG.client_controller.network_engine.domain_manager.GetURLMatch( url )
url_class = HG.client_controller.network_engine.domain_manager.GetURLClass( url )
if url_match is None or url_match.GetURLType() in ( HC.URL_TYPE_FILE, HC.URL_TYPE_POST ):
if url_class is None or url_class.GetURLType() in ( HC.URL_TYPE_FILE, HC.URL_TYPE_POST ):
file_seed = ClientImportFileSeeds.FileSeed( ClientImportFileSeeds.FILE_SEED_TYPE_URL, url )

21
include/ClientImportSubscriptions.py
View File

@ -496,12 +496,12 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
presentation_hashes = []
presentation_hashes_fast = set()
starting_num_urls = file_seed_cache.GetFileSeedCount()
starting_num_unknown = file_seed_cache.GetFileSeedCount( CC.STATUS_UNKNOWN )
starting_num_done = starting_num_urls - starting_num_unknown
while True:
num_urls = file_seed_cache.GetFileSeedCount()
num_unknown = file_seed_cache.GetFileSeedCount( CC.STATUS_UNKNOWN )
num_done = num_urls - num_unknown
file_seed = file_seed_cache.GetNextFileSeed( CC.STATUS_UNKNOWN )
if file_seed is None:
@ -541,9 +541,18 @@ class Subscription( HydrusSerialisable.SerialisableBaseNamed ):
try:
x_out_of_y = 'file ' + HydrusData.ConvertValueRangeToPrettyString( num_done + 1, num_urls ) + ': '
num_urls = file_seed_cache.GetFileSeedCount()
num_unknown = file_seed_cache.GetFileSeedCount( CC.STATUS_UNKNOWN )
num_done = num_urls - num_unknown
job_key.SetVariable( 'popup_gauge_2', ( num_done, num_urls ) )
# 4001/4003 is not as useful as 1/3
human_num_urls = num_urls - starting_num_done
human_num_done = num_done - starting_num_done
x_out_of_y = 'file ' + HydrusData.ConvertValueRangeToPrettyString( human_num_done + 1, human_num_urls ) + ': '
job_key.SetVariable( 'popup_gauge_2', ( human_num_done, human_num_urls ) )
def status_hook( text ):

2
include/ClientLocalServerResources.py
View File

@ -698,7 +698,7 @@ class HydrusResourceClientAPIRestrictedAddFile( HydrusResourceClientAPIRestricte
try:
( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job )
( status, hash, note ) = file_import_job.DoWork()
except:

159
include/ClientMedia.py
View File

@ -37,6 +37,14 @@ def FlattenMedia( media_list ):
return flat_media
def GetDuplicateComparisonScore( shown_media, comparison_media ):
statements_and_scores = GetDuplicateComparisonStatements( shown_media, comparison_media )
total_score = sum( ( score for ( statement, score ) in statements_and_scores.values() ) )
return total_score
def GetDuplicateComparisonStatements( shown_media, comparison_media ):
new_options = HG.client_controller.new_options
@ -50,39 +58,46 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
#
statements = []
score = 0
statements_and_scores = {}
# size
s_size = shown_media.GetSize()
c_size = comparison_media.GetSize()
size_ratio = s_size / c_size
if size_ratio > 2.0:
if s_size != c_size:
statements.append( 'This has a much larger filesize.' )
size_ratio = s_size / c_size
score += duplicate_comparison_score_much_higher_filesize
if size_ratio > 2.0:
operator = '>>'
score = duplicate_comparison_score_much_higher_filesize
elif size_ratio > 1.05:
operator = '>'
score = duplicate_comparison_score_higher_filesize
elif size_ratio < 0.5:
operator = '<<'
score = -duplicate_comparison_score_much_higher_filesize
elif size_ratio < 0.95:
operator = '<'
score = -duplicate_comparison_score_higher_filesize
else:
operator = '\u2248'
score = 0
elif size_ratio > 1.05:
statement = '{} {} {}'.format( HydrusData.ToHumanBytes( s_size ), operator, HydrusData.ToHumanBytes( c_size ) )
statements.append( 'This has a larger filesize.' )
score += duplicate_comparison_score_higher_filesize
elif size_ratio < 0.5:
statements.append( 'This has a much smaller filesize.' )
score -= duplicate_comparison_score_more_tags
elif size_ratio < 0.95:
statements.append( 'This has a smaller filesize.' )
score -= duplicate_comparison_score_higher_filesize
statements_and_scores[ 'filesize' ] = ( statement, score )
# higher/same res
@ -90,7 +105,7 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
s_resolution = shown_media.GetResolution()
c_resolution = comparison_media.GetResolution()
if s_resolution is not None and c_resolution is not None:
if s_resolution is not None and c_resolution is not None and s_resolution != c_resolution:
( s_w, s_h ) = shown_media.GetResolution()
( c_w, c_h ) = comparison_media.GetResolution()
@ -99,36 +114,34 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
if resolution_ratio == 1.0:
if s_resolution != c_resolution:
statements.append( 'The files have the same number of pixels but different resolution.' )
operator = '!='
score = 0
elif resolution_ratio > 2.0:
statements.append( 'This has much higher resolution.' )
operator = '>>'
score = duplicate_comparison_score_much_higher_resolution
score += duplicate_comparison_score_much_higher_resolution
elif resolution_ratio > 1.00:
elif resolution_ratio > 1.0:
statements.append( 'This has higher resolution.' )
score += duplicate_comparison_score_higher_resolution
operator = '>'
score = duplicate_comparison_score_higher_resolution
elif resolution_ratio < 0.5:
statements.append( 'This has much lower resolution.' )
operator = '<<'
score = -duplicate_comparison_score_much_higher_resolution
score -= duplicate_comparison_score_much_higher_resolution
else:
elif resolution_ratio < 1.0:
statements.append( 'This has lower resolution.' )
score -= duplicate_comparison_score_higher_resolution
operator = '<'
score = -duplicate_comparison_score_higher_resolution
statement = '{} {} {}'.format( HydrusData.ConvertResolutionToPrettyString( s_resolution ), operator, HydrusData.ConvertResolutionToPrettyString( c_resolution ) )
statements_and_scores[ 'resolution' ] = ( statement, score )
# same/diff mime
@ -137,7 +150,10 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
if s_mime != c_mime:
statements.append( 'This is ' + HC.mime_string_lookup[ s_mime ] + ', the other is ' + HC.mime_string_lookup[ c_mime ] + '.' )
statement = '{} vs {}'.format( HC.mime_string_lookup[ s_mime ], HC.mime_string_lookup[ c_mime ] )
score = 0
statements_and_scores[ 'mime' ] = ( statement, score )
# more tags
@ -145,28 +161,35 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
s_num_tags = len( shown_media.GetTagsManager().GetCurrentAndPending() )
c_num_tags = len( comparison_media.GetTagsManager().GetCurrentAndPending() )
if s_num_tags > 0 and c_num_tags > 0:
if s_num_tags != c_num_tags:
if s_num_tags > c_num_tags:
if s_num_tags > 0 and c_num_tags > 0:
statements.append( 'This has more tags.' )
if s_num_tags > c_num_tags:
operator = '>'
score = duplicate_comparison_score_more_tags
else:
operator = '<'
score = -duplicate_comparison_score_more_tags
score += duplicate_comparison_score_more_tags
elif s_num_tags > 0:
elif s_num_tags < c_num_tags:
operator = '>>'
score = duplicate_comparison_score_more_tags
statements.append( 'This has fewer tags.' )
elif c_num_tags > 0:
score += duplicate_comparison_score_more_tags
operator = '<<'
score = -duplicate_comparison_score_more_tags
elif s_num_tags > 0:
statement = '{} tags {} {} tags'.format( HydrusData.ToHumanInt( s_num_tags ), operator, HydrusData.ToHumanInt( c_num_tags ) )
statements.append( 'This has tags, the other does not.' )
elif c_num_tags > 0:
statements.append( 'This has no tags, the other does.' )
statements_and_scores[ 'num_tags' ] = ( statement, score )
# older
@ -174,23 +197,27 @@ def GetDuplicateComparisonStatements( shown_media, comparison_media ):
s_ts = shown_media.GetLocationsManager().GetTimestamp( CC.COMBINED_LOCAL_FILE_SERVICE_KEY )
c_ts = comparison_media.GetLocationsManager().GetTimestamp( CC.COMBINED_LOCAL_FILE_SERVICE_KEY )
if s_ts is not None and c_ts is not None:
one_month = 86400 * 30
if s_ts is not None and c_ts is not None and abs( s_ts - c_ts ) > one_month:
if s_ts < c_ts - 86400 * 30:
if s_ts < c_ts:
statements.append( 'This is older.' )
operator = 'older than'
score = duplicate_comparison_score_older
score += duplicate_comparison_score_older
else:
elif c_ts < s_ts - 86400 * 30:
statements.append( 'This is newer.' )
score -= duplicate_comparison_score_older
operator = 'newer than'
score = -duplicate_comparison_score_older
statement = '{} {} {}'.format( HydrusData.TimestampToPrettyTimeDelta( s_ts ), operator, HydrusData.TimestampToPrettyTimeDelta( c_ts ) )
statements_and_scores[ 'time_imported' ] = ( statement, score )
return ( statements, score )
return statements_and_scores
def MergeTagsManagers( tags_managers ):

588
include/ClientNetworkingDomain.py
View File

File diff suppressed because it is too large Load Diff

6
include/ClientRendering.py
View File

@ -50,7 +50,7 @@ def FrameIndexOutOfRange( index, range_start, range_end ):
def GenerateHydrusBitmap( path, mime, compressed = True ):
numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )
numpy_image = ClientImageHandling.GenerateNumPyImage( path, mime )
return GenerateHydrusBitmapFromNumPyImage( numpy_image, compressed = compressed )
@ -94,7 +94,7 @@ class ImageRenderer( object ):
def _Initialise( self ):
self._numpy_image = ClientImageHandling.GenerateNumpyImage( self._path, self._mime )
self._numpy_image = ClientImageHandling.GenerateNumPyImage( self._path, self._mime )
def GetEstimatedMemoryFootprint( self ):
@ -129,7 +129,7 @@ class ImageRenderer( object ):
else:
wx_numpy_image = ClientImageHandling.ResizeNumpyImageForMediaViewer( self._media.GetMime(), self._numpy_image, target_resolution )
wx_numpy_image = ClientImageHandling.ResizeNumPyImageForMediaViewer( self._media.GetMime(), self._numpy_image, target_resolution )
( wx_height, wx_width, wx_depth ) = wx_numpy_image.shape

4
include/ClientSearch.py
View File

@ -804,7 +804,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
( operator, rule_type, rule, description ) = self._value
if rule_type == 'url_match':
if rule_type in ( 'url_match', 'url_class' ):
serialisable_rule = rule.GetSerialisableTuple()
@ -855,7 +855,7 @@ class Predicate( HydrusSerialisable.SerialisableBase ):
( operator, rule_type, serialisable_rule, description ) = serialisable_value
if rule_type == 'url_match':
if rule_type in ( 'url_match', 'url_class' ):
rule = HydrusSerialisable.CreateFromSerialisableTuple( serialisable_rule )

7
include/ClientSerialisable.py
View File

@ -25,13 +25,6 @@ else:
IMREAD_UNCHANGED = cv2.IMREAD_UNCHANGED
png_font = cv2.FONT_HERSHEY_TRIPLEX
greyscale_text_color = 0
title_size = 0.7
payload_description_size = 0.5
text_size = 0.4
def CreateTopImage( width, title, payload_description, text ):
text_extent_bmp = HG.client_controller.bitmap_manager.GetBitmap( 20, 20, 24 )

16
include/ClientVideoHandling.py
View File

@ -44,16 +44,6 @@ def GetCVVideoProperties( path ):
return ( ( width, height ), duration, num_frames )
def GetVideoFrameDuration( path ):
cv_video = cv2.VideoCapture( path )
fps = cv_video.get( CAP_PROP_FPS )
if fps in ( 0, 1000 ): raise HydrusExceptions.CantRenderWithCVException()
return 1000.0 / fps
# the cv code was initially written by @fluffy_cub
class GIFRenderer( object ):
@ -113,7 +103,7 @@ class GIFRenderer( object ):
self._pil_canvas = current_frame
numpy_image = ClientImageHandling.GenerateNumPyImageFromPILImage( self._pil_canvas )
numpy_image = HydrusImageHandling.GenerateNumPyImageFromPILImage( self._pil_canvas )
self._next_render_index = ( self._next_render_index + 1 ) % self._num_frames
@ -195,7 +185,7 @@ class GIFRenderer( object ):
numpy_image = self._GetCurrentFrame()
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, self._target_resolution )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, self._target_resolution )
numpy_image = cv2.cvtColor( numpy_image, cv2.COLOR_BGR2RGB )
@ -222,7 +212,7 @@ class GIFRenderer( object ):
numpy_image = self._GetCurrentFrame()
numpy_image = ClientImageHandling.ResizeNumpyImage( numpy_image, self._target_resolution )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, self._target_resolution )
self._last_frame = numpy_image

2
include/HydrusConstants.py
View File

@ -67,7 +67,7 @@ options = {}
# Misc
NETWORK_VERSION = 18
SOFTWARE_VERSION = 350
SOFTWARE_VERSION = 351
CLIENT_API_VERSION = 6
SERVER_THUMBNAIL_DIMENSIONS = ( 200, 200 )

8
include/HydrusDB.py
View File

@ -671,8 +671,8 @@ class HydrusDB( object ):
# blah_id IN ( ?, ?, ? ) is fast and cacheable but there's a small limit (1024 is too many) to the number of params sql can handle
# so lets do the latter but break it into 256-strong chunks to get a good medium
# this will take a select statement with %s like so:
# SELECT blah_id, blah FROM blahs WHERE blah_id IN %s;
# this will take a select statement with {} like so:
# SELECT blah_id, blah FROM blahs WHERE blah_id IN {};
MAX_CHUNK_SIZE = 256
@ -680,7 +680,7 @@ class HydrusDB( object ):
# and also so we aren't overmaking it when this gets spammed with a lot of len() == 1 calls
if len( xs ) >= MAX_CHUNK_SIZE:
max_statement = select_statement % ( '(' + ','.join( '?' * MAX_CHUNK_SIZE ) + ')' )
max_statement = select_statement.format( '({})'.format( ','.join( '?' * MAX_CHUNK_SIZE ) ) )
for chunk in HydrusData.SplitListIntoChunks( xs, MAX_CHUNK_SIZE ):
@ -691,7 +691,7 @@ class HydrusDB( object ):
else:
chunk_statement = select_statement % ( '(' + ','.join( '?' * len( chunk ) ) + ')' )
chunk_statement = select_statement.format( '({})'.format( ','.join( '?' * len( chunk ) ) ) )
for row in self._c.execute( chunk_statement, chunk ):

5
include/HydrusData.py
View File

@ -75,11 +75,6 @@ def ConvertFloatToPercentage( f ):
return '{:.1f}%'.format( f * 100 )
def ConvertIntToFirst( n ):
# straight from stack, wew
return "%d%s" % (n,"tsnrhtdd"[(n/10%10!=1)*(n%10<4)*n%10::4])
def ConvertIntToPixels( i ):
if i == 1: return 'pixels'

2
include/HydrusExceptions.py
View File

@ -28,7 +28,7 @@ class DecompressionBombException( SizeException ): pass
class ParseException( HydrusException ): pass
class StringConvertException( ParseException ): pass
class StringMatchException( ParseException ): pass
class URLMatchException( ParseException ): pass
class URLClassException( ParseException ): pass
class GUGException( ParseException ): pass
class NetworkException( HydrusException ): pass

76
include/HydrusFileHandling.py
View File

@ -13,7 +13,6 @@ from . import HydrusVideoHandling
import os
import threading
import traceback
import io
# Mime
@ -53,37 +52,14 @@ header_and_mime = [
( 0, b'\x30\x26\xB2\x75\x8E\x66\xCF\x11\xA6\xD9\x00\xAA\x00\x62\xCE\x6C', HC.UNDETERMINED_WM )
]
def SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f ):
# when the palette is limited, the thumbnail antialias won't add new colours, so you get nearest-neighbour-like behaviour
original_file_was_png = pil_image.format == 'PNG'
pil_image = HydrusImageHandling.Dequantize( pil_image )
thumbnail_dimensions = HydrusImageHandling.GetThumbnailResolution( pil_image.size, bounding_dimensions )
HydrusImageHandling.ResizePILImage( pil_image, thumbnail_dimensions )
if original_file_was_png or pil_image.mode == 'RGBA':
pil_image.save( f, 'PNG' )
else:
pil_image.save( f, 'JPEG', quality = 92 )
def GenerateThumbnailBytes( path, bounding_dimensions, mime, width, height, duration, num_frames, percentage_in = 35 ):
def GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames, percentage_in = 35 ):
if mime in ( HC.IMAGE_JPEG, HC.IMAGE_PNG, HC.IMAGE_GIF, HC.IMAGE_WEBP, HC.IMAGE_TIFF ):
thumbnail_bytes = GenerateThumbnailBytesFromStaticImagePath( path, bounding_dimensions, mime )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesFromStaticImagePath( path, target_resolution, mime )
else:
f = io.BytesIO()
if mime == HC.APPLICATION_FLASH:
( os_file_handle, temp_path ) = HydrusPaths.GetTempPath()
@ -92,30 +68,22 @@ def GenerateThumbnailBytes( path, bounding_dimensions, mime, width, height, dura
HydrusFlashHandling.RenderPageToFile( path, temp_path, 1 )
pil_image = HydrusImageHandling.GeneratePILImage( temp_path )
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesFromStaticImagePath( temp_path, target_resolution, mime )
except:
flash_default_path = os.path.join( HC.STATIC_DIR, 'flash.png' )
thumb_path = os.path.join( HC.STATIC_DIR, 'flash.png' )
pil_image = HydrusImageHandling.GeneratePILImage( flash_default_path )
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesFromStaticImagePath( thumb_path, target_resolution, mime )
finally:
del pil_image
HydrusPaths.CleanUpTempPath( os_file_handle, temp_path )
else:
cropped_dimensions = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions )
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, cropped_dimensions )
renderer = HydrusVideoHandling.VideoRendererFFMPEG( path, mime, duration, num_frames, target_resolution )
renderer.read_frame() # this initialises the renderer and loads the first frame as a fallback
@ -130,46 +98,18 @@ def GenerateThumbnailBytes( path, bounding_dimensions, mime, width, height, dura
raise Exception( 'Could not create a thumbnail from that video!' )
pil_image = HydrusImageHandling.GeneratePILImageFromNumpyImage( numpy_image )
numpy_image = HydrusImageHandling.ResizeNumPyImage( numpy_image, target_resolution ) # just in case ffmpeg doesn't deliver right
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
thumbnail_bytes = HydrusImageHandling.GenerateThumbnailBytesNumPy( numpy_image, mime )
renderer.Stop()
del renderer
f.seek( 0 )
thumbnail_bytes = f.read()
f.close()
return thumbnail_bytes
def GenerateThumbnailBytesFromPIL( pil_image, bounding_dimensions, mime ):
f = io.BytesIO()
SaveThumbnailToStreamPIL( pil_image, bounding_dimensions, f )
f.seek( 0 )
thumbnail_bytes = f.read()
f.close()
return thumbnail_bytes
def GenerateThumbnailBytesFromStaticImagePathPIL( path, bounding_dimensions, mime ):
pil_image = HydrusImageHandling.GeneratePILImage( path )
return GenerateThumbnailBytesFromPIL( pil_image, bounding_dimensions, mime )
GenerateThumbnailBytesFromStaticImagePath = GenerateThumbnailBytesFromStaticImagePathPIL
def GetExtraHashesFromPath( path ):
h_md5 = hashlib.md5()

302
include/HydrusImageHandling.py
View File

@ -1,6 +1,9 @@
from . import HydrusConstants as HC
from . import HydrusExceptions
from . import HydrusThreading
import io
import numpy
import numpy.core.multiarray # important this comes before cv!
import os
from PIL import _imaging
from PIL import ImageFile as PILImageFile
@ -36,6 +39,37 @@ warnings.simplefilter( 'ignore', PILImage.DecompressionBombWarning )
OLD_PIL_MAX_IMAGE_PIXELS = PILImage.MAX_IMAGE_PIXELS
PILImage.MAX_IMAGE_PIXELS = None # this turns off decomp check entirely, wew
try:
import cv2
if cv2.__version__.startswith( '2' ):
CV_IMREAD_FLAGS_SUPPORTS_ALPHA = cv2.CV_LOAD_IMAGE_UNCHANGED
CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION = CV_IMREAD_FLAGS_SUPPORTS_ALPHA
# there's something wrong with these, but I don't have an easy test env for it atm
# CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION = cv2.CV_LOAD_IMAGE_ANYDEPTH | cv2.CV_LOAD_IMAGE_ANYCOLOR
CV_JPEG_THUMBNAIL_ENCODE_PARAMS = []
CV_PNG_THUMBNAIL_ENCODE_PARAMS = []
else:
CV_IMREAD_FLAGS_SUPPORTS_ALPHA = cv2.IMREAD_UNCHANGED
CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION = cv2.IMREAD_ANYDEPTH | cv2.IMREAD_ANYCOLOR # this preserves colour info but does EXIF reorientation and flipping
CV_JPEG_THUMBNAIL_ENCODE_PARAMS = [ cv2.IMWRITE_JPEG_QUALITY, 92 ]
CV_PNG_THUMBNAIL_ENCODE_PARAMS = [ cv2.IMWRITE_PNG_COMPRESSION, 9 ]
OPENCV_OK = True
except:
OPENCV_OK = False
def ConvertToPngIfBmp( path ):
with open( path, 'rb' ) as f:
@ -83,6 +117,108 @@ def Dequantize( pil_image ):
return pil_image
def GenerateNumPyImage( path, mime, force_pil = False ):
if HG.media_load_report_mode:
HydrusData.ShowText( 'Loading media: ' + path )
if not OPENCV_OK:
force_pil = True
if mime == HC.IMAGE_GIF or force_pil:
if HG.media_load_report_mode:
HydrusData.ShowText( 'Loading with PIL' )
# a regular cv.imread call, can crash the whole process on random thumbs, hooray, so have this as backup
# it was just the read that was the problem, so this seems to work fine, even if pil is only about half as fast
pil_image = GeneratePILImage( path )
numpy_image = GenerateNumPyImageFromPILImage( pil_image )
else:
if HG.media_load_report_mode:
HydrusData.ShowText( 'Loading with OpenCV' )
if mime == HC.IMAGE_JPEG:
flags = CV_IMREAD_FLAGS_SUPPORTS_EXIF_REORIENTATION
else:
flags = CV_IMREAD_FLAGS_SUPPORTS_ALPHA
numpy_image = cv2.imread( path, flags = flags )
if numpy_image is None: # doesn't support static gifs and some random other stuff
if HG.media_load_report_mode:
HydrusData.ShowText( 'OpenCV Failed, loading with PIL' )
pil_image = GeneratePILImage( path )
numpy_image = GenerateNumPyImageFromPILImage( pil_image )
else:
if numpy_image.dtype == 'uint16':
numpy_image //= 256
numpy_image = numpy.array( numpy_image, dtype = 'uint8' )
shape = numpy_image.shape
if len( shape ) == 2:
# monochrome image
convert = cv2.COLOR_GRAY2RGB
else:
( im_y, im_x, depth ) = shape
if depth == 4:
convert = cv2.COLOR_BGRA2RGBA
else:
convert = cv2.COLOR_BGR2RGB
numpy_image = cv2.cvtColor( numpy_image, convert )
return numpy_image
def GenerateNumPyImageFromPILImage( pil_image ):
pil_image = Dequantize( pil_image )
( w, h ) = pil_image.size
s = pil_image.tobytes()
return numpy.fromstring( s, dtype = 'uint8' ).reshape( ( h, w, len( s ) // ( w * h ) ) )
def GeneratePILImage( path ):
fp = open( path, 'rb' )
@ -91,13 +227,13 @@ def GeneratePILImage( path ):
pil_image = PILImage.open( fp )
except:
except Exception as e:
# pil doesn't clean up its open file on exception, jej
fp.close()
raise
raise HydrusExceptions.MimeException( 'Could not load the image!' + os.linesep + traceback.format_exc() )
if pil_image.format == 'JPEG' and hasattr( pil_image, '_getexif' ):
@ -178,7 +314,7 @@ def GeneratePILImage( path ):
return pil_image
def GeneratePILImageFromNumpyImage( numpy_image ):
def GeneratePILImageFromNumPyImage( numpy_image ):
( h, w, depth ) = numpy_image.shape
@ -195,6 +331,107 @@ def GeneratePILImageFromNumpyImage( numpy_image ):
return pil_image
def GenerateThumbnailBytesFromStaticImagePath( path, target_resolution, mime ):
if OPENCV_OK and mime != HC.IMAGE_GIF:
numpy_image = GenerateNumPyImage( path, mime )
thumbnail_numpy_image = ResizeNumPyImage( numpy_image, target_resolution )
try:
thumbnail_bytes = GenerateThumbnailBytesNumPy( thumbnail_numpy_image, mime )
return thumbnail_bytes
except HydrusExceptions.CantRenderWithCVException:
pass # fall back to pil
pil_image = GeneratePILImage( path )
pil_image = Dequantize( pil_image )
thumbnail_pil_image = pil_image.resize( target_resolution, PILImage.ANTIALIAS )
thumbnail_bytes = GenerateThumbnailBytesPIL( pil_image, mime )
return thumbnail_bytes
def GenerateThumbnailBytesNumPy( numpy_image, mime ):
if not OPENCV_OK:
pil_image = GeneratePILImageFromNumPyImage( numpy_image )
return GenerateThumbnailBytesPIL( pil_image, mime )
( im_y, im_x, depth ) = numpy_image.shape
if depth == 4:
convert = cv2.COLOR_RGBA2BGRA
else:
convert = cv2.COLOR_RGB2BGR
numpy_image = cv2.cvtColor( numpy_image, convert )
if mime == HC.IMAGE_JPEG:
ext = '.jpg'
params = CV_JPEG_THUMBNAIL_ENCODE_PARAMS
else:
ext = '.png'
params = CV_PNG_THUMBNAIL_ENCODE_PARAMS
( result_success, result_byte_array ) = cv2.imencode( ext, numpy_image, params )
if result_success:
thumbnail_bytes = result_byte_array.tostring()
return thumbnail_bytes
else:
raise HydrusExceptions.CantRenderWithCVException( 'Thumb failed to encode!' )
def GenerateThumbnailBytesPIL( pil_image, mime ):
f = io.BytesIO()
pil_image = Dequantize( pil_image )
if mime == HC.IMAGE_PNG or pil_image.mode == 'RGBA':
pil_image.save( f, 'PNG' )
else:
pil_image.save( f, 'JPEG', quality = 92 )
f.seek( 0 )
thumbnail_bytes = f.read()
f.close()
return thumbnail_bytes
def GetGIFFrameDurations( path ):
pil_image = GeneratePILImage( path )
@ -238,19 +475,31 @@ def GetGIFFrameDurations( path ):
def GetImageProperties( path, mime ):
( ( width, height ), num_frames ) = GetResolutionAndNumFrames( path, mime )
if num_frames > 1:
if OPENCV_OK and mime != HC.IMAGE_GIF: # webp here too maybe eventually, or offload it all to ffmpeg
durations = GetGIFFrameDurations( path )
numpy_image = GenerateNumPyImage( path, mime )
duration = sum( durations )
else:
( width, height ) = GetResolutionNumPy( numpy_image )
duration = None
num_frames = None
else:
( ( width, height ), num_frames ) = GetResolutionAndNumFramesPIL( path, mime )
if num_frames > 1:
durations = GetGIFFrameDurations( path )
duration = sum( durations )
else:
duration = None
num_frames = None
return ( ( width, height ), duration, num_frames )
@ -269,7 +518,13 @@ def GetPSDResolution( path ):
return ( width, height )
def GetResolutionAndNumFrames( path, mime ):
def GetResolutionNumPy( numpy_image ):
( image_height, image_width, depth ) = numpy_image.shape
return ( image_width, image_height )
def GetResolutionAndNumFramesPIL( path, mime ):
pil_image = GeneratePILImage( path )
@ -306,10 +561,10 @@ def GetResolutionAndNumFrames( path, mime ):
return ( ( x, y ), num_frames )
def GetThumbnailResolution( image_resolution, bounding_resolution ):
def GetThumbnailResolution( image_resolution, bounding_dimensions ):
( im_width, im_height ) = image_resolution
( bounding_width, bounding_height ) = bounding_resolution
( bounding_width, bounding_height ) = bounding_dimensions
if bounding_width >= im_width and bounding_height >= im_height:
@ -360,10 +615,23 @@ def IsDecompressionBomb( path ):
return False
def ResizePILImage( pil_image, target_resolution ):
def ResizeNumPyImage( numpy_image, target_resolution ):
( target_x, target_y ) = target_resolution
( im_x, im_y ) = pil_image.size
( target_width, target_height ) = target_resolution
( image_width, image_height ) = GetResolutionNumPy( numpy_image )
return pil_image.resize( ( target_x, target_y ), PILImage.ANTIALIAS )
if target_width == image_width and target_height == target_width:
return numpy_image
elif target_width > image_height or target_height > image_width:
interpolation = cv2.INTER_LANCZOS4
else:
interpolation = cv2.INTER_AREA
return cv2.resize( numpy_image, ( target_width, target_height ), interpolation = interpolation )

5
include/HydrusPaths.py
View File

@ -508,10 +508,7 @@ def LaunchFile( path, launch_path = None ):
def MakeSureDirectoryExists( path ):
if not os.path.exists( path ):
os.makedirs( path )
os.makedirs( path, exist_ok = True )
def MakeFileWritable( path, recursive = True ):

2
include/HydrusSerialisable.py
View File

@ -64,7 +64,7 @@ SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER = 46
SERIALISABLE_TYPE_NETWORK_CONTEXT = 47
SERIALISABLE_TYPE_NETWORK_LOGIN_MANAGER = 48
SERIALISABLE_TYPE_MEDIA_SORT = 49
SERIALISABLE_TYPE_URL_MATCH = 50
SERIALISABLE_TYPE_URL_CLASS = 50
SERIALISABLE_TYPE_STRING_MATCH = 51
SERIALISABLE_TYPE_CHECKER_OPTIONS = 52
SERIALISABLE_TYPE_NETWORK_DOMAIN_MANAGER = 53

4
include/HydrusServerResources.py
View File

@ -253,7 +253,9 @@ def ParseFileArguments( path, decompression_bombs_ok = False ):
bounding_dimensions = HC.SERVER_THUMBNAIL_DIMENSIONS
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( path, bounding_dimensions, mime, width, height, duration, num_frames )
target_resolution = HydrusImageHandling.GetThumbnailResolution( ( width, height ), bounding_dimensions )
thumbnail_bytes = HydrusFileHandling.GenerateThumbnailBytes( path, target_resolution, mime, duration, num_frames )
except Exception as e:

26
include/ServerDB.py
View File

@ -782,7 +782,7 @@ class DB( HydrusDB.HydrusDB ):
def _GetHashes( self, master_hash_ids ):
select_statement = 'SELECT hash FROM hashes WHERE master_hash_id IN %s;'
select_statement = 'SELECT hash FROM hashes WHERE master_hash_id IN {};'
return [ hash for ( hash, ) in self._SelectFromList( select_statement, master_hash_ids ) ]
@ -1319,7 +1319,7 @@ class DB( HydrusDB.HydrusDB ):
else:
select_statement = 'SELECT service_hash_id FROM ' + deleted_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN %s;'
select_statement = 'SELECT service_hash_id FROM ' + deleted_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN {};'
deleted_service_hash_ids = self._STI( self._SelectFromList( select_statement, service_hash_ids ) )
@ -1569,7 +1569,7 @@ class DB( HydrusDB.HydrusDB ):
( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
select_statement = 'SELECT service_hash_id FROM ' + current_files_table_name + ' WHERE service_hash_id IN %s;'
select_statement = 'SELECT service_hash_id FROM ' + current_files_table_name + ' WHERE service_hash_id IN {};'
valid_service_hash_ids = self._STL( self._SelectFromList( select_statement, service_hash_ids ) )
@ -1587,7 +1587,7 @@ class DB( HydrusDB.HydrusDB ):
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
select_statement = 'SELECT service_hash_id FROM ' + current_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN %s;'
select_statement = 'SELECT service_hash_id FROM ' + current_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN {};'
valid_service_hash_ids = self._STL( self._SelectFromList( select_statement, service_hash_ids ) )
@ -2146,7 +2146,7 @@ class DB( HydrusDB.HydrusDB ):
( hash_id_map_table_name, tag_id_map_table_name ) = GenerateRepositoryMasterMapTableNames( service_id )
select_statement = 'SELECT master_hash_id FROM ' + hash_id_map_table_name + ' WHERE service_hash_id IN %s;'
select_statement = 'SELECT master_hash_id FROM ' + hash_id_map_table_name + ' WHERE service_hash_id IN {};'
master_hash_ids = [ master_hash_id for ( master_hash_id, ) in self._SelectFromList( select_statement, service_hash_ids ) ]
@ -2527,7 +2527,7 @@ class DB( HydrusDB.HydrusDB ):
( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
select_statement = 'SELECT service_hash_id FROM ' + current_files_table_name + ' WHERE service_hash_id IN %s;'
select_statement = 'SELECT service_hash_id FROM ' + current_files_table_name + ' WHERE service_hash_id IN {};'
valid_service_hash_ids = [ service_hash_id for ( service_hash_id, ) in self._SelectFromList( select_statement, service_hash_ids ) ]
@ -2540,7 +2540,7 @@ class DB( HydrusDB.HydrusDB ):
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
select_statement = 'SELECT service_hash_id FROM ' + current_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN %s;'
select_statement = 'SELECT service_hash_id FROM ' + current_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN {};'
valid_service_hash_ids = [ service_hash_id for ( service_hash_id, ) in self._SelectFromList( select_statement, service_hash_ids ) ]
@ -2884,7 +2884,7 @@ class DB( HydrusDB.HydrusDB ):
( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
select_statement = 'SELECT account_id, COUNT( * ) FROM ' + petitioned_files_table_name + ' WHERE service_hash_id IN %s GROUP BY account_id;'
select_statement = 'SELECT account_id, COUNT( * ) FROM ' + petitioned_files_table_name + ' WHERE service_hash_id IN {} GROUP BY account_id;'
scores = [ ( account_id, count * multiplier ) for ( account_id, count ) in self._SelectFromList( select_statement, service_hash_ids ) ]
@ -2895,7 +2895,7 @@ class DB( HydrusDB.HydrusDB ):
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
select_statement = 'SELECT account_id, COUNT( * ) FROM ' + petitioned_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN %s GROUP BY account_id;'
select_statement = 'SELECT account_id, COUNT( * ) FROM ' + petitioned_mappings_table_name + ' WHERE service_tag_id = ' + str( service_tag_id ) + ' AND service_hash_id IN {} GROUP BY account_id;'
scores = [ ( account_id, count * multiplier ) for ( account_id, count ) in self._SelectFromList( select_statement, service_hash_ids ) ]
@ -3010,7 +3010,7 @@ class DB( HydrusDB.HydrusDB ):
( current_files_table_name, deleted_files_table_name, pending_files_table_name, petitioned_files_table_name, ip_addresses_table_name ) = GenerateRepositoryFilesTableNames( service_id )
select_statement = 'SELECT service_hash_id FROM ' + current_files_table_name + ' WHERE account_id IN %s;'
select_statement = 'SELECT service_hash_id FROM ' + current_files_table_name + ' WHERE account_id IN {};'
service_hash_ids = [ service_hash_id for ( service_hash_id, ) in self._SelectFromList( select_statement, subject_account_ids ) ]
@ -3021,7 +3021,7 @@ class DB( HydrusDB.HydrusDB ):
( current_mappings_table_name, deleted_mappings_table_name, pending_mappings_table_name, petitioned_mappings_table_name ) = GenerateRepositoryMappingsTableNames( service_id )
select_statement = 'SELECT service_tag_id, service_hash_id FROM ' + current_mappings_table_name + ' WHERE account_id IN %s;'
select_statement = 'SELECT service_tag_id, service_hash_id FROM ' + current_mappings_table_name + ' WHERE account_id IN {};'
mappings_dict = HydrusData.BuildKeyToListDict( self._SelectFromList( select_statement, subject_account_ids ) )
@ -3035,7 +3035,7 @@ class DB( HydrusDB.HydrusDB ):
( current_tag_parents_table_name, deleted_tag_parents_table_name, pending_tag_parents_table_name, petitioned_tag_parents_table_name ) = GenerateRepositoryTagParentsTableNames( service_id )
select_statement = 'SELECT child_service_tag_id, parent_service_tag_id FROM ' + current_tag_parents_table_name + ' WHERE account_id IN %s;'
select_statement = 'SELECT child_service_tag_id, parent_service_tag_id FROM ' + current_tag_parents_table_name + ' WHERE account_id IN {};'
pairs = self._SelectFromListFetchAll( select_statement, subject_account_ids )
@ -3049,7 +3049,7 @@ class DB( HydrusDB.HydrusDB ):
( current_tag_siblings_table_name, deleted_tag_siblings_table_name, pending_tag_siblings_table_name, petitioned_tag_siblings_table_name ) = GenerateRepositoryTagSiblingsTableNames( service_id )
select_statement = 'SELECT bad_service_tag_id, good_service_tag_id FROM ' + current_tag_siblings_table_name + ' WHERE account_id IN %s;'
select_statement = 'SELECT bad_service_tag_id, good_service_tag_id FROM ' + current_tag_siblings_table_name + ' WHERE account_id IN {};'
pairs = self._SelectFromListFetchAll( select_statement, subject_account_ids )

9
include/TestClientImageHandling.py
View File

@ -1,3 +1,4 @@
from . import ClientConstants as CC
from . import ClientImageHandling
import collections
from . import HydrusConstants as HC
@ -12,3 +13,11 @@ class TestImageHandling( unittest.TestCase ):
self.assertEqual( phashes, set( [ b'\xb4M\xc7\xb2M\xcb8\x1c' ] ) )
phashes = ClientImageHandling.DiscardBlankPerceptualHashes( { CC.BLANK_PHASH } )
self.assertEqual( phashes, set() )
phashes = ClientImageHandling.DiscardBlankPerceptualHashes( { b'\xb4M\xc7\xb2M\xcb8\x1c', CC.BLANK_PHASH } )
self.assertEqual( phashes, set( [ b'\xb4M\xc7\xb2M\xcb8\x1c' ] ) )